[BUG] Rich 12.0.0 fails to write to stdout attached to a pipe on Windows

[ichard26]

Describe the bug

Since rich 12.0.0, printing to stdout fails running under a subprocess attached to a pipe on Windows.

To reproduce, save the following reproducer and run it first with rich==12.0.0

import subprocess
import sys

code = """\
import sys
from rich.console import Console

console = Console(file=sys.stdout)
console.print("spam")
"""
err_code = code.replace("sys.stdout", "sys.stderr")
cmd = [sys.executable, "-"]

print("[capture_output=True via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=True via sys.stderr]")
proc = subprocess.run(cmd, input=err_code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=False via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=False, encoding="utf-8")

There's a few observations here:

• With rich==12.0.0 installed, the captured stdout is empty. With an older version it works fine (see orange / blue)
• Standard error is completely unaffected. It's also fine letting the subprocess inherit the streams (see green)

Platform

Python: CPython 3.8.5
OS: Windows 10 Home 21H2 (build 19044.1526)
Terminal: Windows Terminal 1.11.3471.0

rich.diagnose

(venv) PS R:\Programming\diff-shades> python -m rich.diagnose
╭───────────────────────── <class 'rich.console.Console'> ─────────────────────────╮
│ A high level console interface.                                                  │
│                                                                                  │
│ ╭──────────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=148 ColorSystem.TRUECOLOR>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                  │
│     color_system = 'truecolor'                                                   │
│         encoding = 'utf-8'                                                       │
│             file = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> │
│           height = 35                                                            │
│    is_alt_screen = False                                                         │
│ is_dumb_terminal = False                                                         │
│   is_interactive = True                                                          │
│       is_jupyter = False                                                         │
│      is_terminal = True                                                          │
│   legacy_windows = False                                                         │
│         no_color = False                                                         │
│          options = ConsoleOptions(                                               │
│                        size=ConsoleDimensions(width=148, height=35),             │
│                        legacy_windows=False,                                     │
│                        min_width=1,                                              │
│                        max_width=148,                                            │
│                        is_terminal=True,                                         │
│                        encoding='utf-8',                                         │
│                        max_height=35,                                            │
│                        justify=None,                                             │
│                        overflow=None,                                            │
│                        no_wrap=False,                                            │
│                        highlight=None,                                           │
│                        markup=None,                                              │
│                        height=None                                               │
│                    )                                                             │
│            quiet = False                                                         │
│           record = False                                                         │
│         safe_box = True                                                          │
│             size = ConsoleDimensions(width=148, height=35)                       │
│        soft_wrap = False                                                         │
│           stderr = False                                                         │
│            style = None                                                          │
│         tab_size = 8                                                             │
│            width = 148                                                           │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭── <class 'rich._windows.WindowsConsoleFeatures'> ───╮
│ Windows features available.                         │
│                                                     │
│ ╭─────────────────────────────────────────────────╮ │
│ │ WindowsConsoleFeatures(vt=True, truecolor=True) │ │
│ ╰─────────────────────────────────────────────────╯ │
│                                                     │
│ truecolor = True                                    │
│        vt = True                                    │
╰─────────────────────────────────────────────────────╯
╭────── Environment Variables ───────╮
│ {                                  │
│     'TERM': None,                  │
│     'COLORTERM': None,             │
│     'CLICOLOR': None,              │
│     'NO_COLOR': None,              │
│     'TERM_PROGRAM': None,          │
│     'COLUMNS': None,               │
│     'LINES': None,                 │
│     'JPY_PARENT_PID': None,        │
│     'VSCODE_VERBOSE_LOGGING': None │
│ }                                  │
╰────────────────────────────────────╯
platform="Windows"

Additional context

This was discovered by GHA Windows CI (the problem is hidden and buried in this test, but this shows it has been reproduced under several environments¹). Original investigation into this strange behaviour happened in the #black-formatter channel on Python Discord.

According to @jack1142 who helped me investigate it, it's related to the recent move to replace colorama with first-party code that interacts with the Windows APIs:

I'm pretty sure it's because it can't use WriteConsole API (the one that's used for legacy console printing) when it's piped because output from WriteConsole is not capturable

They then shared this excerpt from Windows console API documentation:

WriteConsole fails if it is used with a standard handle that is redirected to a file. If an application processes multilingual output that can be redirected, determine whether the output handle is a console handle (one method is to call the GetConsoleMode function and check whether it succeeds). If the handle is a console handle, call WriteConsole. If the handle is not a console handle, the output is redirected and you should call WriteFile to perform the I/O.
https://docs.microsoft.com/en-us/windows/console/writeconsole

Footnotes

1. if you're curious, it's basically testing whether recording the output (via Console(record=True)) and then dumping it as HTML works as expected with the rest of the tool ↩

[willmcgugan]

Thanks. Will take a look.

[github-actions]

Did I solve your problem?

Why not buy the devs a coffee to say thanks?

[ichard26]

OK with 51121fe installed bug.py is working correctly, but there's still three more situations which are crashing or working strangely 🙁

bug_reprint.py -- calling console.print again crashes with an OSError

import subprocess
import sys

code = """\
from rich.console import Console

console = Console(stderr=False)
console.print("spam")
console.print("eggs")
"""
err_code = code.replace("stderr=False", "stderr=True")
cmd = [sys.executable, "-"]

print("[capture_output=True via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=True via sys.stderr]")
proc = subprocess.run(cmd, input=err_code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=False via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=False, encoding="utf-8")

(venv) PS R:\Programming\diff-shades> python .\bug_reprint.py
[capture_output=True via sys.stdout]
proc.stdout='spam\n' proc.stderr='Traceback (most recent call last):\n  File "<stdin>", line 5, in <module>\n  File "R:\\Programming\\diff-shades\\venv\\lib\\site-packages\\rich\\console.py", line 1642, in print\n    self._buffer.extend(new_segments)\n  File "R:\\Programming\\diff-shades\\venv\\lib\\site-packages\\rich\\console.py", line 841, in __exit__\n    self._exit_buffer()\n  File "R:\\Programming\\diff-shades\\venv\\lib\\site-packages\\rich\\console.py", line 799, in _exit_buffer\n    self._check_buffer()\n  File "R:\\Programming\\diff-shades\\venv\\lib\\site-packages\\rich\\console.py", line 1935, in _check_buffer\n    with open(file_no, "w") as output_file:\nOSError: [WinError 6] The handle is invalid\n'
[capture_output=True via sys.stderr]
proc.stdout='' proc.stderr='spam\neggs\n'
[capture_output=False via sys.stdout]
spam
eggs

with rich==11.2.0

(venv) PS R:\Programming\diff-shades> python .\bug_reprint.py
[capture_output=True via sys.stdout]
proc.stdout='spam\neggs\n' proc.stderr=''
[capture_output=True via sys.stderr]
proc.stdout='' proc.stderr='spam\neggs\n'
[capture_output=False via sys.stdout]
spam
eggs

bug_reprint_builtin.py -- calling the builtin print after also crashes with an OSError

import subprocess
import sys

code = """\
from rich.console import Console

console = Console(stderr=False)
console.print("spam")
print("eggs")
"""
err_code = code.replace("stderr=False", "stderr=True")
cmd = [sys.executable, "-"]

print("[capture_output=True via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=True via sys.stderr]")
proc = subprocess.run(cmd, input=err_code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")

print("[capture_output=False via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=False, encoding="utf-8")

(venv) PS R:\Programming\diff-shades> python .\bug_reprint_builtin.py
[capture_output=True via sys.stdout]
proc.stdout='spam\n' proc.stderr="Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>\nOSError: [Errno 9] Bad file descriptor\n"
[capture_output=True via sys.stderr]
proc.stdout='eggs\n' proc.stderr='spam\n'
[capture_output=False via sys.stdout]
spam
eggs
(venv) PS R:\Programming\diff-shades>

with rich==11.2.0

[capture_output=True via sys.stdout]
proc.stdout='spam\neggs\n' proc.stderr=''
[capture_output=True via sys.stderr]
proc.stdout='eggs\n' proc.stderr='spam\n'
[capture_output=False via sys.stdout]
spam
eggs

bug_record.py -- and recording the console output still doesn't work

import os
import subprocess
import sys

code = """\
from rich.console import Console

console = Console(stderr=False, record=True)
console.print("spam")
with open("log.txt", "w", encoding="utf-8") as f:
    f.write(console.export_text())
"""
err_code = code.replace("stderr=False", "stderr=True")
cmd = [sys.executable, "-"]

print("[capture_output=True via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")
with open("log.txt", encoding="utf-8") as f:
    print(f"recorded: {f.read()}")

print("[capture_output=True via sys.stderr]")
proc = subprocess.run(cmd, input=err_code, capture_output=True, encoding="utf-8")
print(f"{proc.stdout=} {proc.stderr=}")
with open("log.txt", encoding="utf-8") as f:
    print(f"recorded: {f.read()}")

print("[capture_output=False via sys.stdout]")
proc = subprocess.run(cmd, input=code, capture_output=False, encoding="utf-8")
with open("log.txt", encoding="utf-8") as f:
    print(f"recorded: {f.read()}")

os.unlink("log.txt")

(venv) PS R:\Programming\diff-shades> python .\bug_record.py
[capture_output=True via sys.stdout]
proc.stdout='spam\n' proc.stderr=''
recorded:
[capture_output=True via sys.stderr]
proc.stdout='' proc.stderr='spam\n'
recorded: spam

[capture_output=False via sys.stdout]
spam
recorded: spam

with rich==11.2.0

(venv) PS R:\Programming\diff-shades> python .\bug_record.py
[capture_output=True via sys.stdout]
proc.stdout='spam\n' proc.stderr=''
recorded: spam

[capture_output=True via sys.stderr]
proc.stdout='' proc.stderr='spam\n'
recorded: spam

[capture_output=False via sys.stdout]
spam
recorded: spam

[darrenburns]

@ichard26 Thanks for investigating in such detail, and for the examples. It really helps! I'll take a look into these issues shortly 👍

[github-actions]

Did I solve your problem?

Why not buy the devs a coffee to say thanks?

[ichard26]

Can confirm all reported cases are fixed on master, many thanks! @darrenburns