Deadlock when exiting Progress context

[eulertour]

I can consistently reproduce this by letting the following script run. Eventually the progress bars will hang when they're nearly full, like so:

from rich.progress import track
from rich.progress import Progress
import random
import time

while True:
    with Progress() as progress:
        task1 = progress.add_task(f"[red]Downloading...", total=10)
        task2 = progress.add_task("[green]Processing...", total=10)
        task3 = progress.add_task("[cyan]Cooking...", total=10)

        while not progress.finished:
            progress.update(task1, advance=random.random())
            progress.update(task2, advance=random.random())
            progress.update(task3, advance=random.random())
            time.sleep(0.1)

I believe the problematic sequence of events is as follows:

1. The main thread exits from a Progress context and acquires Progress._lock in Progress.__exit__()
   https://github.com/willmcgugan/rich/blob/55988c6e721cc03ab4f6dc6aeca3089e7c5c13ae/rich/progress.py#L447-L451

2. The refresh thread finishes waiting on _RefreshThread.done and calls Progress.refresh(), at which point it starts waiting for Progress._lock
   https://github.com/willmcgugan/rich/blob/55988c6e721cc03ab4f6dc6aeca3089e7c5c13ae/rich/progress.py#L346-L348
   https://github.com/willmcgugan/rich/blob/55988c6e721cc03ab4f6dc6aeca3089e7c5c13ae/rich/progress.py#L574-L582

3. The main thread calls Progress._refresh_thread.stop() in Progress.stop(), at which point it waits for the refresh thread to finish with _RefreshThread.join()
   https://github.com/willmcgugan/rich/blob/55988c6e721cc03ab4f6dc6aeca3089e7c5c13ae/rich/progress.py#L422-L436
   https://github.com/willmcgugan/rich/blob/55988c6e721cc03ab4f6dc6aeca3089e7c5c13ae/rich/progress.py#L342-L344

At this point the main thread is holding Progress._lock and waiting on the refresh thread to finish while the refresh thread is waiting to acquire Progress._lock so it can finish.

I don't think it's safe to call _RefreshThread.join() while holding Progress._lock, because there's always a chance that the refresh thread started waiting on Progress._lock the nanosecond after it was acquired.

[willmcgugan]

Thanks for the analysis! Saved a lot of legwork.

You're exactly right about the problem, but I think the solution may be to ensure that the refresh thread join is never done inside the progress._lock.

Also the code I had to make the context manager re-entrant just complicated things. I've made the enter and exit simply call start and stop.

If you wouldn't mind, have a look at #92 It seems to fix the deadlock.

[willmcgugan]

That's in master now. Thanks for the help!

[eulertour]

Do you have any estimate as to when the next pypi release will be?

[willmcgugan]

Released v1.2.3 just recently