Operators & observables notebook raises runtime error after upgrading notebook

[viathor]

Description of the issue

Isolated notebook test fails on operators_and_observables.ipynb after upgrading notebook to 6.4.1 (due to security vulnerability).

How to reproduce the issue

Could not reproduce locally, but CI failed on #4468 with the following complaint:

Run check/pytest -n auto -m slow dev_tools/notebooks/isolated_notebook_test.py -k partition-2
  check/pytest -n auto -m slow dev_tools/notebooks/isolated_notebook_test.py -k partition-2
  shell: /usr/bin/bash -e {0}
  env:
    NOTEBOOK_PARTITIONS: 4
    pythonLocation: /opt/hostedtoolcache/Python/3.8.11/x64
============================= test session starts ==============================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/Cirq/Cirq
plugins: asyncio-0.15.1, cov-2.12.1, xdist-2.2.1, forked-1.3.0
gw0 I / gw1 I
gw0 [9] / gw1 [9]

..F......                                                                [100%]
=================================== FAILURES ===================================
_ test_notebooks_against_released_cirq[partition-2-/home/runner/work/Cirq/Cirq/docs/operators_and_observables.ipynb] _
[gw0] linux -- Python 3.8.11 /opt/hostedtoolcache/Python/3.8.11/x64/bin/python

partition = 'partition-2'
notebook_path = '/home/runner/work/Cirq/Cirq/docs/operators_and_observables.ipynb'
cloned_env = <function cloned_env.<locals>.base_env_creator at 0x7fe7b7cc5280>

    @pytest.mark.slow
    @pytest.mark.parametrize(
        "partition, notebook_path",
        _partitioned_test_cases(filter_notebooks(_list_changed_notebooks(), SKIP_NOTEBOOKS)),
    )
    def test_notebooks_against_released_cirq(partition, notebook_path, cloned_env):
        """Tests the notebooks in isolated virtual environments.
    
        In order to speed up the execution of these tests an auxiliary file may be supplied which
        performs substitutions on the notebook to make it faster.
    
        Specifically for a notebook file notebook.ipynb, one can supply a file notebook.tst which
        contains the substitutes.  The substitutions are provide in the form `pattern->replacement`
        where the pattern is what is matched and replaced. While the pattern is compiled as a
        regular expression, it is considered best practice to not use complicated regular expressions.
        Lines in this file that do not have `->` are ignored.
        """
        notebook_file = os.path.basename(notebook_path)
        notebook_rel_dir = os.path.dirname(os.path.relpath(notebook_path, "."))
        out_path = f"out/{notebook_rel_dir}/{notebook_file[:-6]}.out.ipynb"
        notebook_env = cloned_env("isolated_notebook_tests", *PACKAGES)
    
        notebook_file = os.path.basename(notebook_path)
    
        rewritten_notebook_descriptor, rewritten_notebook_path = rewrite_notebook(notebook_path)
    
        cmd = f"""
    mkdir -p out/{notebook_rel_dir}
    cd {notebook_env}
    . ./bin/activate
    pip list
    papermill {rewritten_notebook_path} {os.getcwd()}/{out_path}"""
        stdout, stderr, status = shell_tools.run_shell(
            cmd=cmd,
            log_run_to_stderr=False,
            raise_on_fail=False,
            out=shell_tools.TeeCapture(),
            err=shell_tools.TeeCapture(),
            # important to get rid of PYTHONPATH specifically, which contains
            # the Cirq repo path due to check/pytest
            env={},
        )
    
        if status != 0:
            print(stdout)
            print(stderr)
>           pytest.fail(
                f"Notebook failure: {notebook_file}, please see {out_path} for the output "
                f"notebook (in Github Actions, you can download it from the workflow artifact"
                f" 'notebook-outputs'). \n"
                f"If this is a new failure in this notebook due to a new change, "
                f"that is only available in master for now, consider adding `pip install --pre cirq` "
                f"instead of `pip install cirq` to this notebook, and exclude it from "
                f"dev_tools/notebooks/isolated_notebook_test.py."
            )
E           Failed: Notebook failure: operators_and_observables.ipynb, please see out/docs/operators_and_observables.out.ipynb for the output notebook (in Github Actions, you can download it from the workflow artifact 'notebook-outputs'). 
E           If this is a new failure in this notebook due to a new change, that is only available in master for now, consider adding `pip install --pre cirq` instead of `pip install cirq` to this notebook, and exclude it from dev_tools/notebooks/isolated_notebook_test.py.

dev_tools/notebooks/isolated_notebook_test.py:186: Failed
----------------------------- Captured stdout call -----------------------------
Pytest worker [gw0] is reusing /tmp/cirq-pytest/isolated_notebook_tests for 'isolated_notebook_tests'.
Package             Version
------------------- ---------
ansiwrap            0.8.4
appdirs             1.4.4
argon2-cffi         20.1.0
attrs               21.2.0
backcall            0.2.0
black               21.7b0
bleach              4.1.0
certifi             2021.5.30
cffi                1.14.6
charset-normalizer  2.0.4
click               8.0.1
cycler              0.10.0
decorator           4.4.2
defusedxml          0.7.1
entrypoints         0.3
idna                3.2
ipykernel           5.3.4
ipython             7.22.0
ipython-genutils    0.2.0
ipywidgets          7.6.3
jedi                0.18.0
Jinja2              3.0.1
jsonschema          3.2.0
jupyter             1.0.0
jupyter-client      7.0.1
jupyter-console     6.4.0
jupyter-core        4.7.1
jupyterlab-pygments 0.1.2
jupyterlab-widgets  1.0.0
kiwisolver          1.3.1
MarkupSafe          2.0.1
matplotlib          3.4.3
mistune             0.8.4
mypy-extensions     0.4.3
nbclient            0.5.4
nbconvert           6.1.0
nbformat            5.1.3
nest-asyncio        1.5.1
notebook            6.4.3
numpy               1.21.2
packaging           21.0
pandas              1.3.2
pandocfilters       1.4.3
papermill           2.3.3
parso               0.8.2
pathspec            0.9.0
pexpect             4.8.0
pickleshare         0.7.5
Pillow              8.3.1
pip                 21.2.3
prometheus-client   0.11.0
prompt-toolkit      3.0.20
ptyprocess          0.7.0
pycparser           2.20
Pygments            2.10.0
pyparsing           2.4.7
pyrsistent          0.18.0
python-dateutil     2.8.2
pytz                2021.1
PyYAML              5.4.1
pyzmq               22.2.1
qtconsole           5.1.1
QtPy                1.10.0
regex               2021.8.21
requests            2.26.0
scipy               1.7.1
seaborn             0.11.2
Send2Trash          1.8.0
setuptools          57.4.0
six                 1.16.0
tenacity            8.0.1
terminado           0.11.1
testpath            0.5.0
textwrap3           0.9.2
tomli               1.2.1
tornado             6.1
tqdm                4.62.2
traitlets           5.0.5
urllib3             1.26.6
wcwidth             0.2.5
webencodings        0.5.1
wheel               0.37.0
widgetsnbextension  3.5.1

WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/bin/python -m pip install --upgrade pip' command.
Input Notebook:  /tmp/tmp6kaqskqm.ipynb
Output Notebook: /home/runner/work/Cirq/Cirq/out/docs/operators_and_observables.out.ipynb

Executing:   0%|          | 0/84 [00:00<?, ?cell/s]Executing notebook with kernel: python3

Executing:   1%|          | 1/84 [00:01<02:16,  1.65s/cell]
Executing:   4%|▎         | 3/84 [00:01<00:38,  2.10cell/s]
Executing:   6%|▌         | 5/84 [00:28<09:13,  7.00s/cell]
Executing:   8%|▊         | 7/84 [00:28<05:12,  4.06s/cell]
Executing:  11%|█         | 9/84 [00:28<03:11,  2.56s/cell]
Executing:  13%|█▎        | 11/84 [00:28<02:02,  1.68s/cell]
Executing:  15%|█▌        | 13/84 [00:28<01:21,  1.15s/cell]
Executing:  18%|█▊        | 15/84 [00:28<00:55,  1.25cell/s]
Executing:  20%|██        | 17/84 [00:28<00:38,  1.76cell/s]
Executing:  23%|██▎       | 19/84 [00:29<00:27,  2.40cell/s]
Executing:  25%|██▌       | 21/84 [00:29<00:19,  3.22cell/s]
Executing:  27%|██▋       | 23/84 [00:29<00:14,  4.20cell/s]
Executing:  30%|██▉       | 25/84 [00:29<00:11,  5.36cell/s]
Executing:  32%|███▏      | 27/84 [00:29<00:08,  6.58cell/s]
Executing:  35%|███▍      | 29/84 [00:29<00:07,  7.81cell/s]
Executing:  37%|███▋      | 31/84 [00:29<00:05,  9.04cell/s]
Executing:  39%|███▉      | 33/84 [00:30<00:05, 10.04cell/s]
Executing:  42%|████▏     | 35/84 [00:30<00:04, 10.71cell/s]
Executing:  44%|████▍     | 37/84 [00:30<00:04, 11.20cell/s]
Executing:  46%|████▋     | 39/84 [00:30<00:03, 11.87cell/s]
Executing:  49%|████▉     | 41/84 [00:30<00:03, 12.19cell/s]
Executing:  51%|█████     | 43/84 [00:30<00:03, 12.36cell/s]
Executing:  54%|█████▎    | 45/84 [00:31<00:03, 12.42cell/s]
Executing:  56%|█████▌    | 47/84 [00:31<00:02, 12.66cell/s]
Executing:  58%|█████▊    | 49/84 [00:31<00:02, 12.73cell/s]
Executing:  61%|██████    | 51/84 [00:31<00:02, 12.61cell/s]
Executing:  63%|██████▎   | 53/84 [00:31<00:02, 12.75cell/s]
Executing:  65%|██████▌   | 55/84 [00:31<00:02, 12.86cell/s]
Executing:  68%|██████▊   | 57/84 [00:32<00:02, 11.50cell/s]
Executing:  70%|███████   | 59/84 [00:32<00:02, 11.68cell/s]
Executing:  73%|███████▎  | 61/84 [00:32<00:01, 11.87cell/s]
Executing:  75%|███████▌  | 63/84 [00:32<00:01, 11.46cell/s]
Executing:  77%|███████▋  | 65/84 [00:32<00:01, 10.98cell/s]
Executing:  80%|███████▉  | 67/84 [00:32<00:01, 11.23cell/s]
Executing:  82%|████████▏ | 69/84 [00:33<00:01,  9.70cell/s]
Executing:  85%|████████▍ | 71/84 [00:33<00:01,  9.16cell/s]
Executing:  86%|████████▌ | 72/84 [00:33<00:01,  8.30cell/s]
Executing:  87%|████████▋ | 73/84 [00:33<00:01,  7.72cell/s]
Executing:  88%|████████▊ | 74/84 [00:34<00:02,  4.80cell/s]
Executing:  88%|████████▊ | 74/84 [00:35<00:04,  2.09cell/s]
Traceback (most recent call last):
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
    execute_notebook(
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/papermill/execute.py", line 122, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/papermill/execute.py", line 234, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [27]":
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-7b05cde6de96> in <module>
      3 
      4 # Provide a sampler. See also: collector.collect(...).
----> 5 collector.collect(sampler=cirq.Simulator())
      6 
      7 # Estimate the observable.

/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c/lib/python3.8/site-packages/cirq/work/collector.py in collect(self, sampler, concurrency, max_total_samples)
    132             https://docs.python.org/3/library/asyncio-task.html
    133         """
--> 134         return asyncio.get_event_loop().run_until_complete(
    135             self.collect_async(
    136                 sampler, concurrency=concurrency, max_total_samples=max_total_samples

/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    590         """
    591         self._check_closed()
--> 592         self._check_running()
    593 
    594         new_task = not futures.isfuture(future)

/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/asyncio/base_events.py in _check_running(self)
    550     def _check_running(self):
    551         if self.is_running():
--> 552             raise RuntimeError('This event loop is already running')
    553         if events._get_running_loop() is not None:
    554             raise RuntimeError(

RuntimeError: This event loop is already running


----------------------------- Captured stderr call -----------------------------
run: ('virtualenv-clone', '/tmp/cirq-pytest/isolated_notebook_tests', '/tmp/cirq-pytest/f37662cf-0b2a-4bde-8eae-82cfc3cc1f8c')
=========================== short test summary info ============================
FAILED dev_tools/notebooks/isolated_notebook_test.py::test_notebooks_against_released_cirq[partition-2-/home/runner/work/Cirq/Cirq/docs/operators_and_observables.ipynb]
=================== 1 failed, 8 passed in 222.43s (0:03:42) ====================

Cirq version

0.13.0.dev

[maffoo]

This looks like it was caused by #4009. I removed the nest-asyncio import because it's no longer needed by duet, but if the notebook is running with an older version of cirq before that change, e.g. the last release version, then we still need to install nest-asyncio. I will put up a PR to fix.