tenacity >= 8.4 breaks unit tests

[tonyandrewmeyer]

Steps to reproduce

1. Clone repo.
2. poetry lock
3. tox

Expected behavior

All tests pass.

Actual behavior

============================================================================= FAILURES =============================================================================
__________________________________________________ TestMySQL.test_wait_until_unit_removed_from_cluster_exception ___________________________________________________
Traceback (most recent call last):
  File "/home/tameyer/code/mysql-k8s-operator/.tox/unit/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/src/mysql_k8s_helpers.py", line 367, in _wait_until_unit_removed_from_cluster
    raise MySQLWaitUntilUnitRemovedFromClusterError("Remove member still in cluster")
mysql_k8s_helpers.MySQLWaitUntilUnitRemovedFromClusterError: Remove member still in cluster

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.12/unittest/case.py", line 634, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/usr/lib/python3.12/unittest/mock.py", line 1390, in patched
    return func(*newargs, **newkeywargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/tests/unit/test_mysql_k8s_helpers.py", line 293, in test_wait_until_unit_removed_from_cluster_exception
    self.mysql._wait_until_unit_removed_from_cluster("mysql-0.mysql-endpoints")
  File "/home/tameyer/code/mysql-k8s-operator/lib/charms/tempo_k8s/v1/charm_tracing.py", line 544, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/.tox/unit/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/.tox/unit/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/.tox/unit/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/tameyer/code/mysql-k8s-operator/.tox/unit/lib/python3.12/site-packages/tenacity/__init__.py", line 419, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x711d3bd159d0 state=finished raised MySQLWaitUntilUnitRemovedFromClusterError>]
[...]
======================================================= 1 failed, 47 passed, 2 warnings in 630.98s (0:10:30) =======================================================
unit: exit 1 (632.06 seconds) /home/tameyer/code/mysql-k8s-operator> poetry run coverage run --source=/home/tameyer/code/mysql-k8s-operator/src -m pytest -v --tb native -s /home/tameyer/code/mysql-k8s-operator/tests/unit pid=3305646
  lint: OK (4.00=setup[0.08]+cmd[1.55,0.27,0.35,0.59,0.38,0.51,0.27] seconds)
  unit: FAIL code 1 (634.52=setup[0.00]+cmd[2.46,632.06] seconds)
  evaluation failed :( (638.55 seconds)

Versions

Operating system: Ubuntu 24.04 LTS

Juju CLI: N/A

Juju agent: N/A

Charm revision: 2b48058

microk8s: N/A

Log output

Juju debug log: N/A

Additional context

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4741

[taurus-forever]

Thank you for the report! I have assigned reviewers to your PR. tnx!

[carlcsaposs-canonical]

thanks for the report & for the early notice about issues in our tests with the newer version of tenacity!

@tonyandrewmeyer is running poetry lock a regular practice? my uninformed expectation is that users of our charm would use the dependencies that we locked & not update the dependencies themselves

[carlcsaposs-canonical]

@paulomach @shayancanonical

this fixture to disable tenacity in router unit tests might be of use:
https://github.com/canonical/mysql-router-k8s-operator/blob/e7225415aecd6eb6eda4bc1811b546e645abe474/tests/unit/conftest.py#L8-L25

although not sure if this issue exclusively affects unit tests (maybe it affects real deployments?)

[tonyandrewmeyer]

@tonyandrewmeyer is running poetry lock a regular practice? my uninformed expectation is that users of our charm would use the dependencies that we locked & not update the dependencies themselves

Right now, ops does it in our CI: we patch in the latest main version of ops and run your tests (also a handful of other charms) to try to warn us if we inadvertently break something. I also do the same in the tool I use offline to run the tests of ~140 charms when we're wanting to evaluate how changes impact charms.

However, I think we are doing this the wrong way, and so opened an ops issue (canonical/operator#1272) to improve things on our side. So I hope that shortly we will not actually be doing this regularly.

[carlcsaposs-canonical]

I'd suggest using poetry lock --no-update and I think you'll get the behavior you're looking for

[paulomach]

@paulomach @shayancanonical

this fixture to disable tenacity in router unit tests might be of use: https://github.com/canonical/mysql-router-k8s-operator/blob/e7225415aecd6eb6eda4bc1811b546e645abe474/tests/unit/conftest.py#L8-L25

although not sure if this issue exclusively affects unit tests (maybe it affects real deployments?)

We should patch tenacity on all unit tests. Unfortunately we did not yet migrated to pytest to use the fixture, but patching in unittest is also possible

[paulomach]

@tonyandrewmeyer , checking this it no longer seems to be the topical. I'm closing. Feel free to reopen it.