Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround strange resolution bug in uv with opentelemetry downgrades #43768

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Nov 6, 2024

Apparently uv 0.4.30 has some strange resolution bug that results in inconsistent dependencies in opentelemetry that are detected by pip check when we are running eager upgrade. This is described in detail in astral-sh/uv#8871 . Until the issue is diagnosed and Solved, we workaround it by providing a hint to uv to use higher version of opentelemetry-proto than it wants to downgrade to - for no apparent reason.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk requested a review from ashb as a code owner November 6, 2024 23:57
@potiuk potiuk requested review from ephraimbuddy, ferruzzi, gopidesupavan, dirrao, eladkal and amoghrajesh and removed request for ashb November 6, 2024 23:57
Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this locally and it does allow breeze ci-image build to complete successfully. 👍

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

Tested this locally and it does allow breeze ci-image build to complete successfully. 👍

Still - it fails when trying to generate constraints using pypi packages.... because it cannot build `apache-beam

    × Failed to download and build `apache-beam==2.0.0`

Which is even stranger!. I will have to workaround it a bit more - and add a beam limitation as well it seems. I think that one will be a head-scratcher for the uv team and @notatallshaw :)

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

From: https://github.com/apache/airflow/actions/runs/11713890095/job/32627785553?pr=43768#step:12:545

Running command: uv pip install --python /usr/local/bin/python3 '.[all-core]' apache-airflow-providers-airbyte apache-airflow-providers-alibaba 'apache-airflow-providers-amazon @ [...]
  Running command: uv pip install --python /usr/local/bin/python3 '.[all-core]' apache-airflow-providers-airbyte apache-airflow-providers-alibaba 'apache-airflow-providers-amazon @ file:///dist/apache_airflow_providers_amazon-9.1.0.dev0-py3-none-any.whl' apache-airflow-providers-apache-beam apache-airflow-providers-apache-cassandra apache-airflow-providers-apache-drill 
  apache-airflow-providers-apache-druid apache-airflow-providers-apache-flink apache-airflow-providers-apache-hdfs apache-airflow-providers-apache-hive apache-airflow-providers-apache-iceberg apache-airflow-providers-apache-impala apache-airflow-providers-apache-kafka apache-airflow-providers-apache-kylin apache-airflow-providers-apache-livy apache-airflow-providers-apache-pig 
  apache-airflow-providers-apache-pinot apache-airflow-providers-apache-spark apache-airflow-providers-apprise apache-airflow-providers-arangodb apache-airflow-providers-asana apache-airflow-providers-atlassian-jira apache-airflow-providers-celery apache-airflow-providers-cncf-kubernetes apache-airflow-providers-cohere apache-airflow-providers-common-compat apache-airflow-providers-common-io 
  apache-airflow-providers-common-sql apache-airflow-providers-databricks apache-airflow-providers-datadog apache-airflow-providers-dbt-cloud apache-airflow-providers-dingding apache-airflow-providers-discord apache-airflow-providers-docker apache-airflow-providers-elasticsearch apache-airflow-providers-exasol apache-airflow-providers-fab apache-airflow-providers-facebook 
  apache-airflow-providers-ftp apache-airflow-providers-github apache-airflow-providers-google apache-airflow-providers-grpc apache-airflow-providers-hashicorp apache-airflow-providers-http apache-airflow-providers-imap apache-airflow-providers-influxdb apache-airflow-providers-jdbc apache-airflow-providers-jenkins apache-airflow-providers-microsoft-azure apache-airflow-providers-microsoft-mssql 
  apache-airflow-providers-microsoft-psrp apache-airflow-providers-microsoft-winrm apache-airflow-providers-mongo apache-airflow-providers-mysql apache-airflow-providers-neo4j apache-airflow-providers-odbc apache-airflow-providers-openai apache-airflow-providers-openfaas apache-airflow-providers-openlineage apache-airflow-providers-opensearch apache-airflow-providers-opsgenie 
  apache-airflow-providers-oracle apache-airflow-providers-pagerduty apache-airflow-providers-papermill apache-airflow-providers-pgvector apache-airflow-providers-pinecone apache-airflow-providers-postgres apache-airflow-providers-presto apache-airflow-providers-qdrant apache-airflow-providers-redis apache-airflow-providers-salesforce apache-airflow-providers-samba apache-airflow-providers-segment 
  apache-airflow-providers-sendgrid apache-airflow-providers-sftp apache-airflow-providers-singularity apache-airflow-providers-slack apache-airflow-providers-smtp apache-airflow-providers-snowflake apache-airflow-providers-sqlite apache-airflow-providers-ssh 'apache-airflow-providers-standard @ file:///dist/apache_airflow_providers_standard-0.0.1.dev0-py3-none-any.whl' 
  apache-airflow-providers-tableau apache-airflow-providers-telegram apache-airflow-providers-teradata apache-airflow-providers-trino apache-airflow-providers-vertica apache-airflow-providers-weaviate apache-airflow-providers-yandex apache-airflow-providers-ydb apache-airflow-providers-zendesk 'opentelemetry-proto>=1.27.0' --resolution highest
  Using Python 3.9.20 environment at /usr/local
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-common-compat`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-common-io`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-common-sql`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-ftp`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-http`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-imap`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-smtp`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-sqlite`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-standard`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-airbyte`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-alibaba`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-amazon`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-beam`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-cassandra`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-drill`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-druid`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-flink`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-hdfs`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-hive`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-iceberg`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-impala`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-kafka`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-kylin`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-livy`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-pig`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-pinot`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apache-spark`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-apprise`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-arangodb`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-asana`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-atlassian-jira`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-celery`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-cloudant`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-cncf-kubernetes`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-cohere`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-databricks`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-datadog`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-dbt-cloud`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-dingding`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-discord`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-docker`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-elasticsearch`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-exasol`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-fab`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-facebook`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-github`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-google`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-grpc`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-hashicorp`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-influxdb`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-jdbc`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-jenkins`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-microsoft-azure`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-microsoft-mssql`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-microsoft-psrp`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-microsoft-winrm`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-mongo`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-mysql`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-neo4j`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-odbc`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-openai`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-openfaas`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-openlineage`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-opensearch`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-opsgenie`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-oracle`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-pagerduty`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-papermill`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-pgvector`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-pinecone`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-postgres`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-presto`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-qdrant`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-redis`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-salesforce`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-samba`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-segment`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-sendgrid`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-sftp`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-singularity`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-slack`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-snowflake`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-ssh`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-tableau`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-telegram`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-teradata`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-trino`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-vertica`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-weaviate`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-yandex`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-ydb`
  warning: Missing version constraint (e.g., a lower bound) for `apache-airflow-providers-zendesk`
    × Failed to download and build `apache-beam==2.0.0`
    ╰─▶ Build backend failed to determine requirements with `build_wheel()`
        (exit status: 1)
  
        [stderr]
        <string>:29: DeprecationWarning: pkg_resources is deprecated as an API.
        See https://setuptools.pypa.io/en/latest/pkg_resources.html
        Traceback (most recent call last):
          File "<string>", line 14, in <module>
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/setuptools/build_meta.py",
        line 333, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=[])
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/setuptools/build_meta.py",
        line 303, in _get_build_requires
            self.run_setup()
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/setuptools/build_meta.py",
        line 521, in run_setup
            super().run_setup(setup_script=setup_script)
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/setuptools/build_meta.py",
        line 319, in run_setup
            exec(code, locals())
          File "<string>", line 53, in <module>
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/pkg_resources/__init__.py",
        line 534, in get_distribution
            dist = get_provider(dist)
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/pkg_resources/__init__.py",
        line 417, in get_provider
            return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/pkg_resources/__init__.py",
        line 1070, in require
            needed = self.resolve(parse_requirements(requirements))
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/pkg_resources/__init__.py",
        line 897, in resolve
            dist = self._resolve_dist(
          File
        "/tmp/.tmpSGooeS/builds-v0/.tmpWagsma/lib/python3.9/site-packages/pkg_resources/__init__.py",
        line 938, in _resolve_dist
            raise DistributionNotFound(req, requirers)
        pkg_resources.DistributionNotFound: The 'pip' distribution was not found
        and is required by the application

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

We have apache-beam==2.60.0 - but uv tries to download 2.0.0 ????

@ferruzzi
Copy link
Contributor

ferruzzi commented Nov 7, 2024

Still - it fails when trying to generate constraints using pypi packages.... because it cannot build `apache-beam

What was the exact command that ran into that? I ran breeze ci-image build --python 3.9 --upgrade-to-newer-dependencies to test it; that failed before the change and worked with this change.

Apparently `uv 0.4.30` has some strange resolution bug that results in
inconsistent dependencies in opentelemetry that are detected by `pip
check` when we are running eager upgrade. This is described in detail in
astral-sh/uv#8871 . Until the issue is
diagnosed and Solved, we workaround it by providing a hint to `uv` to
use higher version of opentelemetry-proto than it wants to downgrade to
- for no apparent reason.
@potiuk potiuk force-pushed the workaround-uv-bug-with-conflicting-resolution branch from a18fd3d to 3afd9d2 Compare November 7, 2024 00:26
@notatallshaw
Copy link
Contributor

We have apache-beam==2.60.0 - but uv tries to download 2.0.0 ????

I'm surprised you don't see this more often: astral-sh/uv#3078

Btw, I have some big optimizations coming to pip and I see the same issue when pip is much more focused on backtracking causes.

I strongly recommend Airflow puts a reasonable minimal bounds on it's providers. Or at least for apache-beam provider.

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

I strongly recommend Airflow puts a reasonable minimal bounds on it's providers. Or at least for apache-beam provider.

We do apache-beam>=2.53.0

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

I strongly recommend Airflow puts a reasonable minimal bounds on it's providers. Or at least for apache-beam provider.

And yes this is what we actively do for quite some time - including very recent #42989 ... so ... I am just surprised it happened, but I am solving this isssue in another way (as discussed in astral-sh/uv#8871 - by removing caching from the picture). So that one with beam == 2.0.0 might be an issue for another day :D.

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

Closing that one. The discussion with @notatallshaw and @charliermarsh on astral-sh/uv#8871 - made me realise this is not an issue with uv and resolution but. ... cache invalidation issue.

@potiuk
Copy link
Member Author

potiuk commented Nov 7, 2024

Closing in favour of #43770

@potiuk potiuk closed this Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants