Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistent python version checks and troubleshooting #42944

Merged
merged 1 commit into from
Oct 11, 2024

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Oct 11, 2024

Follow up after #42766 and #42936.

  • We do not have to check for minimum Python version for Python 3.9 any more (as we do not support 3.8 any more)
  • While Python 3.12 is not yet fully supported by Apache Beam, we should still not allow it for releasing providers, but all other commands should support Python 3.12
  • When you had pre-commit installed with Python 3.8 before, various errors might appear when running pre-commit. Troubleshooting was added quoting the errors and explaining what to do.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
@potiuk potiuk merged commit 92a82eb into apache:main Oct 11, 2024
86 checks passed
@potiuk potiuk deleted the correct-python-version-check branch October 11, 2024 19:22
@ashb
Copy link
Member

ashb commented Oct 12, 2024

I still don't understand how preparing the sdist release on Python 3.12 causes any problem? Surely the Py 3.12 issue is a runtime/install time problem?

For instance: I removed the 3.12 check and ran:

breeze release-management prepare-provider-packages --package-format sdist apache.beam --skip-tag-check

Which generated this file apache_airflow_providers_apache_beam-5.8.1.tar.gz

If you compare the files in that to https://dist.apache.org/repos/dist/dev/airflow/providers/apache_airflow_providers_apache_beam-5.8.1.tar.gz it is 100% identical

To repro:

curl -fSsL https://dist.apache.org/repos/dist/dev/airflow/providers/apache_airflow_providers_apache_beam-5.8.1.tar.gz | tar -xOz - | shasum;
curl -fSsL https://github.com/user-attachments/files/17350787/apache_airflow_providers_apache_beam-5.8.1.tar.gz | tar -xOz - | shasum;

Which prints

c160bea03fb1262af731ba25fb3c1d9a2a8a954c  -
c160bea03fb1262af731ba25fb3c1d9a2a8a954c  -

tar -O says print all the content to stdout -- i.e. these two tar files contain identical content.

@potiuk
Copy link
Member Author

potiuk commented Oct 12, 2024

Good that you checked. The sdist package to wheel conversion did not work before with Python3.12.

As explained few times before when you asked -this was the reason for having the exclusion. I have not checked if it works now it could be caused by other packages that failed with 3.12.

Another reason why it might start working -I think - the version of Beam released a week or so ago could have fixed the sdist failure - because they generally fixed some dependencies in latest version released while I was on vacations.

If my hypothesis is right - If you want to see how it failed before you can pin the beam package to the previous version and see the failure.

If you can confirm that it works now - we can remove the limitation.

@ashb
Copy link
Member

ashb commented Oct 12, 2024

That def explains why I couldn't see any reason for the check. I'll try with the previous version tomorrow/Monday.

kunaljubce pushed a commit to kunaljubce/airflow that referenced this pull request Oct 13, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
pavansharma36 pushed a commit to pavansharma36/airflow that referenced this pull request Oct 14, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
R7L208 pushed a commit to R7L208/airflow that referenced this pull request Oct 17, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants