diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 5df89e4fc7b4d..83a5843860cf6 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -402,6 +402,19 @@ repos: entry: "pydevd.*settrace\\(" pass_filenames: true files: \.py$ + - id: check-links-to-example-dags-do-not-use-hardcoded-versions + name: Check that example dags do not use hard-coded version numbers + description: The links to example dags should use |version| as version specification + language: pygrep + entry: > + (?i) + .*https://github.*[0-9]/tests/system/providers| + .*https://github.*/main/tests/system/providers| + .*https://github.*/master/tests/system/providers| + .*https://github.*/main/airflow/providers/.*/example_dags/| + .*https://github.*/master/airflow/providers/.*/example_dags/ + pass_filenames: true + files: ^docs/apache-airflow-providers-.*\.rst - id: check-safe-filter-usage-in-html language: pygrep name: Don't use safe in templates diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst index 7495044f3dcd8..14db6af5ceeed 100644 --- a/STATIC_CODE_CHECKS.rst +++ b/STATIC_CODE_CHECKS.rst @@ -129,208 +129,210 @@ require Breeze Docker image to be build locally. .. BEGIN AUTO-GENERATED STATIC CHECK LIST -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| ID | Description | Image | -+========================================================+==================================================================+=========+ -| black | Run black (python formatter) | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| blacken-docs | Run black on python code blocks in documentation files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-airflow-config-yaml-consistent | Checks for consistency between config.yml and default_config.cfg | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-airflow-provider-compatibility | Check compatibility of Providers with Airflow | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-apache-license-rat | Check if licenses are OK for Apache | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-base-operator-usage | * Check BaseOperator[Link] core imports | | -| | * Check BaseOperator[Link] other imports | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-breeze-top-dependencies-limited | Breeze should have small number of top-level dependencies | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-builtin-literals | Require literal syntax when initializing Python builtin types | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-changelog-has-no-duplicates | Check changelogs for duplicate entries | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-core-deprecation-classes | Verify using of dedicated Airflow deprecation classes in core | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-daysago-import-from-utils | Make sure days_ago is imported from airflow.utils.dates | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-decorated-operator-implements-custom-name | Check @task decorator implements custom_operator_name | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-docstring-param-types | Check that docstrings do not specify param types | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-example-dags-urls | Check that example dags url include provider versions | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-executables-have-shebangs | Check that executables have shebang | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-extra-packages-references | Checks setup extra packages | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-extras-order | Check order of extras in Dockerfile | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-for-inclusive-language | Check for language that we do not accept as community | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-hooks-apply | Check if all hooks apply to the repository | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-incorrect-use-of-LoggingMixin | Make sure LoggingMixin is not used alone | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-init-decorator-arguments | Check model __init__ and decorator arguments are in sync | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-lazy-logging | Check that all logging methods are lazy | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-merge-conflict | Check that merge conflicts are not being committed | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-newsfragments-are-valid | Check newsfragments are valid | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-no-providers-in-core-examples | No providers imports in core example DAGs | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-no-relative-imports | No relative imports | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-persist-credentials-disabled-in-github-workflows | Check that workflow files have persist-credentials disabled | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-pre-commit-information-consistent | Update information re pre-commit hooks and verify ids and names | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-provide-create-sessions-imports | Check provide_session and create_session imports | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-provider-yaml-valid | Validate providers.yaml files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-providers-init-file-missing | Provider init file is missing | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-providers-subpackages-init-file-exist | Provider subpackage init files are there | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-revision-heads-map | Check that the REVISION_HEADS_MAP is up-to-date | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-safe-filter-usage-in-html | Don't use safe in templates | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-setup-order | Check order of dependencies in setup.cfg and setup.py | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-start-date-not-used-in-defaults | 'start_date' not to be defined in default_args in example_dags | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-system-tests-present | Check if system tests have required segments of code | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-system-tests-tocs | Check that system tests is properly added | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| check-xml | Check XML files with xmllint | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| codespell | Run codespell to check for common misspellings in files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| compile-www-assets | Compile www assets | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| compile-www-assets-dev | Compile www assets in dev mode | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| create-missing-init-py-files-tests | Create missing init.py files in tests | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| debug-statements | Detect accidentally committed debug statements | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| detect-private-key | Detect if private key is added to the repository | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| doctoc | Add TOC for md and rst files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| end-of-file-fixer | Make sure that there is an empty line at the end | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| fix-encoding-pragma | Remove encoding header from python files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| flynt | Run flynt string format converter for Python | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| identity | Print input to the static check hooks for troubleshooting | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| insert-license | * Add license for all SQL files | | -| | * Add license for all rst files | | -| | * Add license for all CSS/JS/PUML/TS/TSX files | | -| | * Add license for all JINJA template files | | -| | * Add license for all shell files | | -| | * Add license for all Python files | | -| | * Add license for all XML files | | -| | * Add license for all YAML files | | -| | * Add license for all md files | | -| | * Add license for all other files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| isort | Run isort to sort imports in Python files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-chart-schema | Lint chart/values.schema.json file | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-css | stylelint | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-dockerfile | Lint dockerfile | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-helm-chart | Lint Helm Chart | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-json-schema | * Lint JSON Schema files with JSON Schema | | -| | * Lint NodePort Service with JSON Schema | | -| | * Lint Docker compose files with JSON Schema | | -| | * Lint chart/values.schema.json file with JSON Schema | | -| | * Lint chart/values.yaml file with JSON Schema | | -| | * Lint airflow/config_templates/config.yml file with JSON Schema | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-markdown | Run markdownlint | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| lint-openapi | * Lint OpenAPI using spectral | | -| | * Lint OpenAPI using openapi-spec-validator | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| pretty-format-json | Format json files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| pydocstyle | Run pydocstyle | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| python-no-log-warn | Check if there are no deprecate log warn | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| pyupgrade | Upgrade Python code automatically | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| replace-bad-characters | Replace bad characters | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| rst-backticks | Check if RST files use double backticks for code | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| run-flake8 | Run flake8 | * | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| run-mypy | * Run mypy for dev | * | -| | * Run mypy for core | | -| | * Run mypy for providers | | -| | * Run mypy for /docs/ folder | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| run-shellcheck | Check Shell scripts syntax correctness | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| static-check-autoflake | Remove all unused code | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| trailing-whitespace | Remove trailing whitespace at end of line | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| ts-compile-and-lint-javascript | TS types generation and ESLint against current UI files | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-breeze-cmd-output | Update output of breeze commands in BREEZE.rst | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-breeze-readme-config-hash | Update Breeze README.md with config files hash | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-er-diagram | Update ER diagram | * | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-extras | Update extras in documentation | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-in-the-wild-to-be-sorted | Sort INTHEWILD.md alphabetically | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-inlined-dockerfile-scripts | Inline Dockerfile and Dockerfile.ci scripts | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-local-yml-file | Update mounts in the local yml file | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-migration-references | Update migration ref doc | * | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-providers-dependencies | Update cross-dependencies for providers packages | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-supported-versions | Updates supported versions in documentation | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-vendored-in-k8s-json-schema | Vendor k8s definitions into values.schema.json | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| update-version | Update version to the latest version in the documentation | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| yamllint | Check YAML files with yamllint | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ -| yesqa | Remove unnecessary noqa statements | | -+--------------------------------------------------------+------------------------------------------------------------------+---------+ ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| ID | Description | Image | ++===========================================================+==================================================================+=========+ +| black | Run black (python formatter) | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| blacken-docs | Run black on python code blocks in documentation files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-airflow-config-yaml-consistent | Checks for consistency between config.yml and default_config.cfg | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-airflow-provider-compatibility | Check compatibility of Providers with Airflow | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-apache-license-rat | Check if licenses are OK for Apache | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-base-operator-usage | * Check BaseOperator[Link] core imports | | +| | * Check BaseOperator[Link] other imports | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-breeze-top-dependencies-limited | Breeze should have small number of top-level dependencies | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-builtin-literals | Require literal syntax when initializing Python builtin types | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-changelog-has-no-duplicates | Check changelogs for duplicate entries | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-core-deprecation-classes | Verify using of dedicated Airflow deprecation classes in core | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-daysago-import-from-utils | Make sure days_ago is imported from airflow.utils.dates | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-decorated-operator-implements-custom-name | Check @task decorator implements custom_operator_name | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-docstring-param-types | Check that docstrings do not specify param types | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-example-dags-urls | Check that example dags url include provider versions | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-executables-have-shebangs | Check that executables have shebang | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-extra-packages-references | Checks setup extra packages | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-extras-order | Check order of extras in Dockerfile | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-for-inclusive-language | Check for language that we do not accept as community | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-hooks-apply | Check if all hooks apply to the repository | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-incorrect-use-of-LoggingMixin | Make sure LoggingMixin is not used alone | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-init-decorator-arguments | Check model __init__ and decorator arguments are in sync | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-lazy-logging | Check that all logging methods are lazy | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-links-to-example-dags-do-not-use-hardcoded-versions | Check that example dags do not use hard-coded version numbers | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-merge-conflict | Check that merge conflicts are not being committed | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-newsfragments-are-valid | Check newsfragments are valid | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-no-providers-in-core-examples | No providers imports in core example DAGs | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-no-relative-imports | No relative imports | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-persist-credentials-disabled-in-github-workflows | Check that workflow files have persist-credentials disabled | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-pre-commit-information-consistent | Update information re pre-commit hooks and verify ids and names | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-provide-create-sessions-imports | Check provide_session and create_session imports | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-provider-yaml-valid | Validate providers.yaml files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-providers-init-file-missing | Provider init file is missing | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-providers-subpackages-init-file-exist | Provider subpackage init files are there | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-revision-heads-map | Check that the REVISION_HEADS_MAP is up-to-date | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-safe-filter-usage-in-html | Don't use safe in templates | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-setup-order | Check order of dependencies in setup.cfg and setup.py | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-start-date-not-used-in-defaults | 'start_date' not to be defined in default_args in example_dags | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-system-tests-present | Check if system tests have required segments of code | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-system-tests-tocs | Check that system tests is properly added | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| check-xml | Check XML files with xmllint | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| codespell | Run codespell to check for common misspellings in files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| compile-www-assets | Compile www assets | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| compile-www-assets-dev | Compile www assets in dev mode | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| create-missing-init-py-files-tests | Create missing init.py files in tests | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| debug-statements | Detect accidentally committed debug statements | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| detect-private-key | Detect if private key is added to the repository | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| doctoc | Add TOC for md and rst files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| end-of-file-fixer | Make sure that there is an empty line at the end | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| fix-encoding-pragma | Remove encoding header from python files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| flynt | Run flynt string format converter for Python | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| identity | Print input to the static check hooks for troubleshooting | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| insert-license | * Add license for all SQL files | | +| | * Add license for all rst files | | +| | * Add license for all CSS/JS/PUML/TS/TSX files | | +| | * Add license for all JINJA template files | | +| | * Add license for all shell files | | +| | * Add license for all Python files | | +| | * Add license for all XML files | | +| | * Add license for all YAML files | | +| | * Add license for all md files | | +| | * Add license for all other files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| isort | Run isort to sort imports in Python files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-chart-schema | Lint chart/values.schema.json file | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-css | stylelint | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-dockerfile | Lint dockerfile | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-helm-chart | Lint Helm Chart | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-json-schema | * Lint JSON Schema files with JSON Schema | | +| | * Lint NodePort Service with JSON Schema | | +| | * Lint Docker compose files with JSON Schema | | +| | * Lint chart/values.schema.json file with JSON Schema | | +| | * Lint chart/values.yaml file with JSON Schema | | +| | * Lint airflow/config_templates/config.yml file with JSON Schema | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-markdown | Run markdownlint | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| lint-openapi | * Lint OpenAPI using spectral | | +| | * Lint OpenAPI using openapi-spec-validator | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| pretty-format-json | Format json files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| pydocstyle | Run pydocstyle | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| python-no-log-warn | Check if there are no deprecate log warn | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| pyupgrade | Upgrade Python code automatically | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| replace-bad-characters | Replace bad characters | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| rst-backticks | Check if RST files use double backticks for code | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| run-flake8 | Run flake8 | * | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| run-mypy | * Run mypy for dev | * | +| | * Run mypy for core | | +| | * Run mypy for providers | | +| | * Run mypy for /docs/ folder | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| run-shellcheck | Check Shell scripts syntax correctness | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| static-check-autoflake | Remove all unused code | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| trailing-whitespace | Remove trailing whitespace at end of line | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| ts-compile-and-lint-javascript | TS types generation and ESLint against current UI files | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-breeze-cmd-output | Update output of breeze commands in BREEZE.rst | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-breeze-readme-config-hash | Update Breeze README.md with config files hash | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-er-diagram | Update ER diagram | * | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-extras | Update extras in documentation | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-in-the-wild-to-be-sorted | Sort INTHEWILD.md alphabetically | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-inlined-dockerfile-scripts | Inline Dockerfile and Dockerfile.ci scripts | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-local-yml-file | Update mounts in the local yml file | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-migration-references | Update migration ref doc | * | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-providers-dependencies | Update cross-dependencies for providers packages | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-supported-versions | Updates supported versions in documentation | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-vendored-in-k8s-json-schema | Vendor k8s definitions into values.schema.json | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| update-version | Update version to the latest version in the documentation | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| yamllint | Check YAML files with yamllint | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ +| yesqa | Remove unnecessary noqa statements | | ++-----------------------------------------------------------+------------------------------------------------------------------+---------+ .. END AUTO-GENERATED STATIC CHECK LIST diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py index b64d92b5c4cf7..4914cb09148cd 100644 --- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py +++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py @@ -47,6 +47,7 @@ "check-incorrect-use-of-LoggingMixin", "check-init-decorator-arguments", "check-lazy-logging", + "check-links-to-example-dags-do-not-use-hardcoded-versions", "check-merge-conflict", "check-newsfragments-are-valid", "check-no-providers-in-core-examples", diff --git a/dev/example_dags/update_example_dags_paths.py b/dev/example_dags/update_example_dags_paths.py index 08251cd213d8d..36eea37158053 100755 --- a/dev/example_dags/update_example_dags_paths.py +++ b/dev/example_dags/update_example_dags_paths.py @@ -40,14 +40,14 @@ EXAMPLE_DAGS_URL_MATCHER = re.compile( r"^(.*)(https://github.com/apache/airflow/tree/(.*)/airflow/providers/(.*)/example_dags)(/?\".*)$" ) - SYSTEM_TESTS_URL_MATCHER = re.compile( r"^(.*)(https://github.com/apache/airflow/tree/(.*)/tests/system/providers/(.*))(/?\".*)$" ) def check_if_url_exists(url: str) -> bool: # type: ignore[return] - response = requests.head(url) + return True # uncomment to check URLs + response = requests.head(url, allow_redirects=True) if response.status_code == 200: return True if response.status_code == 404: @@ -57,12 +57,12 @@ def check_if_url_exists(url: str) -> bool: # type: ignore[return] def replace_match(file: str, line: str, provider: str, version: str) -> str | None: - for matcher in [EXAMPLE_DAGS_URL_MATCHER, SYSTEM_TESTS_URL_MATCHER]: + for index, matcher in enumerate([EXAMPLE_DAGS_URL_MATCHER, SYSTEM_TESTS_URL_MATCHER]): match = matcher.match(line) if match: url_path_to_dir = match.group(4) branch = match.group(3) - if branch.startswith("providers-"): + if branch.startswith("providers-") and branch.endswith(f"/{version}"): console.print(f"[green]Already corrected[/]: {provider}:{version}") continue system_tests_url = ( @@ -73,17 +73,17 @@ def replace_match(file: str, line: str, provider: str, version: str) -> str | No f"https://github.com/apache/airflow/tree/providers-{provider}/{version}" f"/airflow/providers/{url_path_to_dir}/example_dags" ) - if check_if_url_exists(system_tests_url): + if check_if_url_exists(system_tests_url) and index == 1: new_line = re.sub(matcher, r"\1" + system_tests_url + r"\5", line) - elif check_if_url_exists(example_dags_url): + elif check_if_url_exists(example_dags_url) and index == 0: new_line = re.sub(matcher, r"\1" + example_dags_url + r"\5", line) else: console.print( f"[yellow] Neither example dags nor system tests folder" - f" exists for {provider}:{version} -> removing:[/]" + f" exists for {provider}:{version} -> skipping:[/]" ) console.print(line) - return None + return line if line != new_line: console.print(f"[yellow] Replacing in {file}[/]\n{line.strip()}\n{new_line.strip()}") return new_line @@ -101,8 +101,8 @@ def find_matches(_file: Path, provider: str, version: str): if __name__ == "__main__": - curdir = Path(os.curdir).resolve() - dirs = list(filter(os.path.isdir, curdir.iterdir())) + curdir: Path = Path(os.curdir).resolve() + dirs: list[Path] = list(filter(os.path.isdir, curdir.iterdir())) with Progress(console=console) as progress: task = progress.add_task(f"Updating {len(dirs)}", total=len(dirs)) for directory in dirs: @@ -113,8 +113,8 @@ def find_matches(_file: Path, provider: str, version: str): for version_dir in version_dirs: version = version_dir.name console.print(version) - for file_name in ["index.html", "example-dags.html"]: - candidate_file = version_dir / file_name + for file in version_dir.rglob("*.html"): + candidate_file = file if candidate_file.exists(): find_matches(candidate_file, provider, version) progress.advance(task) diff --git a/docs/apache-airflow-providers-airbyte/index.rst b/docs/apache-airflow-providers-airbyte/index.rst index 2dce30a82e173..314ec9b9c2868 100644 --- a/docs/apache-airflow-providers-airbyte/index.rst +++ b/docs/apache-airflow-providers-airbyte/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-alibaba/index.rst b/docs/apache-airflow-providers-alibaba/index.rst index 5439aadc2e4a1..de3eff524358c 100644 --- a/docs/apache-airflow-providers-alibaba/index.rst +++ b/docs/apache-airflow-providers-alibaba/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-amazon/example-dags.rst b/docs/apache-airflow-providers-amazon/example-dags.rst index 253ae79041437..8881ac0174f02 100644 --- a/docs/apache-airflow-providers-amazon/example-dags.rst +++ b/docs/apache-airflow-providers-amazon/example-dags.rst @@ -20,5 +20,5 @@ Example DAGs You can learn how to use Amazon AWS integrations by analyzing the source code of the example DAGs: -* `Amazon AWS `__ -* `Amazon AWS (legacy) `__ +* `Amazon AWS `__ +* `Amazon AWS (legacy) `__ diff --git a/docs/apache-airflow-providers-amazon/operators/athena.rst b/docs/apache-airflow-providers-amazon/operators/athena.rst index 3b1fd3b6e83c2..4cb7fcb1faa99 100644 --- a/docs/apache-airflow-providers-amazon/operators/athena.rst +++ b/docs/apache-airflow-providers-amazon/operators/athena.rst @@ -43,7 +43,7 @@ to run a query in Amazon Athena. In the following example, we query an existing Athena table and send the results to an existing Amazon S3 bucket. For more examples of how to use this operator, please -see the `Sample DAG `__. +see the `Sample DAG `__. .. exampleinclude:: /../../tests/system/providers/amazon/aws/example_athena.py :language: python diff --git a/docs/apache-airflow-providers-apache-beam/index.rst b/docs/apache-airflow-providers-apache-beam/index.rst index 5c749f3b0dc76..55f40e3be5339 100644 --- a/docs/apache-airflow-providers-apache-beam/index.rst +++ b/docs/apache-airflow-providers-apache-beam/index.rst @@ -38,7 +38,7 @@ Content :caption: Resources PyPI Repository - Example DAGs + Example DAGs .. toctree:: :maxdepth: 1 diff --git a/docs/apache-airflow-providers-apache-cassandra/index.rst b/docs/apache-airflow-providers-apache-cassandra/index.rst index 6b3f0a5c34f8f..2efd4abaf05c6 100644 --- a/docs/apache-airflow-providers-apache-cassandra/index.rst +++ b/docs/apache-airflow-providers-apache-cassandra/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-drill/index.rst b/docs/apache-airflow-providers-apache-drill/index.rst index 80f0a733aa633..4ca80b2a65a13 100644 --- a/docs/apache-airflow-providers-apache-drill/index.rst +++ b/docs/apache-airflow-providers-apache-drill/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-druid/index.rst b/docs/apache-airflow-providers-apache-druid/index.rst index 3942887d06ea7..1db5f11b7f5fa 100644 --- a/docs/apache-airflow-providers-apache-druid/index.rst +++ b/docs/apache-airflow-providers-apache-druid/index.rst @@ -46,7 +46,7 @@ Content PyPI Repository Installing from sources - Example DAGs + Example DAGs .. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME! diff --git a/docs/apache-airflow-providers-apache-hive/index.rst b/docs/apache-airflow-providers-apache-hive/index.rst index b020e440e01c4..2203e3321a680 100644 --- a/docs/apache-airflow-providers-apache-hive/index.rst +++ b/docs/apache-airflow-providers-apache-hive/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-kylin/index.rst b/docs/apache-airflow-providers-apache-kylin/index.rst index 46c3675dd6c7b..ecdcb30832157 100644 --- a/docs/apache-airflow-providers-apache-kylin/index.rst +++ b/docs/apache-airflow-providers-apache-kylin/index.rst @@ -38,7 +38,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-livy/index.rst b/docs/apache-airflow-providers-apache-livy/index.rst index c769bf91ba3fe..31699da682427 100644 --- a/docs/apache-airflow-providers-apache-livy/index.rst +++ b/docs/apache-airflow-providers-apache-livy/index.rst @@ -43,7 +43,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-pig/index.rst b/docs/apache-airflow-providers-apache-pig/index.rst index 0f00fdd90d625..65a9978eef70a 100644 --- a/docs/apache-airflow-providers-apache-pig/index.rst +++ b/docs/apache-airflow-providers-apache-pig/index.rst @@ -43,7 +43,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-apache-spark/index.rst b/docs/apache-airflow-providers-apache-spark/index.rst index af7bc319a4edf..18fe4e16ef6e5 100644 --- a/docs/apache-airflow-providers-apache-spark/index.rst +++ b/docs/apache-airflow-providers-apache-spark/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-arangodb/index.rst b/docs/apache-airflow-providers-arangodb/index.rst index 5b5d029546874..9d5c0833661c3 100644 --- a/docs/apache-airflow-providers-arangodb/index.rst +++ b/docs/apache-airflow-providers-arangodb/index.rst @@ -39,7 +39,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs .. toctree:: :maxdepth: 1 diff --git a/docs/apache-airflow-providers-asana/index.rst b/docs/apache-airflow-providers-asana/index.rst index e594a2169a882..a643b5a31b679 100644 --- a/docs/apache-airflow-providers-asana/index.rst +++ b/docs/apache-airflow-providers-asana/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-cncf-kubernetes/index.rst b/docs/apache-airflow-providers-cncf-kubernetes/index.rst index 176b1608d9ad1..c87d995d078c3 100644 --- a/docs/apache-airflow-providers-cncf-kubernetes/index.rst +++ b/docs/apache-airflow-providers-cncf-kubernetes/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-common-sql/index.rst b/docs/apache-airflow-providers-common-sql/index.rst index ed59ef94dc9e1..dfb701b676dc5 100644 --- a/docs/apache-airflow-providers-common-sql/index.rst +++ b/docs/apache-airflow-providers-common-sql/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-databricks/index.rst b/docs/apache-airflow-providers-databricks/index.rst index f02e48e1d019c..c11298b00c5d1 100644 --- a/docs/apache-airflow-providers-databricks/index.rst +++ b/docs/apache-airflow-providers-databricks/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-dbt-cloud/index.rst b/docs/apache-airflow-providers-dbt-cloud/index.rst index 0df50f9432656..28f05b42cb5f1 100644 --- a/docs/apache-airflow-providers-dbt-cloud/index.rst +++ b/docs/apache-airflow-providers-dbt-cloud/index.rst @@ -49,7 +49,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-dingding/index.rst b/docs/apache-airflow-providers-dingding/index.rst index c001f9d6d4e1d..0b4be2a481a1b 100644 --- a/docs/apache-airflow-providers-dingding/index.rst +++ b/docs/apache-airflow-providers-dingding/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-docker/index.rst b/docs/apache-airflow-providers-docker/index.rst index 0629d753df480..c768e0a2a8e43 100644 --- a/docs/apache-airflow-providers-docker/index.rst +++ b/docs/apache-airflow-providers-docker/index.rst @@ -39,7 +39,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-elasticsearch/index.rst b/docs/apache-airflow-providers-elasticsearch/index.rst index d238b52ac6965..a12aee4aa088e 100644 --- a/docs/apache-airflow-providers-elasticsearch/index.rst +++ b/docs/apache-airflow-providers-elasticsearch/index.rst @@ -46,7 +46,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-github/index.rst b/docs/apache-airflow-providers-github/index.rst index 61e2054c68ec2..44335d69aec35 100644 --- a/docs/apache-airflow-providers-github/index.rst +++ b/docs/apache-airflow-providers-github/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-google/example-dags.rst b/docs/apache-airflow-providers-google/example-dags.rst index b912ef3ea4fe6..49f4fbf327f12 100644 --- a/docs/apache-airflow-providers-google/example-dags.rst +++ b/docs/apache-airflow-providers-google/example-dags.rst @@ -19,10 +19,10 @@ Example DAGs ============ You can learn how to use Google integrations by analyzing the source code of the example DAGs: -* `Google Ads `__ -* `Google Cloud (legacy) `__ -* `Google Cloud `__ -* `Google Firebase `__ -* `Google Marketing Platform `__ -* `Google Workplace `__ (formerly Google Suite) -* `Google LevelDB `__ +* `Google Ads `__ +* `Google Cloud (legacy) `__ +* `Google Cloud `__ +* `Google Firebase `__ +* `Google Marketing Platform `__ +* `Google Workplace `__ (formerly Google Suite) +* `Google LevelDB `__ diff --git a/docs/apache-airflow-providers-google/operators/cloud/index.rst b/docs/apache-airflow-providers-google/operators/cloud/index.rst index 22ded79012385..f974c0eb81a4f 100644 --- a/docs/apache-airflow-providers-google/operators/cloud/index.rst +++ b/docs/apache-airflow-providers-google/operators/cloud/index.rst @@ -29,4 +29,4 @@ Google Cloud Operators .. note:: You can learn how to use Google Cloud integrations by analyzing the - `source code `_ of the particular example DAGs. + `source code `_ of the particular example DAGs. diff --git a/docs/apache-airflow-providers-http/index.rst b/docs/apache-airflow-providers-http/index.rst index 60025a889c901..780a2e406161f 100644 --- a/docs/apache-airflow-providers-http/index.rst +++ b/docs/apache-airflow-providers-http/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-influxdb/index.rst b/docs/apache-airflow-providers-influxdb/index.rst index 1a3300c06ae1d..2b1b9962b6faf 100644 --- a/docs/apache-airflow-providers-influxdb/index.rst +++ b/docs/apache-airflow-providers-influxdb/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-jdbc/index.rst b/docs/apache-airflow-providers-jdbc/index.rst index bf149f52b043b..5fcb0a20f8280 100644 --- a/docs/apache-airflow-providers-jdbc/index.rst +++ b/docs/apache-airflow-providers-jdbc/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-jenkins/index.rst b/docs/apache-airflow-providers-jenkins/index.rst index 5c1626ab657c1..b8afb8eb4f8eb 100644 --- a/docs/apache-airflow-providers-jenkins/index.rst +++ b/docs/apache-airflow-providers-jenkins/index.rst @@ -43,7 +43,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-microsoft-azure/index.rst b/docs/apache-airflow-providers-microsoft-azure/index.rst index dda9160a313b8..9405f476aa0f7 100644 --- a/docs/apache-airflow-providers-microsoft-azure/index.rst +++ b/docs/apache-airflow-providers-microsoft-azure/index.rst @@ -47,7 +47,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-microsoft-mssql/index.rst b/docs/apache-airflow-providers-microsoft-mssql/index.rst index 3a2c11aeb7197..ab20bfcf81f09 100644 --- a/docs/apache-airflow-providers-microsoft-mssql/index.rst +++ b/docs/apache-airflow-providers-microsoft-mssql/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-microsoft-winrm/index.rst b/docs/apache-airflow-providers-microsoft-winrm/index.rst index 042cd90b32781..ac244b5120a9c 100644 --- a/docs/apache-airflow-providers-microsoft-winrm/index.rst +++ b/docs/apache-airflow-providers-microsoft-winrm/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-mysql/index.rst b/docs/apache-airflow-providers-mysql/index.rst index 403c958c954b9..290abcef24b97 100644 --- a/docs/apache-airflow-providers-mysql/index.rst +++ b/docs/apache-airflow-providers-mysql/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-neo4j/index.rst b/docs/apache-airflow-providers-neo4j/index.rst index b08d54042f253..8bf095805dc9c 100644 --- a/docs/apache-airflow-providers-neo4j/index.rst +++ b/docs/apache-airflow-providers-neo4j/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-opsgenie/index.rst b/docs/apache-airflow-providers-opsgenie/index.rst index 9109513f16e6b..3dc179c0a1632 100644 --- a/docs/apache-airflow-providers-opsgenie/index.rst +++ b/docs/apache-airflow-providers-opsgenie/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-papermill/index.rst b/docs/apache-airflow-providers-papermill/index.rst index 9b66d74602259..7a7f39daec878 100644 --- a/docs/apache-airflow-providers-papermill/index.rst +++ b/docs/apache-airflow-providers-papermill/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-plexus/index.rst b/docs/apache-airflow-providers-plexus/index.rst index c738c121dceca..3726213c9c7b9 100644 --- a/docs/apache-airflow-providers-plexus/index.rst +++ b/docs/apache-airflow-providers-plexus/index.rst @@ -38,7 +38,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-postgres/index.rst b/docs/apache-airflow-providers-postgres/index.rst index da55f7744f206..6200c3602d609 100644 --- a/docs/apache-airflow-providers-postgres/index.rst +++ b/docs/apache-airflow-providers-postgres/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-presto/index.rst b/docs/apache-airflow-providers-presto/index.rst index 1cc3a6e108cad..76445f2ed3403 100644 --- a/docs/apache-airflow-providers-presto/index.rst +++ b/docs/apache-airflow-providers-presto/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-qubole/index.rst b/docs/apache-airflow-providers-qubole/index.rst index e498e11516359..bbc5f33331fe1 100644 --- a/docs/apache-airflow-providers-qubole/index.rst +++ b/docs/apache-airflow-providers-qubole/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-qubole/operators/index.rst b/docs/apache-airflow-providers-qubole/operators/index.rst index deefeb75392e4..1dbccd05442d5 100644 --- a/docs/apache-airflow-providers-qubole/operators/index.rst +++ b/docs/apache-airflow-providers-qubole/operators/index.rst @@ -29,4 +29,4 @@ Qubole Operators .. note:: You can learn how to use Google Cloud integrations by analyzing the - `source code `_ of the particular example DAGs. + `source code `_ of the particular example DAGs. diff --git a/docs/apache-airflow-providers-salesforce/index.rst b/docs/apache-airflow-providers-salesforce/index.rst index 0d1c99c4a56f8..ec317f3804535 100644 --- a/docs/apache-airflow-providers-salesforce/index.rst +++ b/docs/apache-airflow-providers-salesforce/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-singularity/index.rst b/docs/apache-airflow-providers-singularity/index.rst index 76d61aabacad2..0595c5bccda96 100644 --- a/docs/apache-airflow-providers-singularity/index.rst +++ b/docs/apache-airflow-providers-singularity/index.rst @@ -38,7 +38,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-slack/index.rst b/docs/apache-airflow-providers-slack/index.rst index 5c87dc46806a3..b67f7d21d8126 100644 --- a/docs/apache-airflow-providers-slack/index.rst +++ b/docs/apache-airflow-providers-slack/index.rst @@ -44,7 +44,7 @@ Content Connection Types Python API <_api/airflow/providers/slack/index> - Example DAGs + Example DAGs .. toctree:: :maxdepth: 1 diff --git a/docs/apache-airflow-providers-snowflake/index.rst b/docs/apache-airflow-providers-snowflake/index.rst index 4ff166c55c899..acc8996b1311e 100644 --- a/docs/apache-airflow-providers-snowflake/index.rst +++ b/docs/apache-airflow-providers-snowflake/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-sqlite/index.rst b/docs/apache-airflow-providers-sqlite/index.rst index 875ac35a20a74..ca7fb0d2c519f 100644 --- a/docs/apache-airflow-providers-sqlite/index.rst +++ b/docs/apache-airflow-providers-sqlite/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-tableau/index.rst b/docs/apache-airflow-providers-tableau/index.rst index bacdbb1fa3497..ac99aba77348c 100644 --- a/docs/apache-airflow-providers-tableau/index.rst +++ b/docs/apache-airflow-providers-tableau/index.rst @@ -40,7 +40,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-tabular/index.rst b/docs/apache-airflow-providers-tabular/index.rst index be1193f3d6a30..72704e9c32d93 100644 --- a/docs/apache-airflow-providers-tabular/index.rst +++ b/docs/apache-airflow-providers-tabular/index.rst @@ -38,7 +38,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources Python API <_api/airflow/providers/tabular/index> diff --git a/docs/apache-airflow-providers-telegram/index.rst b/docs/apache-airflow-providers-telegram/index.rst index aaf5637754b38..f137d6c027db5 100644 --- a/docs/apache-airflow-providers-telegram/index.rst +++ b/docs/apache-airflow-providers-telegram/index.rst @@ -44,7 +44,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-trino/index.rst b/docs/apache-airflow-providers-trino/index.rst index fc526b4bdb0b4..d56a7e5e6df9f 100644 --- a/docs/apache-airflow-providers-trino/index.rst +++ b/docs/apache-airflow-providers-trino/index.rst @@ -46,7 +46,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-yandex/index.rst b/docs/apache-airflow-providers-yandex/index.rst index fbfe9ea1761f1..ce70186bda186 100644 --- a/docs/apache-airflow-providers-yandex/index.rst +++ b/docs/apache-airflow-providers-yandex/index.rst @@ -45,7 +45,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/apache-airflow-providers-yandex/operators.rst b/docs/apache-airflow-providers-yandex/operators.rst index 552fd4febd73b..4fab4468d67ab 100644 --- a/docs/apache-airflow-providers-yandex/operators.rst +++ b/docs/apache-airflow-providers-yandex/operators.rst @@ -37,4 +37,4 @@ Prerequisite Tasks Using the operators ^^^^^^^^^^^^^^^^^^^^^ -See the usage examples in `example DAGs `_ +See the usage examples in `example DAGs `_ diff --git a/docs/apache-airflow-providers-zendesk/index.rst b/docs/apache-airflow-providers-zendesk/index.rst index 9f39536f26ce9..b4f95f6295a7a 100644 --- a/docs/apache-airflow-providers-zendesk/index.rst +++ b/docs/apache-airflow-providers-zendesk/index.rst @@ -38,7 +38,7 @@ Content :maxdepth: 1 :caption: Resources - Example DAGs + Example DAGs PyPI Repository Installing from sources diff --git a/docs/conf.py b/docs/conf.py index f75704d9dd666..208883177f237 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -172,6 +172,13 @@ elif PACKAGE_NAME == "docker-stack": # No extra extensions pass +elif PACKAGE_NAME.startswith("apache-airflow-providers-"): + extensions.extend( + [ + "extra_provider_files_with_substitutions", + "autoapi.extension", + ] + ) else: extensions.append("autoapi.extension") # List of patterns, relative to source directory, that match files and @@ -298,7 +305,8 @@ def _get_rst_filepath_from_path(filepath: pathlib.Path): "installation/installing-from-pypi.html", "installation/installing-from-sources.html", ] - +if PACKAGE_NAME.startswith("apache-airflow-providers"): + manual_substitutions_in_generated_html = ["example-dags.html", "operators.html", "index.html"] if PACKAGE_NAME == "docker-stack": # Replace "|version|" inside ```` quotes manual_substitutions_in_generated_html = ["build.html"] diff --git a/docs/exts/extra_provider_files_with_substitutions.py b/docs/exts/extra_provider_files_with_substitutions.py new file mode 100644 index 0000000000000..66ec2356f6a1f --- /dev/null +++ b/docs/exts/extra_provider_files_with_substitutions.py @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from pathlib import Path + + +def fix_provider_references(app, exception): + """Sphinx "build-finished" event handler.""" + from sphinx.builders import html as builders + + if exception or not isinstance(app.builder, builders.StandaloneHTMLBuilder): + return + + # Replace `|version|` in the files that require manual substitution + for path in Path(app.outdir).rglob("*.html"): + if not path.exists(): + continue + with open(path) as input_file: + content = input_file.readlines() + with open(path, "wt") as output_file: + for line in content: + output_file.write(line.replace("|version|", app.config.version)) + + +def setup(app): + """Setup plugin""" + app.connect("build-finished", fix_provider_references) + + return { + "parallel_write_safe": True, + } diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt index a538cf7cd3dfb..6252c914f85f3 100644 --- a/images/breeze/output-commands-hash.txt +++ b/images/breeze/output-commands-hash.txt @@ -50,7 +50,7 @@ setup:version:123b462a421884dc2320ffc5e54b2478 setup:a3bd246c3a425f3e586d11bbdc8937cb shell:51149096ed3fb28d2a01e0a5ad0146ef start-airflow:4bec5d26386dc268ec2de2f637c85272 -static-checks:61c8d96ffe3008bff89a9df39b423be2 +static-checks:7a39e28c87fbca0a9fae0ebfe1591b71 stop:8969537ccdd799f692ccb8600a7bbed6 testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c testing:helm-tests:94a442e7f3f63b34c4831a84d165690a diff --git a/images/breeze/output-commands.svg b/images/breeze/output-commands.svg index 763f7d8c8051d..69f6d44144528 100644 --- a/images/breeze/output-commands.svg +++ b/images/breeze/output-commands.svg @@ -35,8 +35,8 @@ .breeze-help-r1 { fill: #c5c8c6;font-weight: bold } .breeze-help-r2 { fill: #c5c8c6 } .breeze-help-r3 { fill: #d0b344;font-weight: bold } -.breeze-help-r4 { fill: #68a0b3;font-weight: bold } -.breeze-help-r5 { fill: #868887 } +.breeze-help-r4 { fill: #868887 } +.breeze-help-r5 { fill: #68a0b3;font-weight: bold } .breeze-help-r6 { fill: #98a84b;font-weight: bold } .breeze-help-r7 { fill: #8d7b39 } @@ -190,50 +190,50 @@ -Usage: breeze [OPTIONS] COMMAND [ARGS]... +Usage: breeze [OPTIONS] COMMAND [ARGS]... -╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--python-pPython major/minor version used in Airflow image for images.(>3.7< | 3.8 | 3.9 | 3.10)│ -│[default: 3.7]                                              │ -│--backend-bDatabase backend to use.(>sqlite< | mysql | postgres | mssql)[default: sqlite]│ -│--postgres-version-PVersion of Postgres used.(>11< | 12 | 13 | 14 | 15)[default: 11]│ -│--mysql-version-MVersion of MySQL used.(>5.7< | 8)[default: 5.7]│ -│--mssql-version-SVersion of MsSQL used.(>2017-latest< | 2019-latest)[default: 2017-latest]│ -│--integrationIntegration(s) to enable when running (can be more than one).                            │ -│(cassandra | kerberos | mongo | openldap | pinot | rabbitmq | redis | statsd | trino |   │ -│all)                                                                                     │ -│--forward-credentials-fForward local credentials to container when running.│ -│--db-reset-dReset DB when entering the container.│ -│--max-timeMaximum time that the command should take - if it takes longer, the command will fail.│ -│(INTEGER RANGE)                                                                       │ -│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--verbose-vPrint verbose information about performed steps.│ -│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ -│--answer-aForce answer to questions.(y | n | q | yes | no | quit)│ -│--help-hShow this message and exit.│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Basic developer commands ───────────────────────────────────────────────────────────────────────────────────────────╮ -│start-airflow     Enter breeze environment and starts all Airflow components in the tmux session. Compile assets  │ -│if contents of www directory changed.                                                           │ -│static-checks     Run static checks.                                                                              │ -│build-docs        Build documentation in the container.                                                           │ -│stop              Stop running breeze environment.                                                                │ -│shell             Enter breeze environment. this is the default command use when no other is selected.            │ -│exec              Joins the interactive shell of running airflow container.                                       │ -│compile-www-assetsCompiles www assets.                                                                            │ -│cleanup           Cleans the cache of parameters, docker cache and optionally built CI/PROD images.               │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Advanced command groups ────────────────────────────────────────────────────────────────────────────────────────────╮ -│testing                Tools that developers can use to run tests                                                 │ -│ci-image               Tools that developers can use to manually manage CI images                                 │ -│k8s                    Tools that developers use to run Kubernetes tests                                          │ -│prod-image             Tools that developers can use to manually manage PROD images                               │ -│setup                  Tools that developers can use to configure Breeze                                          │ -│release-management     Tools that release managers can use to prepare and manage Airflow releases                 │ -│ci                     Tools that CI workflows use to cleanup/manage CI environment                               │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Basic flags ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--python-pPython major/minor version used in Airflow image for images.(>3.7< | 3.8 | 3.9 | 3.10)│ +│[default: 3.7]                                              │ +│--backend-bDatabase backend to use.(>sqlite< | mysql | postgres | mssql)[default: sqlite]│ +│--postgres-version-PVersion of Postgres used.(>11< | 12 | 13 | 14 | 15)[default: 11]│ +│--mysql-version-MVersion of MySQL used.(>5.7< | 8)[default: 5.7]│ +│--mssql-version-SVersion of MsSQL used.(>2017-latest< | 2019-latest)[default: 2017-latest]│ +│--integrationIntegration(s) to enable when running (can be more than one).                            │ +│(cassandra | kerberos | mongo | openldap | pinot | rabbitmq | redis | statsd | trino |   │ +│all)                                                                                     │ +│--forward-credentials-fForward local credentials to container when running.│ +│--db-reset-dReset DB when entering the container.│ +│--max-timeMaximum time that the command should take - if it takes longer, the command will fail.│ +│(INTEGER RANGE)                                                                       │ +│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--verbose-vPrint verbose information about performed steps.│ +│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ +│--answer-aForce answer to questions.(y | n | q | yes | no | quit)│ +│--help-hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Basic developer commands ───────────────────────────────────────────────────────────────────────────────────────────╮ +│start-airflow     Enter breeze environment and starts all Airflow components in the tmux session. Compile assets  │ +│if contents of www directory changed.                                                           │ +│static-checks     Run static checks.                                                                              │ +│build-docs        Build documentation in the container.                                                           │ +│stop              Stop running breeze environment.                                                                │ +│shell             Enter breeze environment. this is the default command use when no other is selected.            │ +│exec              Joins the interactive shell of running airflow container.                                       │ +│compile-www-assetsCompiles www assets.                                                                            │ +│cleanup           Cleans the cache of parameters, docker cache and optionally built CI/PROD images.               │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Advanced command groups ────────────────────────────────────────────────────────────────────────────────────────────╮ +│testing                Tools that developers can use to run tests                                                 │ +│ci-image               Tools that developers can use to manually manage CI images                                 │ +│k8s                    Tools that developers use to run Kubernetes tests                                          │ +│prod-image             Tools that developers can use to manually manage PROD images                               │ +│setup                  Tools that developers can use to configure Breeze                                          │ +│release-management     Tools that release managers can use to prepare and manage Airflow releases                 │ +│ci                     Tools that CI workflows use to cleanup/manage CI environment                               │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/images/breeze/output_static-checks.svg b/images/breeze/output_static-checks.svg index 05491b2178f42..d25a6fd1f0122 100644 --- a/images/breeze/output_static-checks.svg +++ b/images/breeze/output_static-checks.svg @@ -1,4 +1,4 @@ - + - + @@ -198,9 +198,12 @@ + + + - Command: static-checks + Command: static-checks @@ -211,57 +214,58 @@ -Usage: breeze static-checks [OPTIONS] [PRECOMMIT_ARGS]... +Usage: breeze static-checks [OPTIONS] [PRECOMMIT_ARGS]... Run static checks. -╭─ Pre-commit flags ───────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--type-tType(s) of the static checks to run (multiple can be added).                            │ -│(all | black | blacken-docs | check-airflow-config-yaml-consistent |                    │ -│check-airflow-provider-compatibility | check-apache-license-rat |                       │ -│check-base-operator-partial-arguments | check-base-operator-usage |                     │ -│check-boring-cyborg-configuration | check-breeze-top-dependencies-limited |             │ -│check-builtin-literals | check-changelog-has-no-duplicates |                            │ -│check-core-deprecation-classes | check-daysago-import-from-utils |                      │ -│check-decorated-operator-implements-custom-name | check-docstring-param-types |         │ -│check-example-dags-urls | check-executables-have-shebangs |                             │ -│check-extra-packages-references | check-extras-order | check-for-inclusive-language |   │ -│check-hooks-apply | check-incorrect-use-of-LoggingMixin | check-init-decorator-arguments│ -│| check-lazy-logging | check-merge-conflict | check-newsfragments-are-valid |           │ -│check-no-providers-in-core-examples | check-no-relative-imports |                       │ -│check-persist-credentials-disabled-in-github-workflows |                                │ -│check-pre-commit-information-consistent | check-provide-create-sessions-imports |       │ -│check-provider-yaml-valid | check-providers-init-file-missing |                         │ -│check-providers-subpackages-init-file-exist | check-pydevd-left-in-code |               │ -│check-revision-heads-map | check-safe-filter-usage-in-html | check-setup-order |        │ -│check-start-date-not-used-in-defaults | check-system-tests-present |                    │ -│check-system-tests-tocs | check-xml | codespell | compile-www-assets |                  │ -│compile-www-assets-dev | create-missing-init-py-files-tests | debug-statements |        │ -│detect-private-key | doctoc | end-of-file-fixer | fix-encoding-pragma | flynt | identity│ -│| insert-license | isort | lint-chart-schema | lint-css | lint-dockerfile |             │ -│lint-helm-chart | lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | │ -│pretty-format-json | pydocstyle | python-no-log-warn | pyupgrade |                      │ -│replace-bad-characters | rst-backticks | run-flake8 | run-mypy | run-shellcheck |       │ -│static-check-autoflake | trailing-whitespace | ts-compile-and-lint-javascript |         │ -│update-breeze-cmd-output | update-breeze-readme-config-hash | update-er-diagram |       │ -│update-extras | update-in-the-wild-to-be-sorted | update-inlined-dockerfile-scripts |   │ -│update-local-yml-file | update-migration-references | update-providers-dependencies |   │ -│update-spelling-wordlist-to-be-sorted | update-supported-versions |                     │ -│update-vendored-in-k8s-json-schema | update-version | yamllint | yesqa)                 │ -│--file-fList of files to run the checks on.(PATH)│ -│--all-files-aRun checks on all files.│ -│--show-diff-on-failure-sShow diff for files modified by the checks.│ -│--last-commit-cRun checks for all files in last commit. Mutually exclusive with --commit-ref.│ -│--commit-ref-rRun checks for this commit reference only (can be any git commit-ish reference).        │ -│Mutually exclusive with --last-commit.                                                  │ -│(TEXT)                                                                                  │ -│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│--verbose-vPrint verbose information about performed steps.│ -│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ -│--help-hShow this message and exit.│ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Pre-commit flags ───────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--type-tType(s) of the static checks to run (multiple can be added).                            │ +│(all | black | blacken-docs | check-airflow-config-yaml-consistent |                    │ +│check-airflow-provider-compatibility | check-apache-license-rat |                       │ +│check-base-operator-partial-arguments | check-base-operator-usage |                     │ +│check-boring-cyborg-configuration | check-breeze-top-dependencies-limited |             │ +│check-builtin-literals | check-changelog-has-no-duplicates |                            │ +│check-core-deprecation-classes | check-daysago-import-from-utils |                      │ +│check-decorated-operator-implements-custom-name | check-docstring-param-types |         │ +│check-example-dags-urls | check-executables-have-shebangs |                             │ +│check-extra-packages-references | check-extras-order | check-for-inclusive-language |   │ +│check-hooks-apply | check-incorrect-use-of-LoggingMixin | check-init-decorator-arguments│ +│| check-lazy-logging | check-links-to-example-dags-do-not-use-hardcoded-versions |      │ +│check-merge-conflict | check-newsfragments-are-valid |                                  │ +│check-no-providers-in-core-examples | check-no-relative-imports |                       │ +│check-persist-credentials-disabled-in-github-workflows |                                │ +│check-pre-commit-information-consistent | check-provide-create-sessions-imports |       │ +│check-provider-yaml-valid | check-providers-init-file-missing |                         │ +│check-providers-subpackages-init-file-exist | check-pydevd-left-in-code |               │ +│check-revision-heads-map | check-safe-filter-usage-in-html | check-setup-order |        │ +│check-start-date-not-used-in-defaults | check-system-tests-present |                    │ +│check-system-tests-tocs | check-xml | codespell | compile-www-assets |                  │ +│compile-www-assets-dev | create-missing-init-py-files-tests | debug-statements |        │ +│detect-private-key | doctoc | end-of-file-fixer | fix-encoding-pragma | flynt | identity│ +│| insert-license | isort | lint-chart-schema | lint-css | lint-dockerfile |             │ +│lint-helm-chart | lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending | │ +│pretty-format-json | pydocstyle | python-no-log-warn | pyupgrade |                      │ +│replace-bad-characters | rst-backticks | run-flake8 | run-mypy | run-shellcheck |       │ +│static-check-autoflake | trailing-whitespace | ts-compile-and-lint-javascript |         │ +│update-breeze-cmd-output | update-breeze-readme-config-hash | update-er-diagram |       │ +│update-extras | update-in-the-wild-to-be-sorted | update-inlined-dockerfile-scripts |   │ +│update-local-yml-file | update-migration-references | update-providers-dependencies |   │ +│update-spelling-wordlist-to-be-sorted | update-supported-versions |                     │ +│update-vendored-in-k8s-json-schema | update-version | yamllint | yesqa)                 │ +│--file-fList of files to run the checks on.(PATH)│ +│--all-files-aRun checks on all files.│ +│--show-diff-on-failure-sShow diff for files modified by the checks.│ +│--last-commit-cRun checks for all files in last commit. Mutually exclusive with --commit-ref.│ +│--commit-ref-rRun checks for this commit reference only (can be any git commit-ish reference).        │ +│Mutually exclusive with --last-commit.                                                  │ +│(TEXT)                                                                                  │ +│--github-repository-gGitHub repository used to pull, push run images.(TEXT)[default: apache/airflow]│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--verbose-vPrint verbose information about performed steps.│ +│--dry-run-DIf dry-run is set, commands are only printed, not executed.│ +│--help-hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯