azure-kusto-data 4.6.0 breaks provider tests and static checks

[jscheffl]

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

See: https://github.com/apache/airflow/actions/runs/11093100295

Static checks fail in:

Found 3 errors in providers
Error: The `airflow.providers.microsoft.azure.hooks.adx.AzureDataExplorerHook` object in connection-types list in airflow/providers/microsoft/azure/provider.yaml does not exist or is not a class: No module named 'build'
Error: The `airflow.providers.microsoft.azure.operators.adx` object in operators list in airflow/providers/microsoft/azure/provider.yaml does not exist or is not a module: No module named 'build'
Error: The `airflow.providers.microsoft.azure.hooks.adx` object in hooks list in airflow/providers/microsoft/azure/provider.yaml does not exist or is not a module: No module named 'build'
Error 1 returned

...with traceback...

 Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/run_template_fields_check.py", line 174, in <module>
    [iter_check_template_fields(module) for module in modules_to_validate]
  File "/opt/airflow/scripts/in_container/run_template_fields_check.py", line 174, in <listcomp>
    [iter_check_template_fields(module) for module in modules_to_validate]
  File "/opt/airflow/scripts/in_container/run_template_fields_check.py", line 148, in iter_check_template_fields
    imported_module = importlib.import_module(module)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/airflow/providers/microsoft/azure/operators/adx.py", line 30, in <module>
    from airflow.providers.microsoft.azure.hooks.adx import AzureDataExplorerHook
  File "/opt/airflow/airflow/providers/microsoft/azure/hooks/adx.py", line 33, in <module>
    from azure.kusto.data import ClientRequestProperties, KustoClient, KustoConnectionStringBuilder
  File "/usr/local/lib/python3.8/site-packages/azure/kusto/data/__init__.py", line 5, in <module>
    from .client import KustoClient
  File "/usr/local/lib/python3.8/site-packages/azure/kusto/data/client.py", line 17, in <module>
    from build.lib.azure.kusto.data.exceptions import KustoServiceError
ModuleNotFoundError: No module named 'build'

Provider tests fail in:

==================================== ERRORS ====================================
______ ERROR collecting tests/providers/microsoft/azure/hooks/test_adx.py ______
ImportError while importing test module '/opt/airflow/tests/providers/microsoft/azure/hooks/test_adx.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/providers/microsoft/azure/hooks/test_adx.py:23: in <module>
    from azure.kusto.data import ClientRequestProperties, KustoClient
/usr/local/lib/python3.8/site-packages/azure/kusto/data/__init__.py:5: in <module>
    from .client import KustoClient
/usr/local/lib/python3.8/site-packages/azure/kusto/data/client.py:17: in <module>
    from build.lib.azure.kusto.data.exceptions import KustoServiceError
E   ModuleNotFoundError: No module named 'build'
____ ERROR collecting tests/providers/microsoft/azure/operators/test_adx.py ____
ImportError while importing test module '/opt/airflow/tests/providers/microsoft/azure/operators/test_adx.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/providers/microsoft/azure/operators/test_adx.py:23: in <module>
    from azure.kusto.data._models import KustoResultTable
/usr/local/lib/python3.8/site-packages/azure/kusto/data/__init__.py:5: in <module>
    from .client import KustoClient
/usr/local/lib/python3.8/site-packages/azure/kusto/data/client.py:17: in <module>
    from build.lib.azure.kusto.data.exceptions import KustoServiceError
E   ModuleNotFoundError: No module named 'build'
- generated xml file: /files/test_result-providers_-amazon_google-postgres.xml -
=================== Warning summary. Total: 348, Unique: 334 ===================
other: total 347, unique 333
  collect: total 347, unique 333
tests: total 1, unique 1
  collect: total 1, unique 1
Warnings saved into /files/warnings-providers_-amazon_google-postgres.txt file.
=========================== short test summary info ============================
ERROR tests/providers/microsoft/azure/hooks/test_adx.py
ERROR tests/providers/microsoft/azure/operators/test_adx.py
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
========================= 2 errors in 68.63s (0:01:08) =========================

Apache Airflow version

any

Operating System

any

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

No response

What you think should happen instead

No response

How to reproduce

See https://github.com/apache/airflow/actions/runs/11093100295

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Pyasma]

Hi @jscheffl, is this issue beginner-friendly? Is it still open? I’d also appreciate any advice or suggestions on how to approach it.

[jscheffl]

I hope and assume so. 50% sure :-D
So yesterday I fixed main branch by pinning the package in PR #42576 to be !=4.6.0. If using 4.6.0 then some tests fail. So if you check out main and remove the pinning, rebuild CI image and execute tests... then you should see the failure. The ones printed above in the description. Follow the link to the failed run and check which commands were called leading to the failure.

Static checks should be something like breeze static-checks --all-files --show-diff-on-failure --color always --initialize-environment

The pytests will be breeze testing db-tests --parallel-test-types "API Always BranchExternalPython BranchPythonVenv CLI Core ExternalPython Operators Other PlainAsserts Providers[-amazon,google] Providers[amazon] Providers[google] PythonVenv Serialization WWW"

I assume some API in the kusto package has moved in 4.6.0 and pytests need to be re-worked for changed API.

[potiuk]

The azure-kusto-data 4.6.1 have been released yesterday and seems that canary tests passed and constraints were updated, so we can close this one because !=4.6.0 is the right exclusion