Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically change python target version for MR dependencies #1898

Merged
merged 6 commits into from
Oct 8, 2024

Conversation

lugi0
Copy link
Contributor

@lugi0 lugi0 commented Oct 8, 2024

No description provided.

Signed-off-by: lugi0 <lgiorgi@redhat.com>
@lugi0 lugi0 added the enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes) label Oct 8, 2024
@lugi0 lugi0 self-assigned this Oct 8, 2024
tonyxrmdavidson
tonyxrmdavidson previously approved these changes Oct 8, 2024
Copy link
Contributor

@tonyxrmdavidson tonyxrmdavidson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm for me

Copy link
Contributor

github-actions bot commented Oct 8, 2024

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
547 0 0 547 100

@jstourac
Copy link
Member

jstourac commented Oct 8, 2024

We should check what is the rootcause here. The version of the image for the workbenches has been bumped from 2024.1 to 2024.2 which incorporates version updates for basically all packages including JupyterLab bump from 3.6 to 4.2.5 and elyra bump.

@jstourac
Copy link
Member

jstourac commented Oct 8, 2024

I tried to mimic what this test does on the Standard Data Science image version 2024.2 (uses Python 3.11):

$ pip download --platform=manylinux2014_x86_64 --python-version=3.9 --abi=cp39 --only-binary=:all: --dest=/opt/app-root/src/  model_registry==0.2.6a1

$ pip install --no-index --find-links . model_registry-0.2.6a1-py3-none-any.whl
Looking in links: .
Processing ./model_registry-0.2.6a1-py3-none-any.whl
Requirement already satisfied: aiohttp<4.0.0,>=3.9.5 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (3.10.9)
Processing ./aiohttp_retry-2.8.3-py3-none-any.whl (from model-registry==0.2.6a1)
Processing ./eval_type_backport-0.2.0-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: nest-asyncio<2.0.0,>=1.6.0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (1.6.0)
Processing ./pydantic-2.9.2-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.9.0.post0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (2.9.0.post0)
Requirement already satisfied: typing-extensions<5.0,>=4.8 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (4.12.2)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (2.4.3)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (24.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (6.1.0)
Requirement already satisfied: yarl<2.0,>=1.12.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.13.1)
Processing ./annotated_types-0.7.0-py3-none-any.whl (from pydantic<3.0.0,>=2.7.4->model-registry==0.2.6a1)
INFO: pip is looking at multiple versions of pydantic to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement pydantic-core==2.23.4 (from pydantic) (from versions: none)
ERROR: No matching distribution found for pydantic-core==2.23.4

Then I realized that you use Python 3.9 in your commands, so I did the following instead:

$ pip download --platform=manylinux2014_x86_64 --python-version=3.11 --abi=cp311 --only-binary=:all: --dest=/opt/app-root/src/  model_registry==0.2.6a1

$ pip install --no-index --find-links . model_registry-0.2.6a1-py3-none-any.whl
Looking in links: .
Processing ./model_registry-0.2.6a1-py3-none-any.whl
Requirement already satisfied: aiohttp<4.0.0,>=3.9.5 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (3.10.9)
Processing ./aiohttp_retry-2.8.3-py3-none-any.whl (from model-registry==0.2.6a1)
Processing ./eval_type_backport-0.2.0-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: nest-asyncio<2.0.0,>=1.6.0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (1.6.0)
Processing ./pydantic-2.9.2-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.9.0.post0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (2.9.0.post0)
Requirement already satisfied: typing-extensions<5.0,>=4.8 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (4.12.2)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (2.4.3)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (24.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (6.1.0)
Requirement already satisfied: yarl<2.0,>=1.12.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.13.1)
Processing ./annotated_types-0.7.0-py3-none-any.whl (from pydantic<3.0.0,>=2.7.4->model-registry==0.2.6a1)
Processing ./pydantic_core-2.23.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from pydantic<3.0.0,>=2.7.4->model-registry==0.2.6a1)
Requirement already satisfied: six>=1.5 in /opt/app-root/lib64/python3.11/site-packages (from python-dateutil<3.0.0,>=2.9.0.post0->model-registry==0.2.6a1) (1.16.0)
Requirement already satisfied: idna>=2.0 in /opt/app-root/lib64/python3.11/site-packages (from yarl<2.0,>=1.12.0->aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (3.10)
Installing collected packages: pydantic-core, eval-type-backport, annotated-types, pydantic, aiohttp-retry, model-registry
  Attempting uninstall: pydantic
    Found existing installation: pydantic 1.10.18
    Uninstalling pydantic-1.10.18:
      Successfully uninstalled pydantic-1.10.18
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
codeflare-sdk 0.21.1 requires pydantic<2, but you have pydantic 2.9.2 which is incompatible.
Successfully installed aiohttp-retry-2.8.3 annotated-types-0.7.0 eval-type-backport-0.2.0 model-registry-0.2.6a1 pydantic-2.9.2 pydantic-core-2.23.4

@jstourac
Copy link
Member

jstourac commented Oct 8, 2024

And this is what I got on the minimal image 2024.2:

$ pip install --no-index --find-links . model_registry-0.2.6a1-py3-none-any.whl
Looking in links: .
Processing ./model_registry-0.2.6a1-py3-none-any.whl
Requirement already satisfied: aiohttp<4.0.0,>=3.9.5 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (3.10.5)
Processing ./aiohttp_retry-2.8.3-py3-none-any.whl (from model-registry==0.2.6a1)
Processing ./eval_type_backport-0.2.0-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: nest-asyncio<2.0.0,>=1.6.0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (1.6.0)
Processing ./pydantic-2.9.2-py3-none-any.whl (from model-registry==0.2.6a1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.9.0.post0 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (2.9.0.post0)
Requirement already satisfied: typing-extensions<5.0,>=4.8 in /opt/app-root/lib64/python3.11/site-packages (from model-registry==0.2.6a1) (4.12.2)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (2.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (24.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/app-root/lib64/python3.11/site-packages (from aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (1.11.0)
Processing ./annotated_types-0.7.0-py3-none-any.whl (from pydantic<3.0.0,>=2.7.4->model-registry==0.2.6a1)
Processing ./pydantic_core-2.23.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from pydantic<3.0.0,>=2.7.4->model-registry==0.2.6a1)
Requirement already satisfied: six>=1.5 in /opt/app-root/lib64/python3.11/site-packages (from python-dateutil<3.0.0,>=2.9.0.post0->model-registry==0.2.6a1) (1.16.0)
Requirement already satisfied: idna>=2.0 in /opt/app-root/lib64/python3.11/site-packages (from yarl<2.0,>=1.0->aiohttp<4.0.0,>=3.9.5->model-registry==0.2.6a1) (3.8)
Installing collected packages: pydantic-core, eval-type-backport, annotated-types, pydantic, aiohttp-retry, model-registry
Successfully installed aiohttp-retry-2.8.3 annotated-types-0.7.0 eval-type-backport-0.2.0 model-registry-0.2.6a1 pydantic-2.9.2 pydantic-core-2.23.4

$ pip list | grep model
model-registry            0.2.6a1

Copy link
Member

@jstourac jstourac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't care how you install this but you probably had a reason to do it the original way. If you want to keep the original way, then update the download command so it utilizes Python 3.11 as I commented above, that means:

pip download --platform=manylinux2014_x86_64 --python-version=3.11 --abi=cp311 --only-binary=:all: --dest=/opt/app-root/src/  model_registry==0.2.6a1

With this change it should start to work on the latest 2024.2 images.

Copy link
Member

@jiridanek jiridanek Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm with @jstourac on this. I think the reason for the previous convoluted way is that running pip install directly won't work disconnected. Or that whether that works or not in disconnected depends on whether the packages are mirrored in the disconnected registry. And whether the env variable that shows the mirror's url is exported in the env.

Signed-off-by: lugi0 <lgiorgi@redhat.com>

Maybe Select Kernel
${is_kernel_selected} = Run Keyword And Return Status Page Should Not Contain Element xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]
IF not ${is_kernel_selected} Click Button xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]/..
${is_kernel_selected} = Run Keyword And Return Status Page Should Not Contain Element xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (148/120)
${is_kernel_selected} = Run Keyword And Return Status Page Should Not Contain Element xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]
IF not ${is_kernel_selected} Click Button xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]/..
${is_kernel_selected} = Run Keyword And Return Status Page Should Not Contain Element xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]
IF not ${is_kernel_selected} SeleniumLibrary.Click Button xpath=//div[@class="jp-Dialog-buttonLabel"][.="Select"]/..

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (123/120)
Signed-off-by: lugi0 <lgiorgi@redhat.com>
Get XY Python Version From Jupyterlab
[Documentation] Fetches the X.Y Python version from the current Jupyterlab instance
${output}= Run Cell And Get Output !python --version
${output}= Fetch From Right ${output} ${SPACE}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '=' instead
${output}= Run Cell And Get Output !python --version
${output}= Fetch From Right ${output} ${SPACE}
# Y and Z can be > len 1, split on "." instead of getting substring from indices
@{split_out}= Split String ${output} separator=.

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '=' instead
${output}= Fetch From Right ${output} ${SPACE}
# Y and Z can be > len 1, split on "." instead of getting substring from indices
@{split_out}= Split String ${output} separator=.
${vers}= Set Variable ${split_out}[0].${split_out}[1]

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '=' instead
@lugi0
Copy link
Contributor Author

lugi0 commented Oct 8, 2024

Implemented @jstourac 's suggestion, and also added a step to fetch the current python version in the jupyterlab environment in order to avoid this issue in the future.
@jiridanek the convoluted install is done to ensure that even in disconnected we are able to just install the library locally without having to go through pip - it is similar to mirroring in a local pip for the disconnected cluster, but we do it from the test directly

jiridanek
jiridanek previously approved these changes Oct 8, 2024

Get XY Python Version From Jupyterlab
[Documentation] Fetches the X.Y Python version from the current Jupyterlab instance
${output}= Run Cell And Get Output !python --version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do this

% python -c 'import sys; print(*sys.version_info[:2], sep=".")'
3.11

to avoid robot string manipulation at all costs. But it's been already coded differently, so I agree leave it be.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I wanted to fetch the string directly but a bunch of other tests rely on this keyword and I didn't want to change its behaviour too much in fear of breaking something. I think we can leave it be for now, RF's string manipulation is Python string manipulation at the end of the day

Copy link
Member

@jiridanek jiridanek Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$$\displaystyle {\hat {H}}\psi =E\psi$$

(everything that was, is, will be, and can be, is contained in the expression above, no need to write anything more)

(the $\LaTeX$ support in github is a fairly recent feature)

Signed-off-by: lugi0 <lgiorgi@redhat.com>
Copy link

sonarqubecloud bot commented Oct 8, 2024

@lugi0 lugi0 changed the title Install MR from pip instead of locally in jupyter notebook Dynamically change python target version for MR dependencies Oct 8, 2024
@lugi0 lugi0 requested a review from jiridanek October 8, 2024 14:01
@lugi0
Copy link
Contributor Author

lugi0 commented Oct 8, 2024

Tested locally after the rebase on #1899 and passing against RC build

@lugi0 lugi0 enabled auto-merge (squash) October 8, 2024 14:10
@lugi0 lugi0 merged commit 55a958c into red-hat-data-services:master Oct 8, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants