Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "install from conda forge" CI after ixmp release v3.10.0 #918

Merged
merged 1 commit into from
Feb 21, 2025

Conversation

glatterf42
Copy link
Member

@glatterf42 glatterf42 commented Feb 21, 2025

Unfortunately, it looks like the release v3.10.0 of ixmp broke the "install from conda forge" CI here, but only the tests running on windows machines.

Comparing ixmp 3.9 to 3.10, we see that the way we collect the GAMS information has changed. Not dramatically and on macos, everything still works fine, which makes me think I'm still missing something here. However, looking at the backtrace, we see this bit:

   File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\cli.py", line 59, in main
    mp = ixmp.Platform(name=platform)
  File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\core\platform.py", line 95, in __init__
    self._backend = backend_class(**kwargs)
                    ~~~~~~~~~~~~~^^^^^^^^^^
  File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\backend\jdbc.py", line 264, in __init__
    start_jvm(jvmargs)
    ~~~~~~~~~^^^^^^^^^
  File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\backend\jdbc.py", line 1253, in start_jvm
    gams_info().java_api_dir,  # GAMS system directory
    ~~~~~~~~~^^
  File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\model\gams.py", line 412, in gams_info
    _GAMS_INFO = GAMSInfo()
  File "D:\a\_actions\iiasa\actions\main\setup-conda\Anaconda3\envs\testenv\Lib\site-packages\ixmp\model\gams.py", line 90, in __init__
    output = check_output(
        ["gams", "null.gms", "-LogOption=3"],
    ...<2 lines>...
        universal_newlines=True,
    )

Indicating that gams is now called whenever an instance of an ixmp.Platform is created. However, gams is never in the PATH or even installed in the conda workflow. message-ix show-versions always reports 'gams' executable not in PATH, even for successful runs on windows in the past and with ixmp 3.10.0 on macos. This is another bit I don't quite understand: how could this possibly work?
With earlier versions of ixmp, we possibly didn't execute gams whenever creating a Platform, but today's runs already use ixmp 3.10.0, so should come with that requirement. The only reason I can immediately see for a difference from the windows tests is that check_output gets a different value for shell:

                output = check_output(
                    ["gams", "null.gms", "-LogOption=3"],
                    shell=os.name == "nt",
                    cwd=temp_dir,
                    universal_newlines=True,
                )

So possibly, modifying this argument would also do the trick.

For now, though, I copied the steps that install GAMS in other CI workflows to this one as it seems we should have it installed for ixmp to work properly.

How to review

  • Read the diff and note that the CI checks all pass.

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests; coverage checks both ✅
  • [ ] Add, expand, or update documentation. Just CI.
  • [ ] Update release notes. Just CI.
  • After approval, drop the "TEMPORARY" commit(s).

@glatterf42 glatterf42 added the ci Continuous integration label Feb 21, 2025
@glatterf42 glatterf42 self-assigned this Feb 21, 2025
Copy link

codecov bot commented Feb 21, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.6%. Comparing base (0038b04) to head (9350232).
Report is 6 commits behind head on main.

Current head 9350232 differs from pull request most recent head 6d7293b

Please upload reports for the commit 6d7293b to get more accurate results.

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #918   +/-   ##
=====================================
  Coverage   95.6%   95.6%           
=====================================
  Files         48      48           
  Lines       4401    4401           
=====================================
  Hits        4211    4211           
  Misses       190     190           

@glatterf42
Copy link
Member Author

Note: all conda-forge workflows are passing here and gams is recognized in all of them. The only thing I'm not sure of is if this is the most efficient setup (i.e. if a smaller set of changes would achieve the same thing already). However, I'd say if our workflow actually uses gams, it's good to install it ourselves instead of relying on some unknown magic trick that provides it sometimes. So in the interest of not seeing as many CI failures after the weekend, I'd like to merge the PR today, still.

@Tyler-lc Tyler-lc self-requested a review February 21, 2025 12:03
Copy link

@Tyler-lc Tyler-lc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@glatterf42 glatterf42 force-pushed the fix/install-from-conda-forge-ixmp-3.10.0 branch from 9350232 to 6d7293b Compare February 21, 2025 12:27
@glatterf42
Copy link
Member Author

@khaeru for context: I asked @Tyler-lc for a review here so that we could still merge the fix today. I believe this PR brings the fix we're looking for, but if you want to take another look on Monday, that's of course welcome, too.

@glatterf42 glatterf42 merged commit 42ef4e1 into main Feb 21, 2025
27 checks passed
@glatterf42 glatterf42 deleted the fix/install-from-conda-forge-ixmp-3.10.0 branch February 21, 2025 12:43
@khaeru
Copy link
Member

khaeru commented Feb 21, 2025

Thanks for spotting and diagnosing! I agree with the approach taken here.

The change from 3.9 to 3.10 is that we no longer package (very old) GAMS binary libraries with ixmp, but rather find and use the ones from the system GAMS installation.

These libraries are used by the Java code underneath JDBCBackend to interact with GDX files, mainly. In order for that Java code to know where to find them, we need to identify and supply the appropriate path at the moment of starting the JVM through JPype, which is why this happens the first time any JDBCBackend-ed Platform is instantiated.

No idea why this does not fail on macOS, but sufficient I think to make the changes to fix the Windows jobs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous integration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants