Package naming scheme changed to use underscores without prior notice

[glaubitz]

While updating individual component packages of the Python Azure SDK on openSUSE Tumbleweed today, I noticed that the package naming scheme has changed to use underscores instead of hyphens (s/-/_/g):

refs/tags/azure_mgmt_largeinstance_1.0.0b1 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_frontdoor_1.2.0 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_cdn_13.1.0 Mon Apr 15 15:43:03 2024 +0800

I'm not sure whether this accidental or intentional, but in any case it's a breaking change for downstream distributions which have to adjust source archive names now.

Can this be reverted or or this there a valid reason to use underscores now (which are rather uncommon on PyPi in my experience)?

[weshaggard]

It looks like there was a tooling change to the setuptools package that caused this change.

@msyyc it seems like you by-passed the failed CI in the following PRs (#35200, #34980, #35201) and then released them with those failures which causes for incorrect release tags to be created. We should get those release tags fixed and we might even need to yank the packages if they are incorrect as well.

[msyyc]

Hi @weshaggard This is first time we meet these failures which are treated as fake warning by wrong. To meet the expected SDK release date, we bypass the CI check but now it seems "correct" error. Since we don't change anything about release pipeline, I don't understand why these failures happen. Appreciate if you could provide guidance how to fix it.

[lmazuel]

While updating individual component packages of the Python Azure SDK on openSUSE Tumbleweed today, I noticed that the package naming scheme has changed to use underscores instead of hyphens (s/-/_/g):

refs/tags/azure_mgmt_largeinstance_1.0.0b1 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_frontdoor_1.2.0 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_cdn_13.1.0 Mon Apr 15 15:43:03 2024 +0800

I'm not sure whether this accidental or intentional, but in any case it's a breaking change for downstream distributions which have to adjust source archive names now.

Can this be reverted or or this there a valid reason to use underscores now (which are rather uncommon on PyPi in my experience)?

@glaubitz see https://setuptools.pypa.io/en/stable/history.html#v69-3-0
@xiangyan99 is currently fixing it by pinning to an old setuptools, and we will discuss a plan to move forward with newer version (I don't know yet how we will approach it). Let me know if you have a preference, and we can factor that in our discussion.

[weshaggard]

To help with some of this I've pushed the following tags:

https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-largeinstance_1.0.0b1
https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-frontdoor_1.2.0
https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-cdn_13.1.0

I don't know if the packages that are published are going to be impacted or not but we at least need to have the tags fixed otherwise other systems start breaking. For now @xiangyan99 pinned the setuptools with PR #35212 but once @scbedd is back we probably need to spend more time investigating options.

[scbedd]

Relevant discussion in setuptools repo

It's definitely an intentional change on setuptools part. We should also update our package discovery to handle the new canonicalized names. That's not the part that actually concerns me though.

I'm currently digging into why our pinned setuptools version is getting overridden by the new setuptools version. Will close this issue after I understand the origin/update our wheel discovery logic to handle the new setuptools.

[scbedd]

Updates to the base image versions caused the installation of some global packages to not be updatable by a standard pip install. Even with force enabled.

Merged #35343 to resolve the weird pinning situation.

[glaubitz]

This issue has returned with azure-mgmt-appplatform and azure-mgmt-oracledatabase using underscores in their names while the latest version of azure-ai-ml still using dashes.

The behavior seems inconsistent now.

[weshaggard]

@scbedd could you please take a look https://github.com/Azure/azure-sdk-for-python/releases/tag/azure_mgmt_oracledatabase_1.0.0

[scbedd]

@scbedd could you please take a look https://github.com/Azure/azure-sdk-for-python/releases/tag/azure_mgmt_oracledatabase_1.0.0

1. Yes. This is due to the upgraded setuptools that I added this past Friday.
2. Will patch our builds to produce the correct file names, regardless of the setuptools normalization practices for now

[glaubitz]

Thanks for the fix! Could you retag these affected packages afterwards?

• azure_mgmt_edgezones
• azure_mgmt_oracledatabase
• azure_mgmt_appplatform

[scbedd]

Thanks for the fix! Could you retag these affected packages afterwards?

• azure_mgmt_edgezones
• azure_mgmt_oracledatabase
• azure_mgmt_appplatform

Absolutely. Apologies for inconvenience.

[glaubitz]

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

[scbedd]

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

[glaubitz]

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

Would it be possible to push the packages onto PyPi again? These can still be downloaded with the old filenames only.

[scbedd]

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

Would it be possible to push the packages onto PyPi again? These can still be downloaded with the old filenames only.

If you need the file names updated, I'd likely just cut a patch release for all 3. I can add additional wheel entries for a version, but can't republish the sdist to my knowledge. I'll double check that though.

EDIT: Yeah can't publish sdist twice.

[glaubitz]

If you need the file names updated, I'd likely just cut a patch release for all 3. I can add additional wheel entries for a version, but can't republish the sdist to my knowledge. I'll double check that though.

I suggest another release maybe with a "post1" suffix or so.

We're pulling the source tarballs semi-automatically from PyPi for updating the RPM packages of the SDK.

[scbedd]

@glaubitz sorry for delay hit some unexpected foibles.

I re-released with the post1 versions.

• azure-mgmt-edgezones
• azure-mgmt-appplatform
• azure-mgmt-oracledatabase

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

[glaubitz]

@glaubitz sorry for delay hit some unexpected foibles.

I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.

Will update these packages tomorrow as it's already late now here across the lake.

[scbedd]

@glaubitz sorry for delay hit some unexpected foibles.
I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.

Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

[glaubitz]

@glaubitz sorry for delay hit some unexpected foibles.
I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.
Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

I just had a go and I'm afraid to say the issue is not fully fixed.

While the tarball name has been corrected, the source folder name is still using underscores.

[scbedd]

@glaubitz sorry for delay hit some unexpected foibles.
I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.
Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

I just had a go and I'm afraid to say the issue is not fully fixed.

While the tarball name has been corrected, the source folder name is still using underscores.

Yeah I did just update the tarball. I didn't really want to go backwards on setuptools.

You mind pointing me at the automation? I'll help with a PR if you don't mind. Unfortunately part of the reason we hit this issue was due to a compatibility problem with older setuptools and the new agent images. Frankly I probably should have followed up with you to coordinate updating the automation for newer setuptools when this first occured. It slipped my radar unfortunately.

Re-opened till real resolution.

[scbedd]

Well, I got earlier an earlier version of setuptools working in our pipelines. Any further releases off of main should use setuptools==69.2.0 as well.

@glaubitz Not urgent. Whenever you get a chance, please try post2 versions:

• azure-mgmt-edgezones
• azure-mgmt-appplatform
• azure-mgmt-oracledatabase

Now that main is properly unblocked, I definitely want to help provide a patch for your automation. I don't want this to happen a 3rd time!

EDIT: leaving this issue open until confirmation of resolution this time.

[glaubitz]

Looks like another underscore-named tag just slipped in:

refs/tags/azure_search_documents_11.5.1 Tue Jul 30 10:45:28 2024 -0700

[scbedd]

Looks like another underscore-named tag just slipped in:

refs/tags/azure_search_documents_11.5.1 Tue Jul 30 10:45:28 2024 -0700

This was bad timing. The release branch forked off of main while the incompatible bits were present, so the released version included the underscores 👎

Can you please point me to your repo where your automation parses this value? I'd like to provide a patch if possible. While main will be fine for you until we intentionally upgrade setuptools, I can't guarantee that other patch releases don't ship and break you.

[scbedd]

Hey @glaubitz ? Unfortunately I'm now getting hit by CG alerts that are going to force me to upgrade past setuptools>70.

This is going to break you again, and while I really don't want to do that, I don't really have a choice given the mandates I'm under.

I've got some time to defer the CG alert, but I'd like to respond in the next couple of days. Can I please assist you with a patch so I don't break you when that occurs?

[glaubitz]

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

[scbedd]

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

Yeah, that version of azure-template you're seeing is a test release from this PR #37035 addressing some component governance alerts related to setuptools. This is a sdist normalization change that came with setuptools>=69.2.3. The tag will maintain our existing formatting, but the source distribution will begin to match the whl naming conventions.

I don't know where in your process you rationalize this, but I hope it will be as simple as

replace - in tag with _ to get expected tarball name.

If you point me to a place in your process I'm totally willing to help with that!

I will delay merge of #37035 a couple more days while we figure this out!

[glaubitz]

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

Yeah, that version of azure-template you're seeing is a test release from this PR #37035 addressing some component governance alerts related to setuptools. This is a sdist normalization change that came with setuptools>=69.2.3. The tag will maintain our existing formatting, but the source distribution will begin to match the whl naming conventions.

I don't know where in your process you rationalize this, but I hope it will be as simple as

replace - in tag with _ to get expected tarball name.

If you point me to a place in your process I'm totally willing to help with that!

I will delay merge of #37035 a couple more days while we figure this out!

OK, so if I understand you correctly now you consider the change from hyphen to underscore inevitable.

If that's the case, we'll just have to accept it and I have to adjust the upstream source names for future packages.

[scbedd]

OK, so if I understand you correctly now you consider the change from hyphen to underscore inevitable.

I do. this is the specific setuptools version where this name transition happens

With respect to #37035, I am going to be merging it today. I apologize for the disruption to your process that it will cause, but I need to get these alerts off my plate.

My offer for assistance remains open!

[xiangyan99]

We have merged the change,

Closing the issue and please feel free to re-open it if it is not solved.