francy 2.0.0 release is missing tarball and/or has wrong metadata

[fingolfin]

The francy 2.0.0 PackageInfo.g file indicates that the release archive is at https://github.com/gap-packages/francy/releases/download/v2.0.0/francy-2.0.0.tar.gz but that gives a 404 error. Indeed https://github.com/gap-packages/francy/releases/tag/v2.0.0 has no custom assets, only the auto-generated tarballs. Please do not use those for releases, as their checksums are not guaranteed to be stable -- while they are for now (there was a big kerfuffle about this some time ago), it is a bad practice anyway.

(If you were using ReleaseTools that would automatically take care of all that)

[mcmartins]

Hmm, last release was almost 4 years ago and I looked at my custom release scripts but didn't understand why I was uploading a package with (almost) the same content of the auto-generated tarballs (shrug).

I noticed there is a ticket to create a release tools github action, is this coming out soon(ish)?

I'll make sure I can use the Release Tools to release this project.

[fingolfin]

Hmm, last release was almost 4 years ago and I looked at my custom release scripts but didn't understand why I was uploading a package with (almost) the same content of the auto-generated tarballs (shrug).

It is common these days to store cryptographic checksums of sources archives to ensure integrity (both against transmission error and against attacks). This is not possible with the generated tarballs, as their checksum is not guaranteed to be stabke.

More important, GAP packages are supposed to bundle generated files, such as their documentation. That's not possible with repository snapshots, unless one stores the generated files in the repository, which is considered a bad practice for other reasons. In any case, francy does not seem to do it, so the generated tarballs don't include the documentation and hence GAP help for francy functions won't be available (assuming any exists, I did not check).

I noticed there is a ticket to create a release tools github action, is this coming out soon(ish)?

Nobody is working on this AFAIK.

I'll make sure I can use the Release Tools to release this project.

Let me know if there are any problems, I'll be happy to help.

[mcmartins]

Hi @fingolfin , I'm trying to use the release tools directly on a GitHub action but getting a weird error on the gap manual generation: https://github.com/gap-packages/francy/actions/runs/4319116764/jobs/7538201127

The strange thing is that the GitHub action to generate the documentation generates the pdf but the release tools do not, what am I doing wrong? Any idea?

Thanks!

[fingolfin]

You must install texlive if you want the PDF docs. Try something like

sudo apt-get install --no-install-recommends texlive-latex-base texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended

(Just out of curiosity, why do you re-create gh-pages from scratch each time? That makes it impossible to apply any project specific modifications to the templates?)

[fingolfin]

See PR #97

[fingolfin]

Still needs a release.

But I don't really understand your RELEASE.yml. It seems to trigger on every push to master? But how does it determine when to make a release? I would have expected it to trigger on tag events, for example.

[fingolfin]

Also, it seems to fail accessing gh-pages. That is probably because actions/checkout by default only makes a shallow clone, so no branch refs will be in there.

[mcmartins]

The idea was that a branch is only merged into master for a release. Meaning, master is always up-to-date with the latest release, and all work happens on some other branch and when one is happy they merges/pull request into master and everything runs from there. Since the GAPReleaseTools handle all the creation of tags etc, this could work.
Now, if one needs to fix a text file or anything that does not require a a new version of the software, well that will try to create a new release anyway which will fail. I don't think this is a good approach.

How are other package maintainers releasing their packages?

[fingolfin]

Good news: gap-system/PackageDistro#731 has picked up Francy 2.0.2.

Bad news: there is a test failure:

########> Diff in /home/runner/gap/pkg/francy-2.0.2/tst/canvas.tst:4
# Input is:
FrancyId(canvas);
# Expected output:
"FB9434EB1D9C3443A9C319E151A4721EC"
# But found:
"FDF0DE114906544908458A18956D837D0"
########

Note that the test suite is run twice there: once "normally" (and this passes), and then again with a GAP started with -A (so only a minimal set of packages is needed), and then LoadPackage("francy" : OnlyNeeded); is used to ensure only dependencies declared as "needed" are loaded, but not any that are marked as "suggested".

[fingolfin]

You also asked:

How are other package maintainers releasing their packages?

The vast majority of packages are released by manually running ReleaseTools on a local computer, at a time when the package maintainer(s) consider it ready to be released (in particular after updating the version and release date in PackageInfo.g).

These package in particular use their main/master branch for all development, and just tag releases there (resp. let ReleaseTools create those tags). Branches are just used to drive pull requests. The primary branch is usually always in a releasable state.

Of course it'd be nice to also be able to do it via GH Actions. You have developed a nice proof of concept, but I think it has issues, in particular the fact that it is run on every update to the master branch without any provisions to check that tags don't exist and that the metadata in PackageInfo.g is up-to-date.

For GAP itself, we do create releases via GH Actions, but we use a "tag" as trigger: so once I am ready to make a release, I do something like git tag -m "Version 1.2.3" v1.2.3 && git push v1.2.3 and then the CI job starts. See https://github.com/gap-system/gap/blob/master/.github/workflows/release.yml (we also run that job on PRs as well as daily, to ensure the release automation works, but those "releases" are using a special temporary version and are not uploaded to the release file system etc.)

[mcmartins]

Thanks for the comments @fingolfin .
As for the tests I thought the ‘RANDOM_SEED(1);’ would fix these, and strangely it does when running on the CI action. Is it because it runs the tests twice in the same session? In that case I’ll probably just remove the tests that check the IDs of the components.

As for the releases, I’m not fond of releasing from a personal computer as it is not a good practice. I do agree having a release on each commit to master is not ideal, but perhaps this can be easily implemented (if not already) on the gap release tools, basically it handles everything except the tag, and the tag is then the trigger for the release action. Similar to the gap system release. I’ll fix those tests and then review this release action according to what we’re discussing here.

[fingolfin]

As for the tests I thought the ‘RANDOM_SEED(1);’ would fix these, and strangely it does when running on the CI action.

This is an undocumented internal function, as its name already suggests. It does not reset all RNGs used by GAP either...

Instead, your .tst files should start with START_TEST and end with STOP_TEST (yes, those names also look "internal", but despite this, they are documented in the reference manual, and widely used in .tst files). These functions, among other things, properly reset the RNGs.

Is it because it runs the tests twice in the same session?

No, the test runs are in separate session. It wouldn't be possible to do what I described: remember, the second run is done with a minimal set of packages loaded, and we can't unload packages.

I can easily reproduce the problem locally by following the procedure I outlined above:

$ gap -A -b
gap> TestPackage("francy" : OnlyNeeded);
Architecture: aarch64-apple-darwin21-default64-kv8

testing: /Users/mhorn/Projekte/GAP/repos/pkg/gap-packages/francy/tst/callbacks.tst
       1 ms (0 ms GC) and 301KB allocated for callbacks.tst
testing: /Users/mhorn/Projekte/GAP/repos/pkg/gap-packages/francy/tst/canvas.tst
########> Diff in /Users/mhorn/Projekte/GAP/repos/pkg/gap-packages/francy/tst/canvas\
.tst:4
# Input is:
FrancyId(canvas);
# Expected output:
"FB9434EB1D9C3443A9C319E151A4721EC"
# But found:
"FDF0DE114906544908458A18956D837D0"
########
...

In that case I’ll probably just remove the tests that check the IDs of the components.

As for the releases, I’m not fond of releasing from a personal computer as it is not a good practice.

I am on the fence on this. I see good reasons for doing it in a controlled environment, which just so happens is relatively easy to achieve via GH Action (up to a point, at least). It could also potentially remove a bunch of usage errors by certain package maintainers who keep being unable to follow a step-by-step instruction ;-).

At the same time, full automation also means that the automation must satisfy even higher demands for robustness and reliability. For example I wouldn't feel comfortable recommending this with e.g. some sort of check that the package metadata is plausible (e.g. release date matches current date, version is bigger than the previous one, etc.). I am also afraid people will "fire & forget", i.e.: create a tag and push it, then never notice that their CI job failed (e.g. due to those random glitches GH Actions are experiencing more and more, were, say, it failed to clone the repository and thus dies). With ReleaseTools run locally, you just see if it worked or not.

Don't get me wrong, all those could be solved, I am just trying to express that to me, it is not as clear-cut a win for "do it in the cloud" over "do it on your local computer" as it seems to be for you.

I do agree having a release on each commit to master is not ideal, but perhaps this can be easily implemented (if not already) on the gap release tools, basically it handles everything except the tag, and the tag is then the trigger for the release action. Similar to the gap system release.

Sure it could be implemented. Just someone has to take the time to do it and thoroughly test it. Which is precisely why I just made an issue "would be nice to have" instead of just doing it...

It could also be taught to directly use the GitHub token from the environment.

It should also ideally verify the name of the tag (say v1.2.3) matches the version in the PackageInfo.g (should be easy, of course, 95% of what's needed is in place)

I’ll fix those tests and then review this release action according to what we’re discussing here.

Great!

[fingolfin]

I hope to release GAP 4.13 in mid or late April. It'd be great if we could get francy sorted out in time for that...

(I might have to postpone the GAP release to May anyway, we'll see but I am sure you'd be happy to have this off your plate, too ;-) ).

Please let me know if I can be of assistance.

[mcmartins]

@fingolfin thanks for the reminder!

I'll be back home right after Easter and hopefully will be able to finish this by mid/late April.

[mcmartins]

@fingolfin I've updated the release action, which now just handles NPM and PYPI.org packages. It runs when a release is published (which I shall change to created instead, because running ReleaseTools with --force will re-publish and thus re-run the action).
So now to trigger a release one should make the changes -> update the PackageInfo.g version, and the js versions as per the documentation and then, after merging and pushing this into master, run ReleaseTools locally.
Hopefully this is a bit more aligned with other packages.
Please let me know if you find something odd that must be fixed for the gap-system release.

[fingolfin]

Great! The 2.0.3 release was picked up, passed all tests, and has now been accepted into the GAP package distro. :-)