-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use a cache to store PySM3 data files #147
Conversation
Not relevant for the PR, but just to keep a record of my tests and to keep in mind the amount of storage we can save with different compression schemes. It's also a testimony to Zstandard's awesomeness! I downloaded the PySM3 data archive and create a tarball:
and then I compressed
Zstandard is the winner here. However, the most impressive result comes from compression speed:
Not only Zstandard has achieved the best compression ratio, but it has performed the compression in a fraction of the time required by the other two algorithms. |
There is a problem here, because the cache gets saved and restored as desired, but it is never accessed when running the After a few hours of debugging, I discovered that this happens because the environment variable @NicolettaK, if I understand correctly, you are using this feature so that we can pass our own CMB realizations to PySM3, is this correct? |
@NicolettaK , may you please have a look at mbs.py, line 568? I would like to remove the line where the code changes the value of the environment variable |
Testing the use of |
As it is often the case that PySM3 data files are not available due to network outage, this PR uses https://github.com/actions/cache to cache it. It should be quite efficient, as cache files are compressed using Zstandard, which in my tests showed very good compression ratios for PySM3 maps.
I use here the trick explained in the PySM3 User's Manual