Functionality enhancements to address lazy loading of chunked data, variable length strings, and other minor bug fixes#68

Open

bnlawrence wants to merge 121 commits intojjhelmus:masterfrom NCAS-CMS:wacasoft

Commits on Feb 22, 2024

Using s3 to get at some real data for testing
Bryan Lawrence
committed
Getting the address as well as size into the index
Bryan Lawrence
committed
With timer
Bryan Lawrence
committed

Commits on Feb 23, 2024

Not working yet. Don't reckon I have the arguments to OrthogonalIndexer right yet.
Bryan Lawrence
committed
A few more notes in the code so I can come back to it anon.
Bryan Lawrence
committed
Woops. Need this.
Bryan Lawrence
committed

Commits on Feb 24, 2024

First working lazy read (only reads chunks needed for selection)
Bryan Lawrence
committed
Woops didnt' commit the real oil
Bryan Lawrence
committed
Should now support filtering chunks in the partical chunk loading. Haven't got any tests around this yet.
Bryan Lawrence
committed

Commits on Feb 25, 2024

Some additional documentation
Bryan Lawrence
committed

Commits on Feb 29, 2024

Seems to work, prior to re-integration
Bryan Lawrence
committed

Commits on Mar 1, 2024

Moved chunk support into standard API
Bryan Lawrence
committed
removing playing code
Bryan Lawrence
committed
Merge branch 'jjhelmus:master' into issue6
bnlawrence
authored

Commits on Mar 3, 2024

Commits on Mar 5, 2024

Remove obsolete function
Bryan Lawrence
committed
Support for third party access to contiguous data address and size. Also remove list definition which breaks references.
Bryan Lawrence
committed

Commits on Mar 7, 2024

Commits on Apr 21, 2024

First cut, no tests yet
Bryan Lawrence
committed

Commits on Apr 22, 2024

Improvements
Bryan Lawrence
committed
With some failing tests
Bryan Lawrence
committed
Fixed one test
Bryan Lawrence
committed
All tests for new functionality pass, but I've broken something old
Bryan Lawrence
committed
Now passing all tests
Bryan Lawrence
committed
Checking coverage of get_chunk_info_by_coord(method)
Bryan Lawrence
committed
Missing docstring
Bryan Lawrence
committed

Commits on Apr 26, 2024

Cleaning up
Bryan Lawrence
committed

Commits on Jul 9, 2024

Merge pull request #5 from bnlawrence/h5pyapi
bnlawrence
authored
Ok, these were the pull request fixes that I thought I'd merged
Bryan Lawrence
committed
Merge branch 'h5pyapi'
Bryan Lawrence
committed

Commits on Jul 10, 2024

Adding Datatype and check_enum_dtype in a minimal manner - closes #8
Bryan Lawrence
committed

Commits on Jul 12, 2024

Commits on Jul 14, 2024

Committing to the dtype returned as a numpy dtype, and the extra h5t interface.
Bryan Lawrence
committed

Commits on Jul 19, 2024

Test for reference_list
Bryan Lawrence
committed

Commits on Dec 19, 2024

(New reference list test still broken) H5D has been disconnected from the parent dataobjects so once instantiated it does not need to touch the file metadata.
Bryan Lawrence
committed

Commits on Dec 20, 2024

Interim commit so we have something to point to in a discussion around the problem with references and caching everything we need in a variable.
Bryan Lawrence
committed

Commits on Dec 21, 2024

Transition to H5D cached backend is complete, though we still have the test_reference_list fail
Bryan Lawrence
committed
Removed obsolete DatasetDataObjects
Bryan Lawrence
committed

Commits on Dec 22, 2024

Expose package version in code, and separate testing requirements out in package metadata.
Bryan Lawrence
committed
Attempt to get b-tree logging in h5d
Bryan Lawrence
committed

Commits on Dec 28, 2024

Cleared a few bugs and misunderstandings which arose from workign with h5netcdf (closed #15 )
Bryan Lawrence
committed

Commits on Dec 30, 2024

Continue to use open file in h5d, closes #18
Bryan Lawrence
committed

Commits on Jan 1, 2025

Test for true bytes-io testing (needed for h5netcdf test compatability)
Bryan Lawrence
committed

Commits on Jan 2, 2025

Commits on Jan 3, 2025

Fix location of files so tests run properly from parent directory.
Bryan Lawrence
committed

Commits on Jan 7, 2025

Well, I think this is a fix for #23 , and it's so complex I'm committing it now in case my laptop dies and I can't repeat this :-)
Bryan Lawrence
committed

Commits on Jan 8, 2025

Cleaned up issue23 fixes, all tests pass
Bryan Lawrence
committed
Test localisation, and a new test for laziness outside a context manager
Bryan Lawrence
committed
Changes to support out of context variable access as described in #24
Bryan Lawrence
committed
Removing the pseudo chunking stuff that snuck into the last commit
Bryan Lawrence
committed
catching up to the main trunk in h5netcdf
Bryan Lawrence
committed
Starting to sketch out the pseudo chunking
Bryan Lawrence
committed
threadsafe data access
davidhassell
committed
merge from h5netcdf
davidhassell
committed

Commits on Jan 9, 2025

Commits on Jan 10, 2025

Commits on Jan 13, 2025

Commits on Jan 15, 2025

Merge pull request #26 from NCAS-CMS/mock_s3fs
bnlawrence
authored
Minor changes following V's S3 testing merge
Bryan Lawrence
committed
A framework for testing laziness.
Bryan Lawrence
committed
Merge branch 'h5netcdf' into h5netcdf-fh-2
davidhassell
committed
add test for threadsafe data access on posix and s3
davidhassell
committed
note on number of threadsafe test iterations
davidhassell
committed
Test framework for pseudochunking plus starting to migrate test data to data directory
Bryan Lawrence
committed
Ok, we pass the pseudochunking test, but we don't actually do it yet.
Bryan Lawrence
committed
Pseudo chunking in, with test support (and a missing make data file that I didn't move properly)
Bryan Lawrence
committed
Merge branch 'pseudo_chunking' into h5netcdf
bnlawrence
authored
Merge pull request #28 from NCAS-CMS/h5netcdf
bnlawrence
authored
Merge branch 'h5netcdf' into h5netcdf-fh-2
bnlawrence
authored
Merge pull request #27 from NCAS-CMS/h5netcdf-fh-2
bnlawrence
authored
Tidy up dependencies for testing
Bryan Lawrence
committed
Minor changes which come from upstream advice on my two pull requests jjhelmus#67 and jjhelmus#66
Bryan Lawrence
committed
Suppress reference list warning. It's useless
Bryan Lawrence
committed

Commits on Jan 16, 2025

Using context manager for threadsafe test
Bryan Lawrence
committed

Commits on Jan 20, 2025

no returned memory maps
davidhassell
committed
Test for #16 and #29
Bryan Lawrence
committed
More versions of the #29 tests
Bryan Lawrence
committed
Better .gitignore
Bryan Lawrence
committed
Giving up on in-memory netcdf tests for #29
Bryan Lawrence
committed
explicitly close POSIX files
davidhassell
committed

Commits on Jan 21, 2025

vlen strings data test case, vanilla version, and version with missing defined. tests fail.
Bryan Lawrence
committed
add extra posix test for file closure
davidhassell
committed
Merge branch 'h5netcdf' into fix-memmap
davidhassell
authored
More on h5d and testing. The iter_chunks method is broken and we now have a test which shows that.
Bryan Lawrence
committed
Support for pyactivestorage via a bespoke get_chunk_info_from_chunk_coord method
Bryan Lawrence
committed
better ignore
Bryan Lawrence
committed
The first vlen data test passes with this code
Bryan Lawrence
committed
closer to a solution for #29 . These tests pass, but we need to deal with multidimensional arrays as well.
Bryan Lawrence
committed

Commits on Jan 22, 2025

Commits on Jan 23, 2025

Commits on Jan 24, 2025

vlen related fixes
davidhassell
committed

Commits on Jan 28, 2025

Commits on Jan 30, 2025

Merge branch 'master' into wacasoft
Bryan Lawrence
committed
install only in test mode
valeriupredoi
committed
actual correct name for testing regime
valeriupredoi
committed