Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add write_pvolfile (see #470) #471

Merged
merged 5 commits into from
Jul 15, 2021
Merged

add write_pvolfile (see #470) #471

merged 5 commits into from
Jul 15, 2021

Conversation

adokter
Copy link
Owner

@adokter adokter commented Jun 29, 2021

replaces PR #470

@adokter adokter mentioned this pull request Jun 29, 2021
@adokter adokter added this to the 0.6.0 milestone Jun 29, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jun 29, 2021

Codecov Report

Merging #471 (06d74c0) into master (53d35b4) will increase coverage by 0.75%.
The diff coverage is 96.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #471      +/-   ##
==========================================
+ Coverage   55.72%   56.47%   +0.75%     
==========================================
  Files          54       55       +1     
  Lines        3015     3072      +57     
==========================================
+ Hits         1680     1735      +55     
- Misses       1335     1337       +2     
Impacted Files Coverage Δ
R/write_pvolfile.R 95.65% <95.65%> (ø)
R/pvolfile.R 90.24% <100.00%> (+0.24%) ⬆️
R/read_pvolfile.R 76.66% <100.00%> (+1.37%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53d35b4...06d74c0. Read the comment docs.

@adokter
Copy link
Owner Author

adokter commented Jun 29, 2021

@barthoekstra looking good! I noticed NEXRAD files (available here https://s3.amazonaws.com/noaa-nexrad-level2/index.html) are stored in a considerably larger size than the original. This is because the nexrad_to_odim() function doesn't track the original encoding and stores everything as 64-bit floating point, something that could be improved.

Is there an easy way to force write_pvolfile() to write up to a certain bit size. E.g. a 16 bit floating point would do the trick for most radar quantities and would be a good default.

@barthoekstra
Copy link
Collaborator

Is there some way to reliably identify NEXRAD specific polar volumes? I presume through the fields in the source attribute, but you'll know better than me what a robust identifier would be.

You could then write NEXRAD files using

h5createDataset(fid, dataname, dim(pvol$scans[[i]]$params[[j]]), H5type = "H5T_IEEE_F32LE")

You can consider replacing H5T_IEEE_F32LE with H5T_STD_I16LE or any of the other types listed here (or by running h5const("H5T")) to shave off even more of that filesize. If I understand it right — it's no straightforward reading material — there are no standard 16-bit floating point datatypes in the HDF5 spec.

@adokter
Copy link
Owner Author

adokter commented Jul 15, 2021

Now storing 64-bit floats as 32-bit floats for NEXRAD files, but this results only in a modest reduction of file size. Large file size for NEXRAD needs to be addressed in adokter/vol2bird#180

@adokter adokter merged commit 87307d9 into master Jul 15, 2021
@adokter adokter deleted the add_write_pvolfile branch July 15, 2021 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants