We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arrow.write()
Mmap
Premise:
view()
This seems to take increasing memory as the content of the mask.
mask
I know this doesn't work correctly because if I set memory limit first:
> ulimit -Sv 8000000
julia> using Arrow, DataFrames julia> const df = @time DataFrame(Arrow.Table("./nanoAOD_nocomp.feather"); copycols=false); 2.685720 seconds (4.84 M allocations: 321.109 MiB, 4.41% gc time, 100.89% compilation time) julia> Arrow.write("/home/akako/Downloads/out.feather", @view df[1:1*10^4, :]); julia> Arrow.write("/home/akako/Downloads/out.feather", @view df[1:2*10^4, :]); ERROR: Internal error: encountered unexpected error in runtime: OutOfMemoryError() unknown function (ip: 0x7f9d7329fc99) unknown function (ip: 0x7f9d732935b5) jl_gc_alloc at /home/akako/Documents/github/dotFiles/homedir/.julia/juliaup/julia-1.9.0-rc1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1 (unknown line) ijl_alloc_array_1d at /home/akako/Documents/github/dotFiles/homedir/.julia/juliaup/julia-1.9.0-rc1+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1 (unknown line) unknown function (ip: 0x7f9d5ec6259e) unknown function (ip: 0x7f9d5e36c9dc) unknown function (ip: 0x7f9d5e73ee1f) unknown function (ip: 0x7f9d5e73ed98)
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Premise:
view()
over theMmap
-ed tableArrow.write()
to write filtered table to diskThis seems to take increasing memory as the content of the
mask
.I know this doesn't work correctly because if I set memory limit first:
The text was updated successfully, but these errors were encountered: