julia 1.8 beta3, NetCDF_jll v400.802.102+0: Failure to load OPENDAP URL with TLS

[Alexander-Barth]

Describe the bug

Failure to load OPENDAP URL with TLS

To Reproduce

using NCDatasets
NCDataset("https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1")

full output

julia> using NCDatasets
julia> NCDataset("https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1")
Error:curl error: SSL peer certificate or SSH remote key was not OK
curl error details: 
Warning:oc_open: Could not read url
ERROR: NetCDF error: Opening path https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1: NetCDF: I/O failure (NetCDF error code: -68)
Stacktrace:
 [1] nc_open(path::String, mode::UInt16)
   @ NCDatasets ~/.julia/dev/NCDatasets/src/netcdf_c.jl:273
 [2] NCDataset(filename::String, mode::String; format::Symbol, share::Bool, diskless::Bool, persist::Bool, attrib::Vector{Any})
   @ NCDatasets ~/.julia/dev/NCDatasets/src/dataset.jl:187
 [3] NCDataset (repeats 2 times)
   @ ~/.julia/dev/NCDatasets/src/dataset.jl:157 [inlined]
 [4] top-level scope
   @ REPL[2]:1

Expected behavior

No error

Environment

julia> versioninfo()
Julia Version 1.8.0-beta3
Commit 3e092a2521 (2022-03-29 15:42 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores

Full output

(v1.8) pkg> st --manifest
Status `/mnt/data1/abarth/.julia/environments/v1.8/Manifest.toml`
  [179af706] CFTime v0.1.2
  [da1fd8a2] CodeTracking v1.0.9
  [34da2185] Compat v3.43.0
  [864edb3b] DataStructures v0.18.11
  [692b3bcd] JLLWrappers v1.4.1
  [aa1ae85d] JuliaInterpreter v0.9.13
  [6f1432cf] LoweredCodeUtils v2.2.2
  [85f8d34a] NCDatasets v0.12.4 `~/.julia/dev/NCDatasets`
  [bac558e1] OrderedCollections v1.4.1
  [21216c6a] Preferences v1.3.0
  [ae029012] Requires v1.3.0
  [295af30f] Revise v3.3.3
  [0234f1f7] HDF5_jll v1.12.1+0
  [7243133f] NetCDF_jll v400.802.102+0
  [458c3c95] OpenSSL_jll v1.1.14+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [b27032c2] LibCURL v0.6.3
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.8.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [fa267f1f] TOML v1.0.0
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] LibCURL_jll v7.81.0+0
  [29816b5a] LibSSH2_jll v1.10.2+0
  [c8ffd9c3] MbedTLS_jll v2.28.0+0
  [14a3606d] MozillaCACerts_jll v2022.2.1
  [4536629a] OpenBLAS_jll v0.3.20+0
  [83775a58] Zlib_jll v1.2.12+1
  [8e850b90] libblastrampoline_jll v5.1.0+0
  [8e850ede] nghttp2_jll v1.41.0+1
  [3f19e933] p7zip_jll v16.2.1+1

[Alexander-Barth]

A workaround is to create a file .ncrc with:

HTTP.SSL.CAINFO=/etc/ssl/certs/ca-certificates.crt

The file path is the output of:

using NetworkOptions; NetworkOptions.ca_roots()

(updated to use NetworkOptions.jl instead of MozillaCACerts_jll, thanks @visr )

[visr]

Is it possible to set this through a ccall during __init__()? See also https://github.com/JuliaGeo/GDAL.jl/blob/2366302ee66a189aed673b03dd7da2a9589092e3/src/GDAL.jl#L37-L45. You can use NetworkOptions instead of MozillaCACerts_jll, the former is a stdlib.

[Alexander-Barth]

Unfortunately, I do not see a NetCDF API call (similar to GDAL's cplsetconfigoption) where this option can be set, only a configuration file.

https://github.com/Unidata/netcdf-c/blob/main/docs/auth.md

Maybe NCDatasets can create on start up a temporary config file and sets the environment variable DAPRCFILE accordingly.

I am wondering how it worked before without any configuration....

[visr]

For GDAL.jl we used to set GDAL_DATA as an environment variable in __init__, however this was not as reliable as setting it through a function. I believe there were issues with multiple workers and such. I'm wondering how reliable it would be to create a config file on startup. I don't know when that file is read, and if __init__ is sufficiently early.

For Proj.jl Linux builds also started failing with a certificate issue, but luckily they provided a function to set the path, besides a configuration file and an environment variable, so the fix was easy: JuliaGeo/Proj.jl@e50a7d7

Maybe we should ask if netcdf-c for a nc_set_ca_bundle_path function in the C API?

Also from the documentation link you shared, it says they go to curl_easy_setopt. Perhaps we can call that function directly from the libcurl JLL? Or try it out first using LibCURL.jl.

[Alexander-Barth]

Thank for thinking along 😀

Also from the documentation link you shared, it says they go to curl_easy_setopt. Perhaps we can call that function directly from the libcurl JLL? Or try it out first using LibCURL.jl.

It is not so clear to me which function you mean by "that function" in this sentence. The (to-be-implemented) nc_set_ca_bundle_path or curl_easy_setopt? It is not clear to be if the curl_easy_setopt has to be called for every network connection or it can be called once for all connections. It seems that former is the case, but I am not sure.

[visr]

Yeah in the last paragraph I was talking about curl_easy_setopt. I assume it would only need to be set once.

[Alexander-Barth]

The DAPRCFILE environment variables did not seem to work as I expected it. It seems that the DAPRCFILE environment variable was renamed NCRCENV_RC (as mentioned in PR #174)

julia-1.8 --eval 'using NCDatasets;  ENV["NCRCENV_RC"] = expanduser("~/.ncrc2");  show(NCDataset("https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1"))'
# works

However, if we would point to a different temporary configuration file, it would result in ignoring an already preexisting .ncrc file (with potentially e.g. credentials for opendap resources).

Edit: remove failed attempts with DAPRCFILE.

[visr]

However, if we would point to a different temporary configuration file, it would result in ignoring an already preexisting .ncrc file (with potentially e.g. credentials for opendap resources).

Unfortunately, yes. Given those downsides, this seems to work: fd6d08b

julia> NCDatasets.NCRC[]
"/home/visr/.julia/scratchspaces/7243133f-43d8-5620-bbf4-c2c921802cf3/config/.ncrc"

julia> String(read(ans))
"HTTP.SSL.CAINFO=/etc/ssl/certs/ca-certificates.crt"

I saw this comment in the netcdf-c code on configuration merging:

https://github.com/Unidata/netcdf-c/blob/3bf18fbdb85d38affc9a1e1d993d0bc546ba1de6/libdispatch/drc.c#L167-L177

But that means if we use NCRCENV_RC no merging will take place.