Recently upgraded to clj-http 3.10.0 and :as :json not working #489
Comments
|
Same here: (:body (debug (http/get (debug (str endpoint api-path object-type))
(debug {:as :json
:basic-auth [username password]
:query-params parameters}))))))The inner It works fine for the URLs which return very little data, but fetching the first URL that outputs more (5.5k characters according to |
|
Same here. Weirdly, ":as :json-strict" works. Deps:
|
|
Same here:
It seems that response body returned from http post function is a thing that lazily parses the stream. I replaced (let [response (http/post url {
:content-type :json
:accept :json
:as :byte-array
:socket-timeout 5000
:conn-timeout 5000})
(let [body-string (slurp (:body response))
body (json/parse-string body-string true)]
;; now body is parsed correctly without errors
)
)So it would seem to be a some kind of hard to hit race condition. I have been using this version of clj-http since relase in various servers without any problems. Today I hit a case where this fails about 50% of time. A server process running on a fast server was moved to small Docker container and this problem started to appear. This kind of stack trace happens after the request is made when I am processing the body. |
|
The reason seem to be that I think that |
|
I'll work on a PR to fix this on tomorrow. |
|
I took a deeper look at this issue, and I see two potential solutions -- both have merits, but have pros/cons for users who just want to get something done. The current nomenclanture Here comes the options. Option 1: Do not close .stream when using Option 2: Make From an end user POV, I would prefer option 2 because it is a much saner default. However, I think option 1 is also a nice because the fix is easy, but it has caveats of implicit connection management. |
|
I would definitely choose option 2. This is what I would expect as a user. Also not closing connections automatically could lead to weird/hard to detect bugs. |
|
Just bumped into this too. I was thinking that there should be some way to close the stream only after the lazy seq has been realized, but for now I'll be using @jsyrjala @rymndhng - I think that not closing the stream would be something unexpected by the user; as a rule if you created the stream you should also close it; delegating this to the user would introduce bugs. |
|
@ignorabilis I think the easiest solution is to replace |
|
I encountered the same issue. After looking around I found it's been addressed in #507, but not merged yet, any ETA to that fix? |
|
Hey! This is biting me too :-( I'm trying to batch process a response that contains thousands of JSON objects and it's killing me by having the stream closed. @dakrone please can you consider doing a release to address this issue, for your fans! :-) |
|
Experiencing the same issue @dakrone . |
|
Sorry that this has taken so long, I've merged the PR and released 3.10.1 with the fix. Big thanks to @rymndhng for providing the fix. |
|
I'm with 3.10.1 and get NullPointerException when trying to us |
|
@erez-rabih What version of I think you may be seeing a NPE because the version you are using is older than |
|
you are right |
|
Hi, Not explicitly, but it is mentioned in the changelog: https://github.com/dakrone/clj-http/blob/3.x/changelog.org#3101 |
|
I think we could add it to this section to be clearer about this |
Greetings - we recently upgraded to clj-http 3.10.0. The following doesn't work in 3.10.0, but it worked in 3.9.1. In 3.10.0, we get a java.io.IOException Stream Closed error
If we remove :as :json, we don't get that error, but the body isn't coerced to a clj map. I'm planning on looking through changes leading up to 3.10.0, but before I did that, I figured that I'd post the error for early awareness.
Originally posted by @ecolui in #475 (comment)
The text was updated successfully, but these errors were encountered: