Back Blaze Multi threaded upload will have a recoverable error - fails anyway #8383
Comments
zackees
changed the title
These errors are almost always caused by an overloaded proxy somewhere in the chain returning HTML so they are worth retrying.
|
Thanks for your log with Anyway I have unraveled what is going on. First rclone sends an upload part request Next backblaze replies with a redirect to an internal error page which is odd. I would have expected a 500 error here. Which rclone then fetches And the result is at 200 OK with The body of that is an HTML page which you'll see if you go to the redirected URL in your browser https://www.backblaze.com/status/internal-error The problem here is that backblaze did not return a 500 error, it sent a 301 redirect to an HTML error page which rclone fetched with 200 OK. It then attempted to parse that as JSON and hence the error The API is supposed to return 500 error codes, not human readable HTML so this is really a bug in the Backblaze B2 API. Maybe @metadaddy could take a look? Anyway if we want to make a workaround for this, then what we need to do is detect that the server has sent us an HTML page instead of JSON when we are trying to decode the JSON. Have a go with this and see if it works v1.70.0-beta.8595.53bbe891d.fix-8383-b2-error on branch fix-8383-b2-error (uploaded in 15-30 mins) |
|
I built from source in the branch right now I'm at 9GB of a 9TB download so it looks like the issue has been fixed! |
|
Great news! You should still see those Logs in the debug log but they should be retried now. Do you? |
|
Nick, check your inbox |
|
The errors are being retried in your logs which is good. However I'm not seeing the error I expected though which should be something like Which makes me think those logs might not be from the version above? |
|
I think I spoke too soon. The error does not appear to be fixed. |
|
Can you send me a log of that please? |
|
Next week when I'm back in the office |
Continuation of:
#8315 (comment)
PR for fix: #8384
Almost like clockwork, rclone will fail at 3% for this file below. Even when the b2 concurrency value is set to just one.
@ncw mentioned that it looks like a proxy error with the HTML being sent back as an error code, or vice versa.
Looking at this logs, retry=False was being triggered for this error, borking the entire multi-part download consisting of 4k upload parts. Yes, if any one of those upload parts fails then the entire thing fails.
The reason is this: low-level retries are ONLY enabled if and only if, the error type and message can evaluate for true in list of predicate that will be run on the error. If no rule returns true, then the retries=False is returned.
This means that any unexpected error not covered by the white list will bypass the low level retries. That's what happened in the B2 case.
My fix does not address the bigger picture - it only adds this specific error type to the white list.
It may be useful to consider that retry=True be returned for any 500 "internal error" status code returned for an upload part.
The text was updated successfully, but these errors were encountered: