fix(block-cache): retry when other thread cancel future or meet error

[Little-Wallace]

Signed-off-by: Little-Wallace bupt2013211450@gmail.com

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

When several thread try to fetch the same block from s3, we would only allow one thread to send request to s3 and let other thread wait this thread. But the thread which send request may be canceled by upper caller , or it may meet IO error and other threads would receive RecvError and failed. We do not hope one thread cancel and make all request failed, so I make the request retry when receive an RecvError.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #5898 (25d7705) into main (938f427) will decrease coverage by 0.01%.
The diff coverage is 85.50%.

@@            Coverage Diff             @@
##             main    #5898      +/-   ##
==========================================
- Coverage   74.89%   74.87%   -0.02%     
==========================================
  Files         924      924              
  Lines      147014   146941      -73     
==========================================
- Hits       110104   110023      -81     
- Misses      36910    36918       +8     

Flag	Coverage Δ
rust	74.87% <85.50%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/frontend/src/handler/query.rs	20.49% <0.00%> (ø)
...ntend/src/optimizer/plan_node/stream_table_scan.rs	96.93% <ø> (ø)
src/frontend/src/scheduler/error.rs	8.33% <ø> (ø)
src/frontend/src/scheduler/local.rs	0.00% <0.00%> (ø)
src/stream/src/executor/aggregation/agg_group.rs	82.69% <ø> (ø)
src/frontend/src/scheduler/distributed/stage.rs	21.55% <33.33%> (ø)
...ge/src/hummock/local_version/local_version_impl.rs	81.77% <63.63%> (-0.22%)	⬇️
src/frontend/src/scheduler/plan_fragmenter.rs	76.57% <74.24%> (-8.04%)	⬇️
src/storage/src/hummock/shared_buffer/mod.rs	91.58% <92.85%> (+0.17%)	⬆️
src/storage/src/hummock/block_cache.rs	73.50% <94.44%> (+1.60%)	⬆️
... and 17 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more