feat(meta): wrap etcd client to be able to refresh conn when auth token expired

[yezizp2012]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

If auth enabled in etcd cluster and meta uses user/password to communicate with etcd, meta will crash when the etcd cluster encountered master/standby switchover or reboot.

2022-12-09T06:24:26.224266Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 1
thread 'risingwave-main' panicked at 'called `Result::unwrap()` on an `Err` value: Internal(grpc request error: status: Unauthenticated, message: "etcdserver: invalid auth token", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

But unfortunately Rust etcd library doesn't provide any token refresh APIs, and based on the library's implementation we can impossible refresh the token in all clients using auth interface. See etcdv3/etcd-client#45 for more details.

Currently the only way to achieve it is to create a new connection instead, which is what we did in this PR. Previously I think it might brings some overhead due to the clone of all requests, but since we have already introduced etcd_retry_client, we can leave it to the retry part.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #6872 (1d3fc11) into main (d0c0a55) will decrease coverage by 0.06%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #6872      +/-   ##
==========================================
- Coverage   73.22%   73.15%   -0.07%     
==========================================
  Files        1031     1032       +1     
  Lines      164642   164774     +132     
==========================================
- Hits       120553   120543      -10     
- Misses      44089    44231     +142     

Flag	Coverage Δ
rust	73.15% <0.00%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/meta/src/backup_restore/utils.rs	40.47% <0.00%> (ø)
src/meta/src/rpc/server.rs	79.07% <0.00%> (ø)
src/meta/src/storage/etcd_meta_store.rs	0.00% <0.00%> (ø)
src/meta/src/storage/etcd_retry_client.rs	0.00% <0.00%> (ø)
src/meta/src/storage/wrapped_etcd_client.rs	0.00% <0.00%> (ø)
src/batch/src/executor/group_top_n.rs	68.42% <0.00%> (-6.44%)	⬇️
src/common/src/types/ordered_float.rs	32.23% <0.00%> (+0.19%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[TennyZhuang]

Generally LGTM.

Can we give some feedbacks to upstream?

[yezizp2012]

Generally LGTM.

Can we give some feedbacks to upstream?

@arkbriar has opened an issue here etcdv3/etcd-client#45, I'm afraid the modification in this pr is only applicable to our project and It's quite difficult to achieve it in its library side. 🥵 Let's keep this workaround and see.

[arkbriar]

LGTM! Thanks for addressing this issue! cc. @mikechesterwang for awareness.

[BugenZhao]

LGTM