Dicussion: monitor network usage by ourselves and manage it

[lmatz]

In #7002 and also per the discussion on Slack https://risingwave-labs.slack.com/archives/C048NM5LNKX/p1672121457286999, try to generate more discussion here.

In short, we speculate that #7002 occurs because network usage exceeds certain limits.

As suggested by @zwang28, AWS EC2 has network limits and in particular, EC2 can exceed network limits with even low average network utilization if there are a lot of micro-bursts: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-exceeding-network-limits/, which leads to the question whether we should monitor network statistics by ourselves.

Besides, both getting state/data from S3, shuffling among multiple compute nodes and reading from external sources all consume the network, which suggests we need to manage the network in a unified way to avoid too much network usage happening?

[zwang28]

which leads to the question whether we should monitor network statistics by ourselves.

Just learned that we have already made use of prometheus/node_exporter. It has done these monitoring works.