Gas Price estimation is unstable

[junderw]

System information

Geth version: v1.10.0 (but this issue has been persisting for a while across many versions)
OS & Version: Linux

Expected behaviour

Gas Price estimation should be at least semi-consistent. Sudden drops of 97% should not be happening.

Actual behaviour

We have multiple mainnet geth nodes (for our prod, staging environments etc.) and they all consistently dip in their estimations from the "market price" to low-single-digit gwei. See the following graph for a sample.

This usually only happens during times of increased network congestion. But sometimes we get the occasional dip-to-5-gwei once in a while

Steps to reproduce the behaviour

• Run geth 24/7
• Poll gas price once every minute and plot a graph
• See huge valleys of dipping down to single-digit gwei mostly consistent across multiple instances

Workaround

• Detecting abnormal valleys and triggering a geth restart usually fixes the issue for a couple of hours.

[junderw]

Our infrastructure engineer said that this might be occurring after blocks where abnormally low gas price txes have made it in the block (due to connections with miners etc.)

Which could make sense if we assume the occasional min-timing between instances was due to sync lag on one of the instances.

Also makes sense since the CPU, IOWAIT, etc. metrics are pretty all over the place between instances where as these valleys are mostly in lock step between all the instances.

[bulgakovk]

Also faced the same issue and posted a possible root reason here: #22705 (comment)

[holiman]

This should be closed by #22722 , feel free to reopen if the problem persists