produceBlock takes so much time on mainnet

[twoeths]

Describe the bug

Sometimes it takes 9s to produce block:

1047041:Feb-28 19:45:35.129[rest]            debug: Req req-gsso produceBlockV2
1047160:Feb-28 19:45:44.222[rest]            debug: Res req-gsso produceBlockV2 - 200

I was not able to find which apis cause time in the log

Expected behavior

• Should log the time for produceBlock
• Investigate what caused the issue

Desktop (please complete the following information):

• Version: v1.4.2
• Network: mainnet

[twoeths]

this was caught in the REST api response time on Feb 28

confirmed in the log, there were 4 produceBlockV2 calls in total, the last one took almost 10s

155000:Feb-28 02:40:47.244[rest]            debug: Req req-foga produceBlockV2
155002:Feb-28 02:40:47.671[rest]            debug: Res req-foga produceBlockV2 - 200
560740:Feb-28 10:14:47.038[rest]            debug: Req req-g35r produceBlockV2
560743:Feb-28 10:14:47.634[rest]            debug: Res req-g35r produceBlockV2 - 200
782386:Feb-28 14:34:59.036[rest]            debug: Req req-gbhq produceBlockV2
782395:Feb-28 14:34:59.644[rest]            debug: Res req-gbhq produceBlockV2 - 200
1038917:Feb-28 19:40:48.058[rest]            debug: Req req-gkj0 produceBlockV2
1039048:Feb-28 19:40:57.945[rest]            debug: Res req-gkj0 produceBlockV2 - 200

however block production time were all < 1s

could be something related to block serialization

[twoeths]

was able to catch a big execution payload fetch time on rescue node

[twoeths]

could be something related to block serialization

confirm that streaming block json to vc takes time sometimes, see #5227 (comment)

[dapplion]

Closing non-actionable issue

• producing blocks is a sparse action, logs here are good to debug specific incidents Track produce block steps #5231
• solution 1 Use blinded flow in validator by default #5243
• solution 2 Encode bn - vc comms with SSZ #5244