Block import error block sync

[akru]

Description

Our collators on Kusama catch following error:

2021-09-13 12:02:50 [Parachain] 💔 Error importing block 0x17ca5f93a6106a8a5ca9bd01c29fd8bb0b29e03a3e4e0ad9c7dd9504e0223546: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x1e73 - <unknown>!rust_begin_unwind\\n    1: 0x1bc0 - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b\\n    2: 0x1255 - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a\\n    3: 0x99a60 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::ha78c1d04b2717209\\n    4: 0x98532 - <unknown>!Core_execute_block\\nnote: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information\\n\")")))

And then block production on collators stoped,
node restart help but do you mind that block import error shouldn't break block production.
What could be an issue here?

[0x8f701]

I'm also interested in this error, encountered it once but never got what does it mean

[akru]

@bkchr may be you have some thoughts?

[bkchr]

Very likely: paritytech/substrate#9697

[akru]

I see, thank you @bkchr

[akru]

It looks critical because of blocks Shiden parachain block production at least three times for last month, do you have any suggestions?

[akru]

@bkchr that happens twice a week, currently node restart helps only, but may be we can somehow help to fix it? Should I launch the node in debug mode for example?

[bkchr]

I have spent enough time to look at the problematic code. The path forward is discussed in here: paritytech/substrate#9770

It is being worked on. Sorry for the problems!

[akru]

No problem @bkchr, thank you for your hard work each time, appreciate it!

[h4x3rotab]

Related:

• Khala: Khala v0.1.7 node got occasionally full network stale Phala-Network/phala-blockchain#551
• Shiden: Collator parachain node sync errors - Digest item must match that calculated AstarNetwork/Astar#503