khala-node crashes and stop sync

[jimiflowers]

Hi. I'm having this error before a khala-node crash and stop sync:

{"log":"2022-03-24 19:34:56 [Parachain] Block prepare storage changes error:\n","stream":"stderr","time":"2022-03-24T19:34:56.152741038Z"}

{"log":"RuntimeApiError(Application(Execution(Other(\"Wasm execution trapped: wasm trap: wasm 'unreachable' instruction executed\\nwasm backtrace:\\n 0: 0xa09a - \u003cunknown\u003e!rust_begin_unwind\\n 1: 0x2579 - \u003cunknown\u003e!core::panicking::panic_fmt::h8f11323637b4db3b\\n 2: 0x28c925 - \u003cunknown\u003e!core::panicking::panic_display::hedf123f608c6d14b\\n 3: 0x28a5d1 - \u003cunknown\u003e!frame_executive::Executive\u003cSystem,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade\u003e::execute_block::hd5fa63f32fff0cdb\\n 4: 0x2add8a - \u003cunknown\u003e!Core_execute_block\\n\")))) \n","stream":"stderr","time":"2022-03-24T19:34:56.152756782Z"}

{"log":"2022-03-24 19:34:56 [Parachain] 💔 Error importing block 0x52607a6bd7da05150d6134df8bd86e4fdb54b27e2289dc5ce97b1c3df97f038c: Err(Other(ClientImport(\"Error at calling runtime api: Execution failed: Other(\\\"Wasm execution trapped: wasm trap: wasm 'unreachable' instruction executed\\\\nwasm backtrace:\\\\n 0: 0xa09a - \u003cunknown\u003e!rust_begin_unwind\\\\n 1: 0x2579 - \u003cunknown\u003e!core::panicking::panic_fmt::h8f11323637b4db3b\\\\n 2: 0x28c925 - \u003cunknown\u003e!core::panicking::panic_display::hedf123f608c6d14b\\\\n 3: 0x28a5d1 - \u003cunknown\u003e!frame_executive::Executive\u003cSystem,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade\u003e::execute_block::hd5fa63f32fff0cdb\\\\n 4: 0x2add8a - \u003cunknown\u003e!Core_execute_block\\\\n\\\")\"))) \n","stream":"stderr","time":"2022-03-24T19:34:56.152760848Z"}

The only way to recover from error is restart khala-node container.

I'm using solo-mining-scripts and running v1110.

Lastest blocks that caused this kind of error:

1383708, 1384714, 1384921, 1397861, 1402958, 1410103 and 1413384

[jasl]

Sorry this is a known Substrate bug that need upstream fix (the discussion here paritytech/cumulus#607)

we've logged here Phala-Network/phala-blockchain#551

restart can temporary fix

for workaround, you can add --state-cache-size 0 to khala-node but it may impact some performance, but it's acceptable when your PRuntime has synced

[jimiflowers]

Sorry this is a known Substrate bug that need upstream fix (the discussion here paritytech/cumulus#607)

we've logged here Phala-Network/phala-blockchain#551

restart can temporary fix

for workaround, you can add --state-cache-size 0 to khala-node but it may impact some performance, but it's acceptable when your PRuntime has synced

Hi Jasl. I had not seen it, the bug is very old, it is strange that it is not yet resolved. Thanks for the info.

[jasl]

Sorry this is a known Substrate bug that need upstream fix (the discussion here paritytech/cumulus#607)
we've logged here Phala-Network/phala-blockchain#551
restart can temporary fix
for workaround, you can add --state-cache-size 0 to khala-node but it may impact some performance, but it's acceptable when your PRuntime has synced

Hi Jasl. I had not seen it, the bug is very old, it is strange that it is not yet resolved. Thanks for the info.

It was indroduced about half a year ago I guess, this is the root cause that full Khala network get stuck