Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arbitrum One Nitro not synced correctly with mainnet . Docker image: offchainlabs/nitro-node:v2.3.3-6a1c1a7 #2212

Closed
trungthanhnt opened this issue Apr 1, 2024 · 35 comments

Comments

@trungthanhnt
Copy link

Describe the bug
The chain is not sync when I run with docker
These are some logs :
INFO [04-01|02:36:54.769] catching up to chain batches localBatches=582,290 target=582,752
WARN [04-01|02:36:54.778] error reading inbox err="failed to get blobs: error fetching blobs in 19501975 l1 block: error calling beacon client in blobSidecars: response returned with status 404 Not Found, want 200 OK. body: {"message":"Block not found: block not found: no block roots at slot 8702476","code":404}"

To Reproduce
Steps to reproduce the behavior:
docker run -v /home/bitcoin/Arbitrum/data:/home/user/.arbitrum --restart=always --name="ArbitrumOneNitro" -d
--shm-size=128G --memory=256G --ulimit nofile=1024000:1024000
--net=mynetwork --ip=10.xxx.1.xx -p 10.xxx.1.1:8547:8547 -p 10.xxx.1.1:8548:8548 offchainlabs/nitro-node:v2.3.3-6a1c1a7 --parent-chain.connection.url http://10.xxx.1.1:8545
--parent-chain.blob-client.beacon-url=http://10.xxx.1.1:3500
--chain.id=42161 --http.api=net,web3,eth --http.corsdomain=* --http.addr=0.0.0.0 --http.vhosts=* --init.url="https://snapshot.arbitrum.foundation/arb1/nitro-pruned.tar"

My L1 works well with latest block. We use Prysm version 5.0.0 for beacon chain , the firewall already set to connect well to both beacon rpc and L1 RPC

@Tristan-Wilson
Copy link
Member

Hi, this looks like an issue with your Prysm node not having the blobs. Please upgrade your Prysm node to the latest release, currently 5.0.2 as it contains fixes related to blob handling: https://github.com/prysmaticlabs/prysm/releases

Also take a look at this issue, I think you need to enable the backfill flags: prysmaticlabs/prysm#13783

@ZYS980327
Copy link

嗨,这看起来像是您的 Prysm 节点没有 blob 的问题。请将 Prysm 节点升级到最新版本,当前为 5.0.2,因为它包含与 blob 处理相关的修复: https://github.com/prysmaticlabs/prysm/releases

也看看这个问题,我认为你需要启用回填标志:prysmaticlabs/prysm#13783

My question is this “failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"”

@ZYS980327
Copy link

INFO [04-11|01:18:25.247] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13092 state="{BlockHash:0x1aa5ab7a54c9cb55eac61213665266ea38baca7efba59f73788d22b113c2638b SendRoot:0x55cf415773b9e2c845f63a906c665e03a1ca781bd3f7fe41e3838738c5fba678 Batch:586040 PosInBatch:448}"
WARN [04-11|01:18:33.062] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"
INFO [04-11|01:18:33.449] catching up to chain batches localBatches=579,554 target=586,041
INFO [04-11|01:19:25.286] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13092 state="{BlockHash:0x1aa5ab7a54c9cb55eac61213665266ea38baca7efba59f73788d22b113c2638b SendRoot:0x55cf415773b9e2c845f63a906c665e03a1ca781bd3f7fe41e3838738c5fba678 Batch:586040 PosInBatch:448}"
INFO [04-11|01:19:33.483] catching up to chain batches localBatches=579,554 target=586,041
INFO [04-11|01:20:25.320] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13092 state="{BlockHash:0x1aa5ab7a54c9cb55eac61213665266ea38baca7efba59f73788d22b113c2638b SendRoot:0x55cf415773b9e2c845f63a906c665e03a1ca781bd3f7fe41e3838738c5fba678 Batch:586040 PosInBatch:448}"
INFO [04-11|01:20:33.520] catching up to chain batches localBatches=579,554 target=586,041
WARN [04-11|01:21:24.518] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"
INFO [04-11|01:21:25.332] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13092 state="{BlockHash:0x1aa5ab7a54c9cb55eac61213665266ea38baca7efba59f73788d22b113c2638b SendRoot:0x55cf415773b9e2c845f63a906c665e03a1ca781bd3f7fe41e3838738c5fba678 Batch:586040 PosInBatch:448}"
INFO [04-11|01:21:33.550] catching up to chain batches localBatches=579,554 target=586,041
INFO [04-11|01:22:25.347] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13092 state="{BlockHash:0x1aa5ab7a54c9cb55eac61213665266ea38baca7efba59f73788d22b113c2638b SendRoot:0x55cf415773b9e2c845f63a906c665e03a1ca781bd3f7fe41e3838738c5fba678 Batch:586040 PosInBatch:448}"

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

@ZYS980327 Are you running your prysm node with --enable-experimental-backfill ? If you are not, you should and the problem should resolve

@ZYS980327
Copy link

--enable-experimental-backfill

I used this flag --enable-experimental-backfill=true, and my prysm printed "Backfill batches processed" batchedRemaining=270233 importable=2 imported=2 prefix=backfill, but docker started that Nitro image still reports error INFO [04-11|06:43:31.214] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash: 0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0} "
and WARN [04-11|06:40:42.888] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0" and the batch information is always catching up to chain batches localBatches=579,554 target=586,123

@ZYS980327
Copy link

您是否正在使用 ?如果你不是,你应该并且问题应该解决--enable-experimental-backfill

docker run -d --privileged --rm -it -v /usr/local/nitro-snap/:/usr/local/nitro-snap/ -p 0.0.0.0:8567:8567 -p 0.0.0.0:8568:8568 offchainlabs/nitro-node:v2.3.2-064fa11 --parent-chain.connection.url http://10.150.20.11:8545 --chain.id=42161 --parent-chain.blob-client.beacon-url=http://10.150.20.11:3500 --http.api=net,web3,eth,arb,debug --http.corsdomain=* --http.addr=0.0.0.0 --http.vhosts=* --init.url="file:///usr/local/nitro-snap/nitro-pruned.tar" --execution.tx-lookup-limit=0

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

@ZYS980327 Is your prysm node syncing ? what are its current logs

@ZYS980327
Copy link

INFO [04-11|06:38:29.783] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
WARN [04-11|06:38:36.627] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"
INFO [04-11|06:38:44.827] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:39:29.797] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:39:44.863] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:40:29.810] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:40:44.912] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:41:29.836] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:41:44.960] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:42:29.851] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
WARN [04-11|06:42:35.388] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"
INFO [04-11|06:42:44.995] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:43:29.866] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:43:45.049] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:44:29.877] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:44:45.087] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:45:29.889] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
INFO [04-11|06:45:45.135] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:46:29.901] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}"
WARN [04-11|06:46:39.181] error reading inbox err="failed to get blobs: expected at least 6 blobs for slot 8633650 but only got 0"
INFO [04-11|06:46:45.197] catching up to chain batches localBatches=579,554 target=586,123
INFO [04-11|06:47:29.914] latest assertion not yet in our node staker=0x0000000000000000000000000000000000000000 assertion=13098 state="{BlockHash:0xd76b38002d04388b9021a3299ecec567ffb8862f8e92daf6ec8beb076869aeba SendRoot:0x38eba7af4bd8e46ca9c9498924b14a2b2e5e51817f817ba2f070ef35d4ed7cf6 Batch:586123 PosInBatch:0}

您的 prysm 节点是否同步?它的当前日志是什么

@ZYS980327
Copy link

@nisdas 我认为硝基应该没问题

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Those are nitro logs, can you show prysm logs ?

@ZYS980327
Copy link

@nisdas time="2024-04-11 14:51:31" level=info msg="Backfill batches processed" batchesRemaining=270105 importable=1 imported=1 prefix=backfill
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p
time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832815 currentSlot=8832855 prefix=p2p

time="2024-04-11 14:51:31" level=warning msg="Attestation is too old to broadcast, disscarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:01" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:01" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:01" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:01" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:02" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:02" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:04" level=info msg="Synced new block" block=0x57701e6b... epoch=276026 finalizedEpoch=276024 finalizedRoot=0xbf2f63b5... prefix=blockchain slot=8832858
time="2024-04-11 14:52:04" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x7d048563e9b0 prefix=blockchain slot=8832858 syncBitsCount=511 txCount=130
time="2024-04-11 14:52:04" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832858 prefix=p2p
time="2024-04-11 14:52:24" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832860 prefix=p2p
time="2024-04-11 14:52:24" level=info msg="Backfill batches processed" batchesRemaining=270092 importable=1 imported=1 prefix=backfill
time="2024-04-11 14:52:24" level=warning msg="Attestation is too old to broadcast, discarding it" attestationSlot=8832818 currentSlot=8832860 prefix=p2p

@ZYS980327
Copy link

@nisdas Most of it is broadcasting something that was rejected, and there are also some backfill logs and sync logs

@ZYS980327
Copy link

@nisdas ./prysm.sh beacon-chain --execution-endpoint=http://localhost:8551 --jwt-secret=/usr/local/prysm/jwt.hex --datadir /data/prysm-data/ --accept-terms-of-use=true --checkpoint-sync-url=https://beaconstate.info --genesis-beacon-api-url=https://beaconstate.info --grpc-gateway-host 10.150.20.11 --grpc-gateway-port 3500 --enable-experimental-backfill=true --accept-terms-of-use --backfill-oldest-slot=0

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

@ZYS980327 which version is this ? for prysm

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Can you try running v5.0.3 ? it has a few improvements

@ZYS980327
Copy link

@nisdas I think it should be the latest version 5.0.3, I've restarted it several times

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Can you paste more logs for prysm if possible ? A text file is fine, so that we can see why it is still stuck

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Ok nevermind, I understand it now. The block you are requesting has blobs which have already been pruned(18 day period)
https://beaconcha.in/slot/8633650

This is because of this:
prysmaticlabs/prysm#13853

@ZYS980327
Copy link

@ZYS980327
Copy link

docker-nitro-2.log

@ZYS980327
Copy link

@nisdas I started two nitros, both of which were createblock at the beginning, and then there were no more

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Hey @ZYS980327 when did you start your nitro nodes ? Was this of the time at the fork. A nitro node running for a while should not be requesting expired blobs afaik

@ZYS980327
Copy link

@nisdas One was launched yesterday and just now

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Was this running with an old snapshot ? --init.url="file:///usr/local/nitro-snap/nitro-pruned.tar" Is it possible to run with a newer one

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Nitro is requesting already expired blobs , so it is difficult to get this from the beacon p2p network. Possibly why prysm can't provide them

@ZYS980327
Copy link

@nisdas Okay, thank you, it may be caused by the Ethereum hard fork, the nitro snapshot is before the fork, and the nitro reception didn't get the data before the slot before requesting the prysm node, so it's stuck here, is that what it means?

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

@ZYS980327 Yeah that will be an issue, the snapshot needs to be within the last 18 days otherwise nitro will be requesting already expired blob data from prysm.

@ZYS980327
Copy link

@nisdas This snapshot should be before the Cancun upgrade, I don't know if the official update snapshot

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

@ZYS980327 Can you try fetching the snapshot again to see if it helps ?

@ZYS980327
Copy link

@nisdas Okay, but it will take some time to download and run the snapshot, and please come to this page in two days, and I will send some information about the new snapshot, thanks

@nisdas
Copy link
Contributor

nisdas commented Apr 11, 2024

Sure thing, let us know if it works fine after @ZYS980327

@hdiass
Copy link

hdiass commented Apr 12, 2024

@nisdas I downloaded yesterday the snapshot and i'm having this issue

@hdiass
Copy link

hdiass commented Apr 12, 2024

Can you reopen this @nisdas ?

@ZYS980327
Copy link

@nisdas Even though I downloaded a new snapshot, it still doesn't work, and node sync doesn't get blobs

@kindziora
Copy link

having the same issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants