Txn is too big to fit into one request: unable to bootstrap balances

[samuel-rufi]

Describe the bug

When trying to bootstrap a large bootstrap_balances.json file that contains more than 35k accounts, rosetta-cli breaks with following error: "Txn is too big to fit into one request: unable to bootstrap balances".

To Reproduce
Steps to reproduce the behavior:

1. create a large bootstrap_balances.json file that contains for example more than 35k accounts
2. link the bootstrap_balances.json file with the rosetta-cli configuration file
3. run the rosetta-cli data check

Expected behavior
I expect rosetta-cli to be able to handle large bootstrap_balances.json files that contain many accounts.

[shrimalmadhur]

Thanks @rufsam for bringing this up. I will take a look at it and get back to you.

[maili-cb]

@shrimalmadhur I am thinking this limitation may come from here , the default table size is set to 256MB, which means the default max txn size for badgerDB is ~38MB. Whereas the PerformanceMaxTableSize is 3072MB which is more performance geared.

[shrimalmadhur]

@maili-cb I see. I am wondering if we can break these transactions and then bootstrap balances. Maybe explore more on how we could handle this without changing these settings.

[samuel-rufi]

Hi @maili-cb and @shrimalmadhur! do you have any new findings on this issue? Thanks alot for your efforts.

[shrimalmadhur]

Hi @rufsam

Sorry for the delayed response. I did a little more digging on this and here's what I found. This issue comes from Badger DB configuration settings as mentioned by @maili-cb . So the error comes from the BadgerDB config. Using MaxTableSize config we are limiting the size of memtable.

A way to solve this is using more memory performant settings provided by rosetta-cli config. You can do this by enabling MemoryLimitDisabled config. This will enable the Performance Badger option which would increase the size of memtable significantly and should be able to handle larger transaction size. Please be mindful that this will need a lot more RAM so make sure to have enough free RAM while doing this. Let me know if that doesn't work and we can try to figure out other options.

[samuel-rufi]

Hello @shrimalmadhur, thanks for your help and detailed information. It seems like that the "memory_limit_disabled" = true did solve the issue.

Assuming that there will be many more accounts to be bootstrapped in the future, the high RAM requirement could be an obstacle for integrators. If possible, it would definitely be good to explore other options to solve the problem.

[shrimalmadhur]

@rufsam great to hear it worked. And yes it makes sense to solve this issue in generic way rather than increasing system's RAM. I will create a separate issue for exploring other options.