Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks #796

taariq · 2023-04-12T18:25:06Z

What is happening?

Section description

Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Upgraded genesis with requirements for Cosmos-SDK v0.47.1 is read by palomad, but validators in genesis are not recognized and blocks are not created.

curl -S localhost:26657/status | json_pp returns the following:

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   811  100   811    0     0    98k      0 --:--:-- --:--:-- --:--:--   98k
{
   "id" : -1,
   "jsonrpc" : "2.0",
   "result" : {
      "node_info" : {
         "channels" : "40202122233038606100",
         "id" : "b41cb52b7d024ee7fd237350bf3ebdd63c5c5762",
         "listen_addr" : "45.33.42.146:26656",
         "moniker" : "birdlady",
         "network" : "turtle-dove-net",
         "other" : {
            "rpc_address" : "tcp://0.0.0.0:26657",
            "tx_index" : "on"
         },
         "protocol_version" : {
            "app" : "0",
            "block" : "11",
            "p2p" : "8"
         },
         "version" : "0.37.0"
      },
      "sync_info" : {
         "catching_up" : false,
         "earliest_app_hash" : "",
         "earliest_block_hash" : "",
         "earliest_block_height" : "0",
         "earliest_block_time" : "1970-01-01T00:00:00Z",
         "latest_app_hash" : "",
         "latest_block_hash" : "",
         "latest_block_height" : "0",
         "latest_block_time" : "1970-01-01T00:00:00Z"
      },
      "validator_info" : {
         "address" : "CB031878B4EEB66483B17C723DAA157F477F7159",
         "pub_key" : {
            "type" : "tendermint/PubKeyEd25519",
            "value" : "LaB65QGwVPHNULUQd2CcrHzr1tWzfm7fV/D4+hvKJM8="
         },
         "voting_power" : "0"
      }
   }
}

Although the validators are in genesis, we receive no blocks. We believe this issue is genesis related.

Gist of Log output and the addrbook.json
https://gist.github.com/taariq/6b311fffec32790f5ba233ead4e30ef4

Paloma and pigeon versions and logs

Section description

Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

Palomad Master and Pigeon Master.

How to reproduce?

Section description

Please write detailed steps of what you were doing for this bug to appear.

We use the private testnet genesis that was upgraded with Cosmos-SDK v0.47.1 changes to setup here: https://github.com/VolumeFi/paloma-private-testnet/tree/genesis-updates

What is the expected behaviour?

Section description

If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

Validator keys and validator balances are recognized and blocks are started on the private testnet.

The text was updated successfully, but these errors were encountered:

taariq · 2023-04-13T13:41:57Z

Closed with #799

verabehr · 2023-04-13T14:07:19Z

https://gist.github.com/verabehr/deb65d09722412ae926c31790f9a562c logs from latest commit are here

verabehr · 2023-04-13T15:44:45Z

Note that the pubkey in the log line 35 is different from what shows with the show validator command and what is listed in the priv_validator_key.json while the address matches.

Log output

"{"_msg":"This node is not a validator","addr":"CB031878B4EEB66483B17C723DAA157F477F7159","level":"info","module":"consensus","pubKey":"PubKeyEd25519{2DA07AE501B054F1CD50B51077609CAC7CEBD6D5B37E6EDF57F0F8FA1BCA24CF}

CLI output

palomad tendermint show-validator
{"@type":"/cosmos.crypto.ed25519.PubKey","key":"LaB65QGwVPHNULUQd2CcrHzr1tWzfm7fV/D4+hvKJM8="}

cat priv_validator_key.json
  {
  "address": "CB031878B4EEB66483B17C723DAA157F477F7159",
  "pub_key": {
    "type": "tendermint/PubKeyEd25519",
    "value": "LaB65QGwVPHNULUQd2CcrHzr1tWzfm7fV/D4+hvKJM8="
  },```

taariq · 2023-04-13T17:01:26Z

error

testnet-1 palomad[1865062]: {"_msg":"This node is not a validator","addr":"CB031878B4EEB66483B17C723DAA157F477F7159","level":"info","module":"consensus","pubKey":"PubKeyEd25519{2DA07AE501>
Apr 13 13:54:34 testnet-1 palomad[1865062]: {"ID":"b41cb52b7d024ee7fd237350bf3ebdd63c5c5762","_msg":"P2P Node ID","file":"/root/.paloma/config/node_key.json","level":"info","module":"p2p","ts":"2023-04-1>
Apr 13 13:54:34 testnet-1 palomad[1865062]: {"_msg":"Adding persistent peers","addrs":["12010c5d65fbae451e7a91194dd394044da8e510@45.33.42.196:26656","4ef74a901ffafc51fa5c8c138f672319a7081974@45.33.42.186>
Apr 13 13:54:34

taariq · 2023-04-14T04:09:42Z

@0xmuralik, @aleem1314 did you have a chance to test genesis against master? We were able to get rid of "not a validator" errors at consensus start. The keys are validators, but still not creating blocks. Will you let us know if you're able to identify the issue?

aleem1314 · 2023-04-14T07:36:38Z

@0xmuralik, @aleem1314 did you have a chance to test genesis against master? We were able to get rid of "not a validator" errors at consensus start. The keys are validators, but still not creating blocks. Will you let us know if you're able to identify the issue?

Yeah, tested locally with this script. Upgrade failed with following error

failed to load latest version: version of store feeibc mismatch root store's version; expected 69 got 0; new stores should be added using StoreUpgrades

And to fix above issue, tried upgrade with #797 . This time upgrade went through, after the upgrade all queries returning following error message.

codespace sdk code 18: invalid request: failed to load state at height 75; version does not exist (latest height: 75)

taariq · 2023-04-14T10:47:16Z

Merging master into #797. Not sure it will help.

taariq · 2023-04-14T10:47:30Z

@alexanderbez Any thoughts?

taariq · 2023-04-14T10:56:36Z

@aleem1314 okay, looks like this is related cosmos/cosmos-sdk#14620

alexanderbez · 2023-04-14T15:11:36Z

@0xmuralik, @aleem1314 did you have a chance to test genesis against master? We were able to get rid of "not a validator" errors at consensus start. The keys are validators, but still not creating blocks. Will you let us know if you're able to identify the issue?

Yeah, tested locally with this script. Upgrade failed with following error
failed to load latest version: version of store feeibc mismatch root store's version; expected 69 got 0; new stores should be added using StoreUpgrades
And to fix above issue, tried upgrade with #797 . This time upgrade went through, after the upgrade all queries returning following error message.
codespace sdk code 18: invalid request: failed to load state at height 75; version does not exist (latest height: 75)

All new modules must be stated in the upgrade. E.g.: https://github.com/cosmos/cosmos-sdk/blob/c4489d9785de8b24efe12ba9209669e9dce9e23f/docs/docs/core/15-upgrade.md?plain=1#L88-L98

taariq · 2023-04-14T18:38:38Z

We have started to add new modules here:
19ad490#diff-aeec7ddb4cf8b7e9e6e57625770d369cce5f377cc350b42c3aca0427bba8dfbcR153

But...we are missing others?

taariq · 2023-04-14T21:48:43Z

Fixed with #797. We are producing blocks on upgrade.

taariq · 2023-04-17T04:36:55Z

@aleem1314 will you do a final test to re-confirm that you are able to execute GRPC off master branch?

aleem1314 · 2023-04-17T06:47:18Z

@aleem1314 will you do a final test to re-confirm that you are able to execute GRPC off master branch?

I am still getting the same error after the upgrade "message": "codespace sdk code 18: invalid request: failed to load state at height 153; version does not exist (latest height: 153)",.

taariq · 2023-04-17T13:04:01Z

@aleem1314 will you do a final test to re-confirm that you are able to execute GRPC off master branch?

I am still getting the same error after the upgrade "message": "codespace sdk code 18: invalid request: failed to load state at height 153; version does not exist (latest height: 153)",.

What is the gprc command you are running?

taariq · 2023-04-17T13:36:49Z

Closing and moving the codespace error to its own ticket.

taariq · 2023-04-17T13:43:42Z

The codespace SDK issue moved to #800

aleem1314 · 2023-04-17T14:18:31Z

What is the gprc command you are running?

palomad q staking validators

or

curl http://127.0.0.1/cosmos/staking/v1beta1/validators

taariq added the bug Something isn't working label Apr 12, 2023

taariq added this to the V 1.0 milestone Apr 12, 2023

taariq assigned verabehr, 0xbala-k and aleem1314 Apr 12, 2023

taariq added this to Paloma Apr 12, 2023

taariq closed this as completed Apr 13, 2023

github-project-automation bot moved this to Done in Paloma Apr 13, 2023

taariq reopened this Apr 13, 2023

taariq changed the title ~~Bug: Palomad starts with genesis, but blocks are not being created. No errors in logs, but no blocks~~ Bug: Palomad starts with v 0.45.13 genesis, but blocks are not being created. No errors in logs, but no blocks Apr 13, 2023

taariq changed the title ~~Bug: Palomad starts with v 0.45.13 genesis, but blocks are not being created. No errors in logs, but no blocks~~ Bug: Palomad starts with Cosmos-SDK v0.45.13 genesis, but blocks are not being created. No errors in logs, but no blocks Apr 13, 2023

taariq changed the title ~~Bug: Palomad starts with Cosmos-SDK v0.45.13 genesis, but blocks are not being created. No errors in logs, but no blocks~~ Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks Apr 13, 2023

taariq closed this as completed Apr 14, 2023

taariq reopened this Apr 17, 2023

taariq closed this as completed Apr 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks #796

Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks #796

taariq commented Apr 12, 2023 •

edited

Loading

taariq commented Apr 13, 2023

verabehr commented Apr 13, 2023

verabehr commented Apr 13, 2023 •

edited

Loading

taariq commented Apr 13, 2023

taariq commented Apr 14, 2023

aleem1314 commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

alexanderbez commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 17, 2023

aleem1314 commented Apr 17, 2023

taariq commented Apr 17, 2023

taariq commented Apr 17, 2023

taariq commented Apr 17, 2023

aleem1314 commented Apr 17, 2023

Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks #796

Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks #796

Comments

taariq commented Apr 12, 2023 • edited Loading

What is happening?

Paloma and pigeon versions and logs

How to reproduce?

What is the expected behaviour?

taariq commented Apr 13, 2023

verabehr commented Apr 13, 2023

verabehr commented Apr 13, 2023 • edited Loading

taariq commented Apr 13, 2023

taariq commented Apr 14, 2023

aleem1314 commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

alexanderbez commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 14, 2023

taariq commented Apr 17, 2023

aleem1314 commented Apr 17, 2023

taariq commented Apr 17, 2023

taariq commented Apr 17, 2023

taariq commented Apr 17, 2023

aleem1314 commented Apr 17, 2023

taariq commented Apr 12, 2023 •

edited

Loading

verabehr commented Apr 13, 2023 •

edited

Loading