Error: rosetta: (502) bad gateway #10088
Comments
|
I'm not currently understanding how rosetta can be in any way involved in this. The bad gateway error is returned by rosetta if it is trying to communicate with a node which is unreachable. The following are the two lines of code which return the
This is an API call, which some component is making (I'm not sure which one?). And I'm not understanding why this should in any way hinder an upgrade. Unless there's some conflicting readiness check on rosetta (which to work needs the tendermint RPC node it is pointing to to be functional). Maybe having the full error would be helpful. |
|
Hi @fdymylja, Here is the full error after upgrade, the validator produced a few blocks then: I also find out that before the upgrade height, if I query the node will be blocked (no error of rosseta 502 though) here if I query the proposal before the upgrade height: pic: left is the node that is blocked, right is the validator owns 67% power, and keeps producing blocks. |
|
@yaruwangway my feeling is that rosetta is unrelated to this problem. The bad gateway error is thrown by rosetta in two cases:
I would suggest for now, in order to understand if the issue is really related to rosetta, to disable it in the [rosetta]
enable = false |
|
@fdymylja Thank you!!! |
I think it's related to some misconfiguration, can you show your rosetta section of app.toml? |
|
@fdymylja By "blocking", it is not a real blocking, because if I block validator, the other node can still produce blocks. that means the validator is still signing blocks. (I am not so sure, because if I stop the non-blocked node and start it again, it cannot find the peer - the one is blocked). It is probably a cosmovisor issue, if I use gaiad rather than cosmovisor, I can I can do a follow up if the new version of cosmovisor works. Thank you so much ! |
No problem, let me know how it goes! |
|
Hi, Frojdi, Now I do not have any query issues with the new version of cosmovisor. But still, if I enable rosetta API, no matter I use 7:31PM INF starting API server... module=api-server
7:31PM INF Starting RPC HTTP server on [::]:1327 module=api-server
7:31PM INF Dialing peer address={"id":"592fb09197f46b28fdddd0bc38326e14efac43af","ip":"::1","port":26656} module=p2p
7:31PM INF Error reconnecting to peer. Trying again addr={"id":"592fb09197f46b28fdddd0bc38326e14efac43af","ip":"::1","port":26656} err="dial tcp [::1]:26656: connect: connection refused" module=p2p tries=1
7:31PM INF Inbound Peer rejected err="auth failure: secret conn failed: EOF" module=p2p numPeers=0
7:31PM INF Dialing peer address={"id":"592fb09197f46b28fdddd0bc38326e14efac43af","ip":"::1","port":26656} module=p2p
7:31PM INF Error reconnecting to peer. Trying again addr={"id":"592fb09197f46b28fdddd0bc38326e14efac43af","ip":"::1","port":26656} err="dial tcp [::1]:26656: connect: connection refused" module=p2p tries=2
Error: rosetta: (502) bad gatewaythe other node: ERR dialing failed (attempts: 1): auth failure: secret conn failed: EOF addr={"id":"592fb09197f46b28fdddd0bc38326e14efac43af","ip":"::1","port":26656} module=pexThe two nodes' config of Rosetta: # Enable defines if the Rosetta API server should be enabled.
enable = true
# Address defines the Rosetta API server to listen on.
address = ":8081"
# Network defines the name of the blockchain that will be returned by Rosetta.
blockchain = "app"
# Network defines the name of the network that will be returned by Rosetta.
network = "network"
# Retries defines the number of retries when connecting to the node before failing.
retries = 3
# Offline defines if Rosetta server should run in offline mode.
offline = falseThe other node: # Enable defines if the Rosetta API server should be enabled.
enable = false
# Address defines the Rosetta API server to listen on.
address = ":8080"
# Network defines the name of the blockchain that will be returned by Rosetta.
blockchain = "app"
# Network defines the name of the network that will be returned by Rosetta.
network = "network"
# Retries defines the number of retries when connecting to the node before failing.
retries = 3
# Offline defines if Rosetta server should run in offline mode.
offline = falseIf I disable rosetta for both nodes, Do you have any clues of debug ? Thank you! @fdymylja |
|
Could you increase the number of retries to 1000? just to understand if maybe rosetta is failing to connect because the node is not ready yet |
|
If I change retries to 1000, the validator node will produce a couple of blocks then fail: 2:39PM INF indexed block height=7368552 module=txindex
Error: rosetta: (502) bad gateway |
a bit more info: I printed the error: INF indexed block height=7368570 module=txindex
RPC error -32603 - Internal error: height 1 is not available, lowest height is 7368387
Error: rosetta: (502) bad gatewaythis error is after upgrade. and 7368387 is the height to which I exported the genesis file. |
|
Yes we were discussing it on the architecture call. It's not a bug but rather a lack of a feature. Which is defining a custom genesis height. I will implement this feature asap and we can backport it. |
…10150) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description closes: #10121 closes: #10088 rosetta server fails to run with genesis.json has initial_heights 2 or larger rosetta-cli check:data fails to parse EventTypeCoinBurn --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
…10150) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description closes: #10121 closes: #10088 rosetta server fails to run with genesis.json has initial_heights 2 or larger rosetta-cli check:data fails to parse EventTypeCoinBurn --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 5a72b5e)
…10150) (#10165) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description closes: #10121 closes: #10088 rosetta server fails to run with genesis.json has initial_heights 2 or larger rosetta-cli check:data fails to parse EventTypeCoinBurn --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 5a72b5e) Co-authored-by: Geoff Lee <thirdchance@gmail.com>
Summary of Bug
I run a local test of gaia upgrade (upgrade dependency of cosmos-sdk from 0.42 to 0.43) and have the bug of resetta bad gateway after upgrade.
I export the genesis file from live cosmoshub-4 and change the genesis file: I change two validators in the exported genesis file to be my nodes, and change one validator's power over 67% so I can start a local chain. then I use cosmovisor starts the two nodes and let them listen to each other. I submit upgrade proposal and the node upgrades successfully upon the proposed height, however, the validator stops and throw the
Error: rosetta: (502) bad gatewayafter upgrade and produced a few blocks. I can restart the validator again ,but it will just produce a couple more blocks and throw the same error.INF ABCIQuery data= module=rpc path=/cosmos.bank.v1beta1.Query/TotalSupply result={"code":0,"codespace":"","height":"7368611","index":"0","info":"","key":null,"log":"","proofOps":null,"value":"CkwKRGliYy8wNzkxMkMyNDAwNDkzMkNENTYxQjE3NTE1NjJCMjJFQTc4N0YzMUY5ODIxNTY4Qjg4RjU1QThGNTFEMzI2NzIyEgQ1MDAwClIKRGliYy8wODBENzYwNEQ5QzE5NjBBMzU2MkUxQUU0RThFQThGMzBCNzgxMzgxMEFFMzA2NjBENDFERDY3NzBCMUM1OTI5EgoxMDAwMDAwMDAwCkkKRGliYy8wODgzNEE3NkY0RTVBRUQwODY5MDkxNkY2MUVBMTJBQTcxQ0ZENjM2QkJBMzI4MDYyMDI3REY5RkE2MjBCN0UzEgExCkkKRGliYy8wRU ... ... Error: rosetta: (502) bad gatewayVersion
upgrade dependency of cosmos-sdk 0.42 to 0.43
Steps to Reproduce
details of procedures:
gaia vega upgrade
For Admin Use
The text was updated successfully, but these errors were encountered: