bug: v018 delegation rewards error still persists

[hard-nett]

two validators delegations still incur the bug present for v018:
bitsongvaloper1qxw4fjged2xve8ez7nu779tm8ejw92rv0vcuqr
bitsongvaloper1xnc32z84cc9vwftvv4w0v02a2slug3tjt6qyct

afaik, it is relevant due to the introduction of the bug described here: #260

[hard-nett]

Resolution Steps

1. determine which delegations are broken from these validators, and why.
2. add logic to a new upgrade handler that will resolve bug
3. update app-suite test logic to confirm that upgrade handler will resolve bug.

Step 1: Determine Root of Issue

- there are 2 delegations with 0 shares: This is confirmed by iterating through all delegations for a given validator, and asserting that the delegation is greater than the minimum decimal value. If not, it is 0, and it can safely be removed from the store.

- there are two validators missing a slash event: 2 slashing events tracked by the distribution store (distribution uses staking hooks for this functionality, which were broken during v018 upgrade) the distribution invariant logic ReferenceCountInvariant shows that the expected and the resulted count is two off.

Using Custom Export

Using a custom upgradehandler logic that can be imported throughout the app will allow us to simulate & verify the functions of the logic prior to the upgrade.

The logic to resolve this bug is required to:

1. iterate all delegations
2. check if delegation need patching to distribution reward calculation
3. manually claim rewards if patch is applied to distribution reward calculation for delegation
4. Insert mock slashing events for the two impacted validators
5. re-iterate and assert no patches are needed anymore

Reproduce In Testnet Environment

Since the store is modified from the mainnet state, when using the in-place-testnet logic, we need to ensure that we modify delegations such that delegations to our new testnet validator incur the same issue being experienced on mainnet. To do this, we will have to:

• manually set known impacted validator state to new validator:
  • total voting power:
  • total coins:
  • all delegations:
• remove all other validators & delegations

Step 2: Upgrade Handler Tests

• use app suite tests
• use custom export to mimic logic

[hard-nett]

two validators delegations still incur the bug present for v018: bitsongvaloper1qxw4fjged2xve8ez7nu779tm8ejw92rv0vcuqr bitsongvaloper1xnc32z84cc9vwftvv4w0v02a2slug3tjt6qyct

after investigating, it appears these two validators are the only validators containing a historical reward reference count of 2.

further investigation shows that by manually setting a slashing event for these two operators allows us to pass crisis invariant checks. this lines up with the hypothesis that distribution calculations are inaccurate as there is a missing slashing event in the store for these two validators.