Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Electra: No aggregated attestation for slot #7548

Open
twoeths opened this issue Mar 11, 2025 · 3 comments
Open

Electra: No aggregated attestation for slot #7548

twoeths opened this issue Mar 11, 2025 · 3 comments
Labels
meta-bug Issues that identify a bug and require a fix. spec-electra 🦒 Issues targeting the Electra spec version

Comments

@twoeths
Copy link
Contributor

twoeths commented Mar 11, 2025

Describe the bug

this was found by @nflaig
Image

this could be due to #7546

Expected behavior

  • at least, we should have 1 attestation from ourself. Or explain the reason
  • but first, need to separate this metric for api vs gossip lodestar_attestation_pool_insert_outcome_total

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.27.1

@twoeths twoeths added the meta-bug Issues that identify a bug and require a fix. label Mar 11, 2025
@twoeths
Copy link
Contributor Author

twoeths commented Mar 11, 2025

Image

some api attestations may come late and it's after the cutoff time of the attestation pool. In that case the attestation pool should accept it because the getAggregate() call only comes after that and api attestation has higher priority

nflaig pushed a commit that referenced this issue Mar 12, 2025
**Motivation**

- api attestations could be late but we should allow them to be added to
OpPool in that case see
#7548 (comment)

**Description**

- add flag to oppool to specify a priority (api) attestation. This is
the same to the way we do attestation validation
https://github.com/ChainSafe/lodestar/blob/ca29dbb8edad29e1eb5ac2c576059abcb1c82ec7/packages/beacon-node/src/chain/validation/attestation.ts#L209
- same feature for SyncCommittee

part of #7548

---------

Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
@nflaig
Copy link
Member

nflaig commented Mar 12, 2025

this still happens on our holesky-prod fleet, see logs from last 6 hours, we can use this query to check once we have deployed a fix

also we should probably add a metric to better keep track of get aggregate cache misses

if (!aggregate) {
// TODO: Add metric for missing aggregates
return null;

@nflaig
Copy link
Member

nflaig commented Mar 12, 2025

some api attestations may come late and it's after the cutoff time of the attestation pool. In that case the attestation pool should accept it because the getAggregate() call only comes after that and api attestation has higher priority

while I agree that this is a good change I am not really convinced that it is the root cause that we have aggregate cache misses

just looking at the logs of holesky-prod-0 from last 30 days it seems the issue has started after the electra fork

Image

@philknows philknows added the spec-electra 🦒 Issues targeting the Electra spec version label Mar 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta-bug Issues that identify a bug and require a fix. spec-electra 🦒 Issues targeting the Electra spec version
Projects
None yet
Development

No branches or pull requests

3 participants