Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: tracer to track delivered message if duplicate #385

Merged
merged 2 commits into from
Dec 22, 2022

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Dec 14, 2022

Motivation

  • To investigate broken promises in v1.2.1: Low mesh peers due to p7 lodestar#4818
  • Consider this scenario
    • 2 messages have same fastMsgId but different msgIds (collision)
    • node sends IWANT control message and add a promise
    • message 1 comes and being tracked in fastMsgId
    • message 2 (as a response to IWANT) comes, since it has same fastMsgId it's a duplicate we didn't mask as "message delivered" in tracer, and it looks like the other node did not send us the requested messages

Description

  • call tracer. deliverMessage() in the case of message duplicate

part of ChainSafe/lodestar#4818

@twoeths twoeths requested a review from a team as a code owner December 14, 2022 06:10
@twoeths twoeths changed the title feat: tracer to track delivered message if duplicate fix: tracer to track delivered message if duplicate Dec 14, 2022
@codecov-commenter
Copy link

codecov-commenter commented Dec 14, 2022

Codecov Report

Base: 83.26% // Head: 83.22% // Decreases project coverage by -0.03% ⚠️

Coverage data is based on head (55cdaad) compared to base (dbeb879).
Patch coverage: 45.45% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #385      +/-   ##
==========================================
- Coverage   83.26%   83.22%   -0.04%     
==========================================
  Files          48       48              
  Lines       11787    11797      +10     
  Branches     1266     1266              
==========================================
+ Hits         9814     9818       +4     
- Misses       1973     1979       +6     
Impacted Files Coverage Δ
src/metrics.ts 21.75% <0.00%> (-0.16%) ⬇️
src/tracer.ts 77.01% <50.00%> (-0.45%) ⬇️
src/index.ts 69.82% <100.00%> (+0.04%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add metrics to verify this hypothesis?

  • Add a second parameter to deliverMessage(msg, isDuplicate: boolean)
  • In that function, if promise resolved and isDuplicate then this.metrics.iwantPromiseResolvedFromDuplicate.inc(1)

@twoeths twoeths marked this pull request as draft December 14, 2022 08:33
@twoeths twoeths force-pushed the tuyen/tracer_more_deliveredMessage branch from a249d0e to 55cdaad Compare December 15, 2022 02:56
@twoeths
Copy link
Contributor Author

twoeths commented Dec 15, 2022

thanks @dapplion for the metric, it shows a lot of promises could be resolved by duplicate messages (this is from a mainnet node subscribing to all subnets)

Screen Shot 2022-12-15 at 10 06 02

it could reduce 80% 90% of broken promises (this was only tested 3h but we could trust as these 2 metrics match each other)

Screen Shot 2022-12-15 at 10 07 23

@twoeths twoeths marked this pull request as ready for review December 15, 2022 03:09
@wemeetagain wemeetagain merged commit 0c8ddee into master Dec 22, 2022
@wemeetagain wemeetagain deleted the tuyen/tracer_more_deliveredMessage branch December 22, 2022 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants