Pusher PingPong is making clients go offline too often

[tgolen]

Problem

Once the pusher pingpong was implemented in #55326, it's resulted in more clients being forced offline than I had expected. This prevents users from using the app.

Solution

Turn off the portion of the ping-pong that forces clients to go offline when the PONG event is not received and analyze the logs to see what behavior is happening with the ping-pong so it can be dialed in better.

[melvin-bot]

Triggered auto assignment to @yuwenmemon (DeployBlockerCash), see https://stackoverflowteams.com/c/expensify/questions/9980/ for more details.

[melvin-bot]

💬 A slack conversation has been started in #expensify-open-source

[github-actions]

👋 Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open `StagingDeployCash` deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

1. Identify the pull request that introduced this issue and revert it.
2. Find someone who can quickly fix the issue.
3. Fix the issue yourself.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.93-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] [CP Staging] Disable the network going offline for now when a PONG is not received #56183

If no regressions arise, payment will be issued on 2025-02-10. 🎊

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.94-25 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] [CP Staging] Disable the network going offline for now when a PONG is not received #56183

If no regressions arise, payment will be issued on 2025-02-13. 🎊

[tgolen]

Daily Update

• This is deployed, and I'm still seeing a much higher rate of PONGs not being received

Next Steps

• I need to explore the grafana data and logs to see if I can spot any specific issues
• I maybe need to add some more logging

ETA

• Friday, Feb. 14

[tgolen]

Daily Update

• I didn't get a chance yesterday to look into this yet

Next Steps

• See if I can get to it today after chores are done.

[tgolen]

Daily Update

• I discovered from the logs, that once a PONG is missed, it continues to log the same PONG being missed over and over
• This is most likely due to the client being offline (which is normal)
• This makes log analysis difficult because there is no way to see when the client FIRST missed a PONG
• I have created a new App PR which will make the logs more useful for tracking the first missed PONG

Next Steps

• @blimpich review and merge the PR
• @tgolen Wait for deploy, then continue checking the logs and analyzing

ETA

• Friday, Feb. 14

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[mallenexpensify]

(long shot) any chance this bug in #quality is related? Asking cuz I searched script-src from the error then ended up here.

[tgolen]

It shouldn't be, no. I still haven't re-enabled the code where Pusher PINGPONG actually puts clients offline. I'm still just logging it and trying to see what the behavior of the PINGPONG is.

[tgolen]

Weekly Update

• After the recent changes, the logs look MUCH better and now the rate of missing PONGs is nearly identical to the rate of general Pusher connection errors

Next Step

• @tgolen create a PR for E/App that will make the client go offline when a PONG is missed, and come back online when a PONG is received

ETA

• PR created today, Feb. 21

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.1.7-2 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] Change offline status when Pusher PINGPONG fails #57255

If no regressions arise, payment will be issued on 2025-03-10. 🎊

[tgolen]

This issue can be closed for now. I have reverted the code which makes the client go offline too often. We're only logging it for now and I need to contact Pusher to discuss about the low delivery rates we're having with our PINGPONG. I'll open another issue for that.

[tgolen]

Opened #58024 to continue the investigation.