fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client

[adhorodyski]

Explanation of Change

When the client detects a gap in server update IDs it fetches the missing range with GetMissingOnyxMessages. Production traces show the server sometimes answers that fetch with a 200 that doesn't move the client forward. The client never advances, so every incoming Pusher update re-detects the same gap and fires the same fetch again, about once per second. The earlier fix in #93948 only coalesced fetches that overlap in time, so it cannot break this loop of back-to-back fetches.

This PR adds a stall guard:

1. A successful fetch must move the client past the update ID it was fired from (read from the response itself, so the check doesn't race the Onyx write).
2. The first 200 that fails this check fires one incremental ReconnectApp, whose response always applies and moves the client to the server head, and raises a Log.alert so stalled clients are visible in monitoring.
3. Further fetches from the same client state back off for 60 seconds. The back-off is time-boxed, not permanent: one useless answer only proves the server couldn't serve the range at that moment.

There is no timer machinery. The guard stores when it latched, and the check runs when the next Pusher update arrives anyway. Two ways out of the back-off:

• The client advances (the normal case): instant release. The guard is keyed to the stuck update ID, so the moment the reconnect response moves the client forward, the next event fetches immediately.
• The client is still stuck: the first event past the 60-second mark retries the fetch, and a repeat stall escalates and alerts again.

What this guarantees:

• A stalled client always tries again — no single lost request can permanently stop sync.
• The storm stays dead: at most one fetch and one reconnect per minute per stalled client, versus about one fetch per second before.
• The queue always resumes: no skipped cycle can freeze the app.
• A truly stuck client stays visible: repeating alerts, one per minute, instead of silence.

Failed or unanswered fetches keep the existing retry behavior. Both fetch call sites in OnyxUpdateManager (the gap branch and the pending-updates branch) share the guard, and contributingGuides/SEQUENTIAL_QUEUE.md documents it.

Supersedes #94974. The stall detection idea comes from @elirangoshen's work there; this PR reads the stall from the fetch response instead of a retry counter, and recovers the client instead of going silent.

Fixed Issues

$ #95486
PROPOSAL:

Tests

1. Check out this branch and run npm run test -- tests/unit/OnyxUpdateManagerTest.ts.
2. Verify all 22 tests pass, including the 8 new ones that cover the stall guard:
   • escalates to a single incremental ReconnectApp when the gap fetch returns no progress (also asserts the alert fires and that fetching resumes once the client advances)
   • retries the gap fetch and escalates again after the back-off expires when the client still has not advanced
   • does not escalate when the gap fetch advances the client
   • does not escalate or back off when the fetch response reports progress
   • does not escalate when the client advances through another path while the fetch is in flight
   • does not escalate when the gap fetch fails
   • backs off a pending-updates fetch that returns no progress
   • resumes the SequentialQueue exactly once per backed-off gap push after escalating
3. Sign in with a high-traffic account, trigger a burst of incoming updates (for example open a busy chat while another client posts), and verify the app stays responsive and data stays in sync, with no repeating GetMissingOnyxMessages requests in the network tab.
4. Verify that no errors appear in the JS console.

Offline tests

1. Go offline, then come back online with pending updates on the server.
2. Verify the client fetches missing updates in order, applies them, and resumes the SequentialQueue, with no repeating GetMissingOnyxMessages loop and no stuck or paused queue.

QA Steps

// TODO: These must be filled out, or the issue title must include "[No QA]."

1. Sign in with a high-traffic account.
2. Generate a burst of incoming updates (for example have another user post rapidly in a shared chat, or reconnect after being offline).
3. Verify the app stays responsive during and after the burst, with no freezes or dead clicks.
4. Verify reports and chats stay in sync with the server after the burst settles.
5. Verify that no errors appear in the JS console.

PR Author Checklist

• I linked the correct issue in the ### Fixed Issues section above
• I wrote clear testing steps that cover the changes made in this PR
  • I added steps for local testing in the Tests section
  • I added steps for the expected offline behavior in the Offline steps section
  • I added steps for Staging and/or Production testing in the QA steps section
  • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
  • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
  • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
• I included screenshots or videos for tests on all platforms
• I ran the tests on all platforms & verified they passed on:
  • Android: Native
  • Android: mWeb Chrome
  • iOS: Native
  • iOS: mWeb Safari
  • MacOS: Chrome / Safari
• I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
• I followed proper code patterns (see Reviewing the code)
  • I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
  • I verified that comments were added to code that is not self explanatory
  • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
  • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
• If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
• I followed the guidelines as stated in the Review Guidelines
• I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
• If any new file was added I verified that:
  • The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
• If a new CSS style is added I verified that:
  • A similar style doesn't already exist
  • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))
• If new assets were added or existing ones were modified, I verified that:
  • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
  • The assets load correctly across all supported platforms.
• If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
• If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
• If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
• If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
• If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
  • I verified that all the inputs inside a form are aligned with each other.
  • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
• I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
• If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

🤖 Generated with Claude Code

[mountiny]

This looks sound to me and the comments are self explanatory, thank you!

[adhorodyski]

@codex review

[melvin-bot]

@Krishna2323 Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

[mkhutornyi]

I'll be reviewing this as continuation of #94974

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a68e3353fc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mountiny]

@MelvinBot review

[MelvinBot]

Review — looks solid ✅

The approach is sound and the backend-behavior claims that the whole fix rests on check out against the code:

• The escalation recovery is guaranteed to apply. RECONNECT_APP is in requestsToIgnoreLastUpdateID, so the incremental reconnect's response is applied via OnyxUpdates.apply regardless of the update-ID gap — it can't itself be re-caught by the gap flow. Same set also contains GET_MISSING_ONYX_MESSAGES, which is precisely why a stalled fetch never self-advances and the guard is needed. (SaveResponseInOnyx.ts:16-44)
• The duplicate-drop claim holds. The escalation goes through writeWithNoDuplicatesReconnectConflictAction, so an identical in-flight/queued reconnect is coalesced.
• No freeze on a skipped cycle. isFetchAlreadyStalled sets a resolved query promise, so handleMissingOnyxUpdates still hits finalizeUpdatesAndResumeQueue → clear() (which defaults to shouldUnpauseSequentialQueue: true). Matches the "resume the SequentialQueue exactly once per backed-off push" test.
• Reading progress from response.lastUpdateID first (rather than the module var) correctly avoids racing the Onyx write, and the lastUpdateIDAppliedToClient > lastUpdateIDFromClient fallback covers the advanced-mid-flight case.
• Mock change is scoped. Only the two OnyxUpdateManager suites activate the manual __mocks__/App.ts; the other suites that touch reconnectApp define their own inline jest.mock('@userActions/App', …), so flipping the manual mock's reconnectApp to a jest.fn() doesn't affect them.

Test coverage is comprehensive (escalate, don't-escalate-on-response-progress, don't-escalate-on-advance-mid-flight, don't-escalate/don't-latch-on-failure, pending-updates branch, single-unpause). I could not run the suite in this sandbox (jest execution is blocked here), so I relied on static tracing plus the author's stated 19/19 pass — worth confirming CI is green.

Two things to confirm (not blockers)

1. Guard release depends entirely on the client advancing — what backstops a failed escalation reconnect?

stalledFetchClientUpdateID only clears when lastUpdateIDAppliedToClient moves past it (or via resetDeferralLogicVariables). The recovery therefore hinges on the escalated incremental ReconnectApp actually advancing the client. If that reconnect fails (network error) or the server also can't serve the range and returns no lastUpdateID, the guard stays latched at that ID and every subsequent same-state gap push is skipped indefinitely. There's no freeze (the queue keeps resuming) and the Log.alert gives monitoring visibility — but the client won't re-attempt a fetch or reconnect for that state on its own until something else advances it. Is that acceptable, i.e. is the alert the intended safety net, or should a failed escalation reconnect release the latch so the next push retries? (index.ts:82-113)

2. Primary stall check assumes GetMissingOnyxMessages responses always carry lastUpdateID.

The progress check is Number(response.lastUpdateID ?? 0) > lastUpdateIDFromClient. If a genuinely-successful partial fetch applied onyxData but the response envelope happened to lack lastUpdateID, the fallback lastUpdateIDAppliedToClient > lastUpdateIDFromClient could still race the Onyx write and misfire a (harmless, self-correcting) escalation. In practice these responses do carry lastUpdateID, so this is low risk — just flagging the assumption. (index.ts:89-93)

Nice work — the doc update in SEQUENTIAL_QUEUE.md and the tightly-scoped shared guard across both fetch sites are especially good.

[Copilot]

Pull request overview

This PR adds a “stall guard” to OnyxUpdateManager so that when GetMissingOnyxMessages repeatedly returns a successful (200) response that does not advance the client’s applied update ID, the client escalates recovery to a single incremental ReconnectApp and then backs off further missing-update fetch attempts until the client advances. This aims to prevent a ~1 request/second fetch loop and keep the SequentialQueue from freezing under high-traffic update bursts.

Changes:

• Add stall detection to missing-update fetches and escalate to incremental ReconnectApp when a 200 response doesn’t advance the client.
• Share the stall/backoff guard across both the gap-detection branch and the pending-updates branch.
• Add unit tests and mock plumbing to cover stall escalation, backoff behavior, and queue resume invariants; document the guard in SEQUENTIAL_QUEUE.md.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/unit/OnyxUpdateManagerTest.ts	Adds new unit tests validating stall escalation/backoff and SequentialQueue resume behavior.
src/libs/actions/OnyxUpdateManager/index.ts	Implements stall detection, escalation to incremental reconnectApp, and backoff gating shared by both fetch call sites.
src/libs/actions/mocks/App.ts	Extends the App action mock to support returning a “no progress” GMOM response and to observe reconnectApp calls in tests.
contributingGuides/SEQUENTIAL_QUEUE.md	Documents the new stall guard behavior and its interaction with SequentialQueue deduping and recovery.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[adhorodyski]

Answering the two confirm-items from #95477 (comment):

1. What backstops a failed escalation reconnect? The escalated reconnectApp goes through API.writeWithNoDuplicatesReconnectConflictAction, i.e. a persisted SequentialQueue write — a network failure retries it automatically until it's answered, so the recovery can't silently vanish. The latch holding until the client advances is intentional: releasing it on failure would just resume the useless fetch loop this PR exists to stop. The only residual case is a server that 200s the reconnect without advancing the client, and the Log.alert is exactly the safety net for that.

2. lastUpdateID assumption on GMOM responses. Correct, and accepted: worst case is one redundant incremental ReconnectApp that the queue dedupes and that self-corrects.

[adhorodyski]

@mkhutornyi addressed everything

[mountiny]

@mkhutornyi lets prioritize this one please! thanks

[mkhutornyi]

Reviewer Checklist

• I have verified the author checklist is complete (all boxes are checked off).
• I verified the correct issue is linked in the ### Fixed Issues section above
• I verified testing steps are clear and they cover the changes made in this PR
  • I verified the steps for local testing are in the Tests section
  • I verified the steps for Staging and/or Production testing are in the QA steps section
  • I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
  • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
• I checked that screenshots or videos are included for tests on all platforms
• I included screenshots or videos for tests on all platforms
• I verified that the composer does not automatically focus or open the keyboard on mobile unless explicitly intended. This includes checking that returning the app from the background does not unexpectedly open the keyboard.
• I verified tests pass on all platforms & I tested again on:
  • Android: HybridApp
  • Android: mWeb Chrome
  • iOS: HybridApp
  • iOS: mWeb Safari
  • MacOS: Chrome / Safari
• If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
• I verified proper code patterns were followed (see Reviewing the code)
  • I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
  • I verified that comments were added to code that is not self explanatory
  • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
  • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
• If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
• I verified that this PR follows the guidelines as stated in the Review Guidelines
• I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
• If a new component is created I verified that:
  • A similar component doesn't exist in the codebase
  • All props are defined accurately and each prop has a /** comment above it */
  • The file is named correctly
  • The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
  • The only data being stored in the state is data necessary for rendering and nothing else
  • For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
  • Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
  • All JSX used for rendering exists in the render method
  • The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
• If any new file was added I verified that:
  • The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
• If a new CSS style is added I verified that:
  • A similar style doesn't already exist
  • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)
• If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
• If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
• If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
• If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
• If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
  • I verified that all the inputs inside a form are aligned with each other.
  • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
• For any bug fix or new feature in this PR, I verified that sufficient unit tests are included to prevent regressions in this flow.
• If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
• I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: HybridApp

web-android.mov

Android: mWeb Chrome

iOS: HybridApp

ios-web.mov

iOS: mWeb Safari

MacOS: Chrome / Safari

web-android.mov

[MelvinBot]

Review — LGTM ✅

Re-traced the stall guard against the current branch (95432833). The logic is correct, the two concerns from my earlier pass were answered by the author, and CI is green: all 8 test jobs pass (the only failing check is the human PR Author/Reviewer Checklist gate). The reviewer checklist is complete.

Key invariants re-verified against the code:

Details

• Escalation fires an incremental reconnect. reconnectApp(lastUpdateIDFromClient) is a partial reconnect whenever the ID is truthy (updateIDFrom truthy → isFullReconnect = false), which is always the case on the gap branch since !lastUpdateIDFromClient is handled earlier. (App.ts:504-508)
• Progress is read from the response first, module var as fallback — Number(response.lastUpdateID ?? …) > lastUpdateIDFromClient || lastUpdateIDAppliedToClient > lastUpdateIDFromClient — which avoids racing the Onyx write and still catches the advanced-mid-flight case. (OnyxUpdateManager/index.ts:92)
• The back-off never freezes the queue. isFetchAlreadyStalled sets a resolved query promise, so checkIfClientNeedsToBeUpdated still returns true → finally(finalizeUpdatesAndResumeQueue) → clear() unpauses. The dedicated "resume exactly once per backed-off push" test covers this. (index.ts:118-120)
• Guard release is keyed to the stuck ID, so the moment the reconnect advances the client the next push (higher lastUpdateIDFromClient) no longer matches and fetches immediately; time-boxing at 60s is the fallback when the client stays stuck.
• Failure doesn't latch — the jsonCode !== 200 early return leaves failed/unanswered fetches to the existing retry paths, so a network blip can't wedge the guard.
• Both fetch sites share fetchMissingUpdates, and the mock change is scoped (only the two OnyxUpdateManager suites use the manual __mocks__/App.ts).

One non-blocking nit: escalateIfFetchStalled skips validateAndApplyDeferredUpdates, so finalizeUpdatesAndResumeQueue clears the deferred queue on the backed-off cycle. That's fine (those updates sit behind the unclosed gap and the reconnect re-fetches from head), and the inline comment already explains it — just calling out that the deferred payload is discarded rather than held, which is the intended behavior.

Nice work — the shared guard, the response-first stall read, and the SEQUENTIAL_QUEUE.md doc update are all clean.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9543283773

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mkhutornyi]

NAB: I think inline comments can be simplified and more user-friendly.
For now, they're difficult to read.

Examples:

[OSBotify]

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

[OSBotify]

🚀 Deployed to staging by https://github.com/mountiny in version: 9.4.33-0 🚀

platform	result
🕸 web 🕸	success ✅
🤖 android 🤖	success ✅
🍎 iOS 🍎	cancelled 🔪

[MelvinBot]

🤖 I reviewed the changes and no help site updates are required under App/docs/articles.

This PR is an internal sync-reliability change — it adds a stall guard to the client's GetMissingOnyxMessages fetch loop in OnyxUpdateManager so a server 200 that doesn't advance the client can't trigger a per-second fetch storm. The changed files are all invisible plumbing:

• src/libs/actions/OnyxUpdateManager/index.ts — client-side Onyx update sync logic
• src/CONST/index.ts — a supporting constant
• contributingGuides/SEQUENTIAL_QUEUE.md — internal developer documentation (not the public help site)
• tests/unit/OnyxUpdateManagerTest.ts + src/libs/actions/__mocks__/App.ts — tests/mocks

There is no change to any user-facing feature, setting, tab, or button, so nothing in the customer-facing help site needs to be updated. No draft docs PR was created.

@adhorodyski, please review the linked help site PR and confirm it reflects the current behavior. Then mark the linked help site PR Ready for review