fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client

[mountiny]

Problem

When the app spots a gap in the server's update IDs it fetches the missing range with GetMissingOnyxMessages. Production traces show a failure mode where the server keeps answering that fetch with a 200 that carries no usable updates. The client never advances, so every incoming Pusher update re-detects the same gap and fires the same fetch again, about once per second (traces show 12 to 47 requests in a row). The earlier fix in #93948 only stopped fetches that overlap in time; it cannot stop this loop because each fetch finishes before the next one starts.

Solution

This PR treats one useless answer as proof instead of retrying. A successful fetch must advance the client past the update ID it was fired from, checked via response.lastUpdateID so it does not race the Onyx write. The first 200 that fails this check escalates to a single incremental ReconnectApp from the client's current update ID (deduplicated by the SequentialQueue). Until the client advances, further fetches from the same client state are skipped while the SequentialQueue still resumes each cycle. Both the gap branch and the pending-updates branch of OnyxUpdateManager share this guard, and a Log.alert captures client and response update IDs for monitoring.

PR

#95477

Issue Owner

Current Issue Owner: @adhorodyski

[MelvinBot]

Issue Analysis

Root cause (confirmed, high confidence): In the current main code, nothing forces the client's lastUpdateIDAppliedToClient watermark to advance after a GetMissingOnyxMessages (GMOM) fetch returns 200. The watermark only advances when OnyxUpdates.apply sees a response.lastUpdateID that is truthy and strictly greater than the current value. So a 200 that carries no usable updates (empty onyxData, or lastUpdateID missing/<= the fired-from ID) leaves the gap intact. After the fetch settles, finalizeUpdatesAndResumeQueue clears all three guards, so the next Pusher update re-detects the same gap and re-fires GMOM — a self-perpetuating serial loop (~1/sec, matching the reported 12–47 requests in a row).

The earlier fix in #93948 is correctly characterized as insufficient: it added the in-flight getMissingOnyxUpdatesQueryPromise() check to the gap branch guard, which only coalesces time-overlapping fetches. That promise is null once each fetch finalizes, so it provides zero protection against a serial loop where each fetch completes before the next one begins.

PR #95477 — assessment: The approach is sound and targets the exact gap above. It treats one non-advancing 200 as proof (rather than retrying): escalateIfFetchStalled checks response.lastUpdateID against the fired-from ID (reading the response directly, so it doesn't race the Onyx write), escalates once to an incremental ReconnectApp deduped by the SequentialQueue, and skips further fetches from the same client state until the client actually advances. Both the gap branch and the pending-updates branch share the guard, and a Log.alert surfaces stalled clients for monitoring.

Investigation details & evidence

Trigger path. OnyxUpdateManager subscribes to ONYX_UPDATES_FROM_SERVER and runs handleMissingOnyxUpdates per update. The gap branch fires GMOM whenever a server update implies the client is behind, unless one of three guards short-circuits: areDeferredUpdatesQueued, isFetchingForPendingUpdates, or an in-flight getMissingOnyxUpdatesQueryPromise().

Why the watermark can stall. GET_MISSING_ONYX_MESSAGES is in requestsToIgnoreLastUpdateID in SaveResponseInOnyx, so the response is applied directly. In OnyxUpdates.apply, didAdvanceLastUpdateID is only true for a real, higher lastUpdateID; otherwise the watermark holds and applyHTTPSOnyxUpdates applies an empty onyxData (no-op).

Why it loops. After settling, .finally(finalizeUpdatesAndResumeQueue) resolves/recreates the query promise, clears deferred updates, and resets isFetchingForPendingUpdates. With the watermark unchanged, the next Pusher update re-enters the gap branch with every guard clear → same gap, same fetch. validateAndApplyDeferredUpdates's repeated-params abort only protects the recursive in-fetch loop for a single invocation, not across separate Pusher-driven invocations.

Related: #92541 is the parent performance issue (GMOM/ReconnectApp/OpenApp stampedes) that PR #93948 addressed; this issue is the follow-up serial-loop variant that #93948 could not cover.

Open question (backend, outside App): The exact server-side trigger — whether GetMissingOnyxMessages genuinely returns 200 with lastUpdateID <= updateIDFrom / empty onyxData for the requested range — lives in Auth/Web-Expensify and can't be confirmed from the App repo. It isn't needed to confirm the client-side loop is real, but the PR's Log.alert (client + response update IDs) is the right hook to feed that separate backend investigation. Worth confirming from production logs that consecutive GMOM requests repeat the same updateIDFrom at ~1/sec.

---

This issue is authored by mountiny with PR #95477 already in review, so the fix is in flight — this is a validation of the root cause and approach rather than a new proposal. Reply with @MelvinBot <feedback> to refine.