[$1000] Web - Onyx: out-of-order updates

[kbecciv]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

This bug comes from discussion in #26592
TL;DR: Onyx processes the successData and API response in the wrong order on iOS and Android.
Dive deeper:

App/src/libs/actions/Policy.js

Line 408 in 3352cc8

API.write('AddMembersToWorkspace', params, {optimisticData, successData, failureData});

When adding a member to workspace here, we send the following data:

"optimisticData": {
    "onyxMethod": "set",
    "key": "report_3008335418540078",
    "value": {
        "type": "chat",
        "chatType": "policyExpenseChat",
        "ownerAccountID": 899964195,
        "participantAccountIDs": [
            15664021,
            899964195
        ],
        "policyID": "41A8E180FAEB43DF",
        "reportID": "3008335418540078",
        "reportName": "Chat Report",
        "pendingFields": {
            "createChat": "add"
        },
        "isOptimisticReport": true
        ...
    }
},
"successData": {
    "onyxMethod": "merge",
    "key": "report_3008335418540078",
    "value": {
        "pendingFields": {
            "createChat": null
        },
        "errorFields": {
            "createChat": null
        },
        "isOptimisticReport": false
    }
}

And the API call returns the following onyx merge_collection action:

{
    "jsonCode": 200,
    "requestID": "806fecea0afd3512-WAW",
    "onyxData": [
         {
            "onyxMethod": "mergecollection",
            "key": "report_",
            "value": {
                ...
                "report_3008335418540078": {
                    "reportID": "3008335418540078",
                    "reportName": "",
                    "type": "chat",
                    "chatType": "policyExpenseChat",
                    "ownerEmail": "[pasha@test.co](mailto:pasha@test.co)",
                    "ownerAccountID": 15671904,
                    "policyID": "41A8E180FAEB43DF",
                    "participants": [
                        "[pasha@test.co](mailto:pasha@test.co)",
                        "[paultsimura+3@gmail.com](mailto:paultsimura+3@gmail.com)"
                    ],
                    "participantAccountIDs": [
                        15664021,
                        15671904
                    ],
                    "lastActorAccountID": 15671904,
                },   
            }
        },
    ]
}

It should replace the temporary accountID of the report.ownerAccountID with the real one, which comes from the server.
On the web, the onyx actions are executed in the following order:

1. persist optimisticData;
2. persist successData;
3. persist API response;
   The issue is that on iOS, the onyx actions are executed in the following order:
4. persist optimisticData;
5. persist API response;
6. persist successData;
   The successData holds merge operation of only several certain fields, which shouldn't affect the report much.
   However, here's the tricky part:
   This piece of code, which creates a promise for future merge operation, is executed before Onyx persists the API response:
   https://github.com/Expensify/react-native-onyx/blob/616c4c5286b7a7fadac18d477cd75edd60865b12/lib/Onyx.js#L1136-L1147
   Meaning the following execution chain:
7. persist optimisticData;
8. create persistSuccessData promise;
9. persist API response;
10. execute persistSuccessData promise;
    This means, at the moment persistSuccessData is created, the existingValue is still not updated, so even though it should update only a couple of fields, the modifiedData here is saved in a state of pre-API-response.
    Then the report is updated by the "persist API response" operation, but gets overwritten by the persistSuccessData operation with the outdated data.
    If we debug step-by-step, you'll notice that the chat name is displayed correctly between the steps 3 and 4 from above.
    As an outcome of this complete operation, the state of Onyx is the following:

• personalDetailsList contains the new permanent ownerAccountID;
• report has the temporary ownerAccountID, which is already removed from Onyx.

Expected Result:

Onyx updates are executed in correct order without data being overwritten

Actual Result:

Onyx updates face a race condition and some data is overwritten

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: 1.3.76.0
Reproducible in staging?: n/a
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation

n/a

Expensify/Expensify Issue URL:
Issue reported by: @@paultsimura
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1696287156456449

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~013373cff08a2575e9
• Upwork Job ID: 1709846441891180544
• Last Price Increase: 2023-10-05

[melvin-bot]

Triggered auto assignment to @CortneyOfstad (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[ospfranco]

I believe I found the cause.

When the update function is called, all the operations are fired in an async manner. However, this causes a race condition where certain operations will be written into the DB before others. It's a built-in race condition.

Just for documentation purposes: I suggested we need to make the update operations sequentially await. @marcaaron @tgolen are however worried about the app taking a big performance hit. I will try another solution and report back if I manage to get the app to do reliable updates.

[pradeepmdk]

I believe I found the cause.

When the update function is called, all the operations are fired in an async manner. However, this causes a race condition where certain operations will be written into the DB before others. It's a built-in race condition.

Just for documentation purposes: I suggested we need to make the update operations sequentially await. @marcaaron @tgolen are however worried about the app taking a big performance hit. I will try another solution and report back if I manage to get the app to do reliable updates.

for note: that is what I already told in my proposal. #26592 (comment)
cc: @Julesssss @ospfranco

[pradeepmdk]

Another problem in the App side:

Proposal

Please re-state the problem that we are trying to solve in this issue.

Onyx updating on race condition

What is the root cause of that problem?

https://github.com/Expensify/App/assets/16595846/736946e5-183f-4b16-969a-c0ef118ad228

@marcaaron I saw recently it changed to onyx to memory here
#25614

1. we are flushed out to Onyx once all the api is completed in the queue (networkRequestQueue). I think this will create a performance issue right? ex: here in the queue 2035 is pending after coming online it gets a response object in the queuedOnyxUpdates to 46k object.

2. Save queued updates to memory only #25614 Due to this change incase the middle of the response I refresh the page means it will never update to onyx.

What changes do you think we should make in order to solve the problem?

App/src/libs/Network/SequentialQueue.js

Line 126 in 41ddd74

flushOnyxUpdatesQueue();

we need to flush out each api success or failure.

cc: @marcaaron @tgolen @ospfranco @mountiny

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~013373cff08a2575e9