Investigate why some changes didn't make it to staging as expected

[roryabraham]

Context

• https://expensify.slack.com/archives/C03V9A4TB/p1659395657181009
• https://expensify.slack.com/archives/C03TQ48KC/p1659459369071089

Problem

There is a very obscure edge case with our git CI that caused some changes on main to not be correctly propagated to staging during a staging deploy. There were a few affected files, but the one we've focused on as our case study is src/pages/settings/Payments/PaymentsPage/BasePaymentsPage.js. Here's my best attempt at summarizing the history of this file.

1. The OfflineIndicator was added in cc59b0a (#9589).
2. That change was deployed to staging in 1.1.86-0
3. The change was reverted here. Note that this PR was meant to be a revert of Update Global Offline Indicator #9895, but also included an additional (potentially unrelated) change in BasePaymentsPage.
4. The revert was cherry-picked to staging version 1.1.86-2 in ffd8581 (#10115)
5. The OfflineIndicator was added back in f9cf909 (#10135)

At this point, main contained the OfflineIndicator in BasePaymentsPage, but staging did not. However, when main was merged into staging, the usage of OfflineIndicator was carried over to main but the import was not.

In order to reproduce this locally and see it in action:

git checkout main
git pull
git checkout staging
git pull
git checkout main
git reset e66fa20d689e704fe6cf30485bc45f1e5e1397d7
git checkout -b fake-main
git reset --hard HEAD
git checkout staging
git reset 751915e541bfad2352237ce9d6030413b23938f0
git checkout -b fake-staging
git reset --hard HEAD
git clean -fd
git checkout -b update-fake-staging-from-fake-main

Then look at the diff between update-fake-staging-from-fake-main:

You would think both of these changes would be carried over, but then run:

git merge -Xtheirs fake-main

And you'll see that the import is missing.

Solution

TBD, but we have some additional observations/hypotheses:

• When we cherry-picked the revert to staging, we somehow muddled the history such that git thinks that change (ffd8581) is newer than the change on main that added it back (f9cf909).
• An examination of the git log shows that on main the commits from the revert PR are part of a linear history, while on staging the merge commit is included and is treated as a separate branch?

• The next step to investigate how to prevent this from happening in the future is probably to try and reproduce it in a fresh repo (perhaps utilizing this existing test script), and fiddle with the cherry-pick command that's used to see if we can keep these histories in line and prevent this problem.

[roryabraham]

cc @Expensify/mobile-deployers – @johnmlee101, @rafecolton, @yuwenmemon, and I spent a few hours trying to figure this out this morning. We made some good progress but didn't quite get it worked out.

It seems like a pretty extreme edge case, but the main moral for now is be careful with your cherry-picks, particularly with reverts. Hopefully we'll have more tangible guidance or improvements here soon.

[roryabraham]

cc @kidroca in case you're interested in investigating

[roryabraham]

Found another problem scenario in this workflow run:

• Staging was at b568742

• ee8989d was CP'd to staging:

  git cherry-pick -S -x --mainline 1 --strategy=recursive -Xtheirs ee8989dc7cb5a1debcbf71ea3227e9105112afe1

• Then staging included this unrelated change, which was CP'd in 🍒 Cherry pick PR #10229 to staging 🍒 #10295

  buildConfigField "boolean", "IS_NEW_ARCHITECTURE_ENABLED", isNewArchitectureEnabled().toString()
  
      if (isNewArchitectureEnabled()) {
          // We configure the NDK build only if you decide to opt-in for the New Architecture.
          externalNativeBuild {
              ndkBuild {
                  arguments "APP_PLATFORM=android-21",
                      "APP_STL=c++_shared",
                      "NDK_TOOLCHAIN_VERSION=clang",
                      "GENERATED_SRC_DIR=$buildDir/generated/source",
                      "PROJECT_BUILD_DIR=$buildDir",
                      "REACT_ANDROID_DIR=$rootDir/../node_modules/react-native/ReactAndroid",
                      "REACT_ANDROID_BUILD_DIR=$rootDir/../node_modules/react-native/ReactAndroid/build",
                      "NODE_MODULES_DIR=$rootDir/../node_modules"
                  cFlags "-Wall", "-Werror", "-fexceptions", "-frtti", "-DWITH_INSPECTOR=1"
                  cppFlags "-std=c++17"
                  // Make sure this target name is the same you specify inside the
                  // src/main/jni/Android.mk file for the `LOCAL_MODULE` variable.
                  targets "rndiffapp_appmodules"
              }
          }
          if (!enableSeparateBuildPerCPUArchitecture) {
              ndk {
                  abiFilters (*reactNativeArchitectures())
              }
          }
      }

[roryabraham]

I'm able to reproduce the initial problem in a fresh repo. Added unit tests covering this here: #10316

Now need to figure out how to fix this

[roryabraham]

I believe this article basically explains exactly what is happening to us

[rafecolton]

This is the next article in the series where the author describes how to fix the problem. I'm not immediately sure how to do that in code in our automation, but given that they are describing the exact problem we are experiencing, I think this is the solution.

[tgolen]

Oh man, that article ends with more questions than it begins with!

…

On Tue, Aug 9, 2022 at 2:26 PM Rafe Colton ***@***.***> wrote: This <https://devblogs.microsoft.com/oldnewthing/20180314-00/?p=98235> is the next article in the series where the author describes how to fix the problem. I'm not immediately sure how to do that in code in our automation, but given that they are describing the exact problem we are experiencing, I think this is the solution. — Reply to this email directly, view it on GitHub <#10214 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMAB6GWO4QYDMKG3YKZQLVYK5G3ANCNFSM55MK2K7Q> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[roryabraham]

It's really hard to create a TL;DR of this problem because it's pretty convoluted, but these diagrams are pretty good. Note that in our example, our equivalent of master is actually staging, and our equivalent of feature is actually main:

Before (with cherry-pick)

After (by merging into a "helper" branch)

---

So I pushed a number of changes to this PR to:

• DRY things up to (hopefully) make the tests more readable
• Recreate the problem scenario and cover it with some tests / assertions
• Implement the solution in the article

If you check out that PR, you can run the tests locally with: ./tests/unit/gitTestCI.sh || rm -rf ~/DumDumRepo.

If you comment out the assertions in the tests that look like this:

assert_equal "$output" "[ '7', '5', '1' ]"

Then you'll see the solution outlined in the article indeed fixes the scenario where a revert PR is persisted on staging, even after the same change is un-reverted on main and then main is merged into staging. However, I don't think this really works for us because it breaks other tests. I'm trying hard to understand the problem and solution here, but due to the urgency I could use more eyes.

[roryabraham]

Okay, I believe I know how to fix this. I included some diagrams to explain the fix in the draft PR. I also got tests passing but need to implement the changes in the actual GitHub Workflows. Also I just want to carry this out with more examples to try and see if we can find any other cases where it doesn't work. For example, I haven't yet explicitly tested this slightly different scenario that's currently blocking deploys. I'm also not sure if there's anything we'll need to do to get main + staging back into a state where this solution will work.

The TL;DR of this solution is that we actually merge the temporary CP branch into both main and staging, instead of just staging. That way, it becomes a more recent common ancestor of both those branches and fixes the later merge that happens when main is merged back into staging. One sort of annoying side-effect of this is that it will mean one more OSBotify PR merged into main for every PR we cherry-pick. Not a dealbreaker though, that's just what we have to do to auto-commit code into protected branches.

[tgolen]

My concern with this is that we're digging so deep into the weeds that we're ("we" as in, anyone that isn't Rory 😁 ) never going to be able to maintain this or fix it if it breaks.

Thinking back to web-e deploying, we've never ran into this problem and that's a deploy that's been pretty stable for many years (with a CP process to both staging and production branches).

So, at some point... do we sit back and ask ourselves the question: Is this still the right direction to be pushing ourselves in?

GitHub actions work great for simple repos (@AndrewGable loved implementing them on comp, k2, etc.), but when it comes to very complex deploy flows like this, I think that's when the complexity begins to be more than we have the capacity to manage.

I know that doesn't really help to solve this particular issue, and I'm still very lost on it, but I can't escape the feeling like we've dug ourselves into a hole and we just keep digging deeper :(