[#quality Improvement] Monitor bugs posted in #quality to get help faster.

[mallenexpensify]

Coming from this post by @quinthar in #quality.

Regarding the above, I feel like most of the real world issues I experience are really hard to reproduce, so fixing them it feels like we need to get really good at diagnosing in the real world. This is not a new statement, but despite us talking about it again and again it doesn't feel like we're getting much better at this. Can we create a process for essentially pulling in an engineer quickly?

3/27 Update as I'm wanting to close this out. We can create additional issue with more granular improvements, as needed.

• https://github.com/Expensify/Expensify/issues/471436
  • This is important cuz priority issues denoted as CRITICAL or HIGH in #quality bug reports will get AutoAssignerNewDotQuality added to assign an engineer. These issues are now Daily too (used to be Weekly)
• Updated How do I submit a NewDot / New Expensify performance or reliability bug to #quality? Troubleshooting to include a tl,dr at top so BZs and others don't have to read the full SO. (posting faster > fixed faster)
• Post the updated tl,dr in #quality and pin to room
  • Include adding :ladybug: in post above or separate
• Post in #bug-zero (as part of weekly update)
• Chatting with Isabella with QA about this

Later/v2

• Full audit/review/update of How do I submit a NewDot / New Expensify performance or reliability bug to #quality? Troubleshooting
• I created a separate issue to work on the below too
• [HOLD][#quality improvement] Audit our code and output what we think are the arch patterns that prevent us from finding the root causes of bugs easily
• Get a C+ or agency dev or multiples to monitor #quality (knowing the hours are often 'off')?

Other/old ideas

• Get a C+ or multiples to monitor (likely better time zones, not as much knowledge about performance as some agency devs). They could start by shadowing agency devs, all of them should be in #quality now.
• Offer compensation for C+ who are able to help, on a one-off basis (ie. for documenting reproduction steps or for each live troubleshooting session.
• Auto-assign someone, somehow to an hourly GH (which internal engineers would likely hate)
• Build out our tool kit to help capture logs and profiling easier (currently working on)
• Work with a third party, like jam.dev to help with monitoring and troubleshooting (since we'd need a compliance review, we could start using the tool with a small set of users then expand)
• Do something with the QA team as part of their reporting process (unsure what this might be)

[melvin-bot]

@mallenexpensify Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[mallenexpensify]

Will get to next week, off Thurs/Fri.

[melvin-bot]

@mallenexpensify Huh... This is 4 days overdue. Who can take care of this?

[mallenexpensify]

Struggling to get caught up, once I do, I'll address this (I have ideas)

[mallenexpensify]

First thought while updating statuses. We only care about HIGH or CRITICAL issues, so denoting those (or possible ones to get quick 👀 on) is what's important. This could be done with a reacji but I think just tagging me, someone, or a saved term might be quickest/easiest (I'm out two weeks in April, so planning for that.

Also need to encourage add the ladybug emoji to OP ASAP for any issues that might be HIGH or CRITICAL, also AutoAssignerNewDotQuality.

Using a Slack Workflow or Custom App (Best for One Channel)

If you’re in a paid Slack workspace, you can create a workflow in that specific channel:
1. Use Slack Workflow Builder (if enabled).
2. Create a workflow that triggers when a message contains a specific word.
3. Set it to send a DM to you whenever that word appears in that channel.

💡 Best option if you have Workflow Builder access!

Using a Slack Bot or External App (Advanced)

If you’re tech-savvy, you can:
• Set up a Zapier automation to notify you via DM when a keyword appears in a certain channel.
• Use a Slack bot (like Reacji Channeler) to monitor a keyword and alert you.

💡 Requires extra setup but can work in one channel only!

[blazejkustra]

Thanks for opening this discussion! I agree that we need to improve how we diagnose real-world issues, but I have some thoughts on the points raised:

Regarding the above, I feel like most of the real-world issues I experience are really hard to reproduce…

I don’t think this is entirely true. While David’s account may be a special case and difficult to debug, most issues are reproducible. The bigger challenge is likely that our app logic is complex, making root cause analysis harder.

I think we need to do something different, otherwise these issues will just keep piling up.

I completely agree. Personally, I would prioritize improving async debugging tools and making it clearer and easier for users to record and report issues. Right now, we often have to manually ask for Onyx state or screen recordings—this process could be streamlined.

First thought while updating statuses. We only care about HIGH or CRITICAL issues.

This might be tricky because it’s not always clear how to determine if an issue should be marked as HIGH or CRITICAL. Having clear guidelines for this would help ensure consistency.