[Due for payment 2025-10-30] Make export onyx state feature have a whitelist of things we can export and have it redact everything else

[iwiznia]

Problem

The export onyx state feature exports all the data in our onyx tables. We had some data privacy issues in the past where things that we should not be exporting were exported, so we want to avoid that by implementing a whitelist. Everything we whitelist, should be exported as is, everything not in the whitelist should be redacted.

We did this for a few keys here #70230 and here #70223 but we want to extend that to all keys.

Issue Owner

Current Issue Owner: @abekkala

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ishpaul777 (External)

[melvin-bot]

Triggered auto assignment to @abekkala (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[samranahm]

@iwiznia Is this open for external contributor? I can pick this UP.

[iwiznia]

Yes, it is

[mkzie2]

🚨 Edited by proposal-police: This proposal was edited at 2025-09-25 19:00:48 UTC.

Proposal

Please re-state the problem that we are trying to solve in this issue.

Make export onyx state feature have a whitelist of things we can export and have it redact everything else

What is the root cause of that problem?

Improvement

What changes do you think we should make in order to solve the problem?

1. We need to create a whitelist configuration const (We need to re-confirm again in PR phase with internal engineer)

const ONYX_KEY_EXPORT_POLICIES = {
    // This is for early return key to skip extra logics, if there is any key that we don't want to be saved, put it here
    EXCLUDED_KEYS: [
        ONYXKEYS.NVP_PRIVATE_PUSH_NOTIFICATION_ID,
        ONYXKEYS.CREDENTIALS,
        ONYXKEYS.STASHED_CREDENTIALS,
        ONYXKEYS.NVP_PRIVATE_STRIPE_CUSTOMER_ID,
        ONYXKEYS.NVP_PRIVATE_BILLING_DISPUTE_PENDING,
        ONYXKEYS.NVP_PRIVATE_BILLING_STATUS,
        ONYXKEYS.PLAID_LINK_TOKEN,
        ONYXKEYS.ONFIDO_TOKEN,
        ONYXKEYS.ONFIDO_APPLICANT_ID,
        // Add other sensitive keys that should never be exported
    ],
    
    FIELD_POLICIES: {
        [ONYXKEYS.SESSION]: {
            allowList: ['email', 'accountID', 'loading', 'creationDate', 'errors'],
            maskList: [], // Fields to mask but preserve structure
        },
        [ONYXKEYS.STASHED_SESSION]: {
            allowList: ['email', 'accountID', 'loading', 'creationDate', 'errors'],
            maskList: [],
        },
        [ONYXKEYS.ACCOUNT]: {
            allowList: ['validated', 'isFromPublicDomain', 'isUsingExpensifyCard', 'accountID'],
            maskList: ['primaryLogin', 'email'], // Masked but preserved for debugging
        },
        [ONYXKEYS.PERSONAL_DETAILS_LIST]: {
            allowList: ['accountID', 'timezone', 'status', 'pronouns'],
            maskList: ['firstName', 'lastName', 'displayName', 'avatar', 'login'],
        },
        [ONYXKEYS.COLLECTION.REPORT]: {
            allowList: ['reportID', 'type', 'chatType', 'stateNum', 'statusNum', 'isOwnPolicyExpenseChat', 'participantAccountIDs', 'total', 'currency', 'created'],
            maskList: ['reportName', 'description', 'ownerAccountID', 'managerID'],
        },
        [ONYXKEYS.COLLECTION.TRANSACTION]: {
            allowList: ['transactionID', 'reportID', 'amount', 'currency', 'created', 'category', 'tag', 'billable'],
            maskList: ['merchant', 'description', 'comment'],
        },
        [ONYXKEYS.COLLECTION.POLICY]: {
            allowList: ['id', 'type', 'role', 'outputCurrency', 'isPolicyExpenseChatEnabled', 'areCategoriesEnabled', 'areTagsEnabled'],
            maskList: ['name', 'avatar'],
        },
        [ONYXKEYS.USER_WALLET]: {
            allowList: ['currentBalance', 'availableBalance', 'tierName'],
            maskList: [],
        },
        [ONYXKEYS.BANK_ACCOUNT_LIST]: {
            allowList: ['accountType', 'currency'],
            maskList: ['accountNumber', 'routingNumber', 'addressName'],
        },
        [ONYXKEYS.CARD_LIST]: {
            allowList: ['accountID', 'bank', 'isVirtual', 'cardID'],
            maskList: ['lastFourPAN', 'nameOnCard'],
        },
        // Add more keys as needed...
    },
    
    // Default rule for keys not explicitly configured
    DEFAULT_RULE: {
        allowList: [], // By default, allow nothing
        maskList: [], // Don't mask anything by default
        // Everything gets redacted unless explicitly allowed
    }
};

2. Add the util function

type ExportRule = {
    allowList: string[];
    maskList: string[];
};

const processOnyxKeyWithRule = (key: string, data: unknown, rule: ExportRule): unknown => {
    if (data === null || data === undefined) {
        return data;
    }
    
    if (Array.isArray(data)) {
        return data.map(item => 
            typeof item === 'object' ? processOnyxKeyWithRule(key, item, rule) : item
        );
    }
    
    if (typeof data === 'object') {
        const processedData: Record<string, unknown> = {};
        
        Object.keys(data).forEach(fieldKey => {
            const fieldValue = data[fieldKey];
            
            if if (rule.maskList.includes(fieldKey)) {
                // Field should be masked but preserved
                processedData[fieldKey] = maskValuePreservingLength(fieldValue);
            } else (rule.allowList.includes(fieldKey))  {
                // Field is explicitly allowed
                processedData[fieldKey] = fieldValue;
            } else {
                // Field is not in allowList or maskList - redact it
                processedData[fieldKey] = MASKING_PATTERN;
            }
        });
        
        return processedData;
    }
    
    return data;
};

const maskOnyxState: MaskOnyxState = (data, isMaskingFragileDataEnabled) => {
    let onyxState = { ...data };
    
    // Process each onyx key according to its rule
    Object.keys(onyxState).forEach(key => {
        // Check if key should be completely excluded - This is optional, we can remove this logic in PR phase if we don't find it helpful
        if (ONYX_KEY_EXPORT_POLICIES.EXCLUDED_KEYS.includes(key)) {
            delete onyxState[key];
            return;
        }
        
        // Handle collection keys (keys that start with a collection prefix)
        let ruleKey = key;
        const collectionKey = Object.values(ONYXKEYS.COLLECTION).find(cKey => key.startsWith(cKey));
        if (collectionKey) {
            ruleKey = collectionKey;
        }
        
        // Get rule for this key 
        const rule = ONYX_KEY_EXPORT_POLICIES.FIELD_POLICIES[ruleKey];
        
        // Process the key's data according to its rule
       if (rule) {
            onyxState[key] = processOnyxKeyWithRule(key, onyxState[key], rule);
       }
    });
    
    // Apply additional fragile data masking if enabled (for backwards compatibility)
    if (isMaskingFragileDataEnabled) {
        onyxState = maskFragileData(onyxState) as OnyxState;
    }
    
    emailMap.clear();
    return onyxState;
};

Then we want to apply to all places such as

• Primary export: src/pages/settings/Troubleshoot/TroubleshootPage.tsx
• Debug tools: src/pages/Debug/DebugJSON.tsx (copy onyx data functionality)
• Context menu: src/pages/home/report/ContextMenu/ContextMenuActions.tsx (copy onyx data action)
• Troubleshoot data tools: src/components/RecordTroubleshootDataToolMenu/ (all platform variants)
• Import/Export utilities: All components that use ExportOnyxState.maskOnyxState()

....

What specific scenarios should we cover in automated tests to prevent reintroducing this issue in the future?

Update ExportOnyxStateTest to cover the whitelist changes

What alternative solutions did you explore? (Optional)

Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

[melvin-bot]

@ishpaul777 Eep! 4 days overdue now. Issues have feelings too...