Skip to content

Fix rebalance/monitor races: probe kill, oscillation, double-pay#521

Draft
markettes wants to merge 8 commits into
mainfrom
fix/rebalance-payment-hash-bug
Draft

Fix rebalance/monitor races: probe kill, oscillation, double-pay#521
markettes wants to merge 8 commits into
mainfrom
fix/rebalance-payment-hash-bug

Conversation

@markettes

Copy link
Copy Markdown
Contributor

Three related races between RebalanceService.ExecuteAsync, queued RebalanceJob retries, and MonitorRebalancesJob.

  • Probe kill (MonitorRebalancesJob): PaymentHashHex is persisted right after AddInvoice, but SendPaymentV2 doesn't dispatch until after the probe succeeds — so "no record in LND" is the
    expected state during the probe. The monitor was flipping such rows to Failed, killing in-progress probes. Fixed by only flipping Failed on null when Status == InFlight;
    Pending/Probing are left to local execution. Stale rows past the reconciliation window are still reaped.
  • Status oscillation (MonitorRebalancesJob): ScheduleRetryIfEligibleAsync leaves the prior attempt's hash on a queued-for-retry row. Mid-backoff, the monitor was tracking that stale
    hash and writing the prior failure back on top — producing NoRoute → Pending → NoRoute → Probing → … plus a spurious RebalanceCompleted/Failure audit event between every two attempts.
    Fixed by returning without writing when the row is Pending/Probing regardless of LND's response; the retry path owns those states.
  • Double-pay (RebalanceService.ExecuteAsync): with the monitor (correctly) staying back from queued retries, nothing was checking whether the prior attempt actually settled in LND
    despite a local Failed. The retry would call AddInvoice (new hash) and send again, paying fees twice. Each retry creates a fresh invoice, so Lightning's own invoice-uniqueness rule
    doesn't catch it. Fixed with a pre-flight TrackPaymentV2 at the top of ExecuteAsync: on Succeeded, adopt the outcome and return without sending.

@markettes markettes marked this pull request as draft June 19, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant