DevOps: Add contract and backend monitoring with alerting

[CelestinaBeing]

Summary

There is no monitoring or alerting configured for the Trivela system on mainnet. Contract pause events, backend downtime, Soroban RPC degradation, or abnormal error rates will go undetected until users report them. For a mainnet platform, proactive monitoring is non-negotiable.

Problem

• Backend /health and /metrics endpoints exist but nothing consumes them with alerting rules
• No alerting on: contract paused event, backend 5xx spike, RPC health check failure, campaign DB write errors
• No uptime monitoring (e.g. UptimeRobot, BetterStack) configured
• No runbook for common failure scenarios

Acceptance Criteria

Prometheus/Grafana (self-hosted option)

• Add prometheus.yml scrape config targeting the backend /metrics endpoint
• Add alerting_rules.yml with alerts for:
  • Backend error rate > 5% over 5 min
  • RPC health status degraded for > 2 min
  • Process uptime reset (restart detected)
• Add Grafana dashboard JSON (monitoring/dashboards/trivela.json) with: request rate, error rate, uptime, route breakdown
• Add monitoring/ directory with compose override (compose.monitoring.yml) for local Prometheus + Grafana

Soroban Event Monitoring

• Add an alert when contract paused event is indexed (from issue Backend: Implement Soroban event indexer worker #283 indexer)
• Add an alert when RPC returns consecutive errors for > 60s

Runbook

• Add docs/RUNBOOK.md with procedures for: backend restart, RPC failover, contract pause response, DB backup restore

References

• backend/src/index.js — /metrics endpoint (Prometheus format)
• compose.yaml
• docs/ARCHITECTURE_OVERVIEW.md

[andrianvaleanu]

Mainnet needs more than just basic checks. This resource https://pulsetic.com/ helps wiht downtime and status alerts.

[Jumongweb]

@Jumongweb has applied to work on this issue as part of the Stellar Wave Program's 5th wave.

Hi maintainer, i will like to solve this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Jumongweb to this issue.

[SheyeJDev]

@SheyeJDev has applied to work on this issue as part of the Stellar Wave Program's 5th wave.

Hi maintainer,

Can I work on this issue?

ℹ️ Repo Maintainers: To accept this application, review their application or assign @SheyeJDev to this issue.

[drips-wave]

Congratulations, @SheyeJDev! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 2, 2026.

🧑‍💻 @SheyeJDev: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[drips-wave]

This issue has been completed by @SheyeJDev as part of the Stellar Wave Program's 5th Wave 🥳

😎 @SheyeJDev: You earned 200 Points for completing this issue! After the current Wave ends, you'll be eligible for a percentage of the Wave's reward pool based on the percentage of total points you've earned. Learn more here. You can also Leave a review to share your experience working on this issue.

🧑‍💻 Repo maintainers: How'd the contributor do? Leave a review to share your experience working with them.