Investigate deployment of Valkey

[mutambaraf]

We need to investigate how we are going to deploy valkey to the new cluster being created.

[mutambaraf]

Multi-Cluster Valkey Sentinel Architecture - Diagram

System Overview: Three Clusters

    ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
    │                        MULTI-CLUSTER VALKEY WITH SENTINEL TOPOLOGY                                   │
    │                                                                                                      │
    │  Applications in Cluster Only Connect to Local Valkey | Replication Agent Syncs Between Clusters     │
    │                                                                                                      │
    └──────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌───────────────────────────────────┐  ┌───────────────────────────────────┐  ┌───────────────────────────────────┐
│       CLUSTER: PROXMOX            │  │       CLUSTER: CLOUD              │  │   CLUSTER: ISIS VMS               │
├───────────────────────────────────┤  ├───────────────────────────────────┤  ├───────────────────────────────────┤
│                                   │  │                                   │  │                                   │
│  LOCAL APPLICATIONS               │  │  LOCAL APPLICATIONS               │  │  LOCAL APPLICATIONS               │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │ App-1                       │  │  │  │ App-1                       │  │  │  │ App-1                       │  │
│  │ App-2 (connect locally)     │  │  │  │ App-2 (connect locally)     │  │  │  │ App-2 (connect locally)     │  │
│  │ ...                         │  │  │  │ ...                         │  │  │  │ ...                         │  │
│  └────────────┬────────────────┘  │  │  └────────────┬────────────────┘  │  │  └────────────┬────────────────┘  │
│               │                   │  │               │                   │  │               │                   │
│               ▼                   │  │               ▼                   │  │               ▼                   │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │  VALKEY CLUSTER             │  │  │  │  VALKEY CLUSTER             │  │  │  │  VALKEY CLUSTER             │  │
│  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  │  Master (port:6379)         │  │  │  │  Master (port:6379)         │  │  │  │  Master (port:6379)         │  │
│  │  ├─ Replica-1               │  │  │  │  ├─ Replica-1               │  │  │  │  ├─ Replica-1               │  │
│  │  ├─ Replica-2               │  │  │  │  ├─ Replica-2               │  │  │  │  ├─ Replica-2               │  │
│  │  └─ Replication Offset: X   │  │  │  │  └─ Replication Offset: Y   │  │  │  │  └─ Replication Offset: Z   │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │
│               │                   │  │               │                   │  │               │                   │
│               ▼                   │  │               ▼                   │  │               ▼                   │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │  SENTINEL CLUSTER           │  │  │  │  SENTINEL CLUSTER           │  │  │  │  SENTINEL CLUSTER           │  │
│  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │
│  │ Sentinel-1 (port:26379)     │  │  │  │ Sentinel-1 (port:26379)     │  │  │  │ Sentinel-1 (port:26379)     │  │
│  │ Sentinel-2 (Quorum: 2/3)    │  │  │  │ Sentinel-2 (Quorum: 2/3)    │  │  │  │ Sentinel-2 (Quorum: 2/3)    │  │
│  │ Sentinel-3 (Monitors Local) │  │  │  │ Sentinel-3 (Monitors Local) │  │  │  │ Sentinel-3 (Monitors Local) │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  │ Status: Master Up           │  │  │  │ Status: Master Up           │  │  │  │ Status: Master Up           │  │
│  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │
│               │                   │  │               │                   │  │               │                   │
│               ▼                   │  │               ▼                   │  │               ▼                   │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │  REPLICATION AGENT          │  │  │  │  REPLICATION AGENT          │  │  │  │  REPLICATION AGENT          │  │
│  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │
│  │ Reads from: Local Master    │  │  │  │ Reads from: Local Master    │  │  │  │ Reads from: Local Master    │  │
│  │ Writes to:                  │  │  │  │ Writes to:                  │  │  │  │ Writes to:                  │  │
│  │  ├─ Cloud Master            │  │  │  │  ├─ Proxmox Master          │  │  │  │  ├─ Proxmox Master          │  │
│  │  ├─ ISIS Master             │  │  │  │  ├─ ISIS Master             │  │  │  │  ├─ Cloud Master            │  │
│  │  └─ Lag: < 1 sec            │  │  │  │  └─ Lag: < 1 sec            │  │  │  │  └─ Lag: < 1 sec            │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │
│                                   │  │                                   │  │                                   │
│               ▼                   │  │               ▼                   │  │               ▼                   │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │  EXTERNAL LOADBALANCER      │  │  │  │  EXTERNAL LOADBALANCER      │  │  │  │  EXTERNAL LOADBALANCER      │  │
│  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │
│  │ Type: LoadBalancer          │  │  │  │ Type: LoadBalancer          │  │  │  │ Type: LoadBalancer          │  │
│  │ Fixed IP: 192.168.1.100     │  │  │  │ Fixed IP: 192.168.2.100     │  │  │  │ Fixed IP: 192.168.3.100     │  │
│  │ Port: 6379                  │  │  │  │ Port: 6379                  │  │  │  │ Port: 6379                  │  │
│  │ Routes to: Local Master     │  │  │  │ Routes to: Local Master     │  │  │  │ Routes to: Local Master     │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │
│                                   │  │                                   │  │                                   │
└───────────────────────────────────┘  └───────────────────────────────────┘  └───────────────────────────────────┘
         PROXMOX CLUSTER                      CLOUD CLUSTER                      ISIS VMS CLUSTER

                                                    ▲
                                                    │
                                                    │
                                    DNS Round Robin Resolution
                                    ┌─────────────────┬──────────────────┬─────────────────┐
                                    │                 │                  │                 │
                                    ▼                 ▼                  ▼                 ▼
                              valkey.example.com → [192.168.1.100 → 192.168.2.100 → 192.168.3.100]
                                    │                 │                  │
                              Return IP in Round Robin: 1st query → 192.168.1.100
                              2nd query → 192.168.2.100, 3rd query → 192.168.3.100

---

Data Flow Diagram

Write Operation Flow

┌────────────────────────────────────────────────────────────────────────────────────┐
│                          WRITE OPERATION ACROSS CLUSTERS                           │
└────────────────────────────────────────────────────────────────────────────────────┘

Application in Proxmox Cluster writes:
    SET key1 value1

    ▼

[Proxmox Master] ◄── Application connects locally only
    │
    ├─► [Proxmox Replica-1]  ◄── Synchronous replication (fast, local)
    │
    └─► [Proxmox Replica-2]

    ▼

[Proxmox Replication Agent] detects change
    │
    ├─► [Cloud Master] ◄── Async replication (eventual consistency)
    │
    └─► [ISIS Master]  ◄── Async replication (eventual consistency)

    ▼

[Cloud Replication Agent] receives update from Proxmox
    │
    ├─► [Proxmox Master] ◄── Updates already present, skip
    │
    └─► [ISIS Master]

    ▼

[ISIS Replication Agent] receives update from Proxmox/Cloud
    │
    ├─► [Proxmox Master] ◄── Updates already present, skip
    │
    └─► [Cloud Master]  ◄── Updates already present, skip

Result: Data eventually consistent across all three clusters (< 1 second typically)

Failover Scenario: Proxmox Master Fails

┌────────────────────────────────────────────────────────────────────────────────────┐
│                    LOCAL FAILOVER: PROXMOX MASTER FAILS                            │
└────────────────────────────────────────────────────────────────────────────────────┘

BEFORE: Proxmox Applications use Proxmox Master
    App → [Proxmox Master] ✓

DETECTION (T+0 to T+5s):
    Sentinel-1, Sentinel-2, Sentinel-3 ping Proxmox Master
    └─► No response → Mark as SDOWN (Subjective Down)
    └─► Reach consensus (2/3 agree) → Mark as ODOWN (Objective Down)

FAILOVER DECISION (T+5s to T+10s):
    Sentinels elect leader
    └─► Leader initiates failover
    └─► Promotes best replica to new master
    └─► Reorders other replicas

NEW STATE (T+10s):
    [Proxmox Replica-1] → Promoted to [Proxmox Master] ✓
    [Proxmox Replica-2] → Syncs from new master
    [Proxmox Master (old)] → Offline, rejoins as replica when back

AFTER: Proxmox Applications automatically reconnected
    Sentinel provides updated endpoints
    App → [Proxmox Master (ex-Replica-1)] ✓

RESULT: Failover complete, no data loss, no application changes needed

Cluster Failure Scenario: Proxmox Cluster Goes Offline

┌────────────────────────────────────────────────────────────────────────────────────┐
│                 CLUSTER OUTAGE: PROXMOX CLUSTER OFFLINE                            │
└────────────────────────────────────────────────────────────────────────────────────┘

T+0: Proxmox Cluster Becomes Unreachable
    ├─ Proxmox Master: Offline ✗
    ├─ Proxmox Replicas: Offline ✗
    ├─ Proxmox Sentinels: Offline ✗
    └─ Proxmox Applications: No local Valkey, handle with cache/fallback

T+1 to T+N: Traffic Routed to Available Clusters
    Cloud Applications → Cloud Master ✓
    ISIS Applications → ISIS Master ✓
    Proxmox Applications → Cloud Master OR ISIS Master (via DNS fallback)

    Replication Agent (Cloud) sends updates:
    ├─ To ISIS Master ✓
    └─ To Proxmox Master (retries fail, backoff)

    Replication Agent (ISIS) sends updates:
    ├─ To Cloud Master ✓
    └─ To Proxmox Master (retries fail, backoff)

    Data Divergence:
    └─ Cloud & ISIS: Fully in sync (replicating to each other)
    └─ Proxmox: Stale (offline during outage)

T+N: Proxmox Cluster Comes Back Online
    ├─ Proxmox Master: Online ✓ (with stale data)
    ├─ Proxmox Replicas: Online ✓
    ├─ Proxmox Sentinels: Online ✓
    └─ Replication Agent: Online ✓

T+N+1 to T+N+3: Recovery Process
    Step 1: Download RDB snapshot from Cloud Master
    Step 2: Load RDB into Proxmox Master
    Step 3: Proxmox Replicas sync from new master
    Step 4: Replication Agent catches up Streams

T+N+4: Proxmox Fully Recovered
    ├─ Proxmox Master: Synced with Cloud & ISIS ✓
    ├─ Proxmox Applications: Can reconnect ✓
    └─ All three clusters: Consistent again ✓

DATA GUARANTEE: No write loss - all writes during Proxmox outage preserved in Cloud & ISIS

Cluster Recovery with RDB Snapshot

┌────────────────────────────────────────────────────────────────────────────────────┐
│                       RDB SNAPSHOT RECOVERY PROCESS                                │
└────────────────────────────────────────────────────────────────────────────────────┘

Prerequisite: Proxmox cluster is online but has stale data

Step 1: Trigger RDB Export on Cloud Master
    redis-cli BGSAVE
    └─► Creates background snapshot (non-blocking)
    └─► RDB file size: ~2GB (depends on data)
    └─► Time to complete: ~30 seconds to 2 minutes

Step 2: Download RDB to Proxmox
    scp cloud-master:/var/lib/redis/dump.rdb proxmox:/tmp/

Step 3: Stop Proxmox Master
    kubectl delete pod valkey-proxmox-0 -n valkey

Step 4: Copy RDB to PersistentVolume
    kubectl cp /tmp/dump.rdb valkey-proxmox-0:/data/dump.rdb

Step 5: Start Proxmox Master
    kubectl rollout status sts/valkey-proxmox
    └─► Master starts with recovered data

Step 6: Proxmox Replicas Sync
    Replica-1 connects to new master
    └─► SYNC from new master (fast, recent snapshot)
    └─► Replica-2 connects to new master

Step 7: Replication Agent Sync Streams
    Agent reads Streams from Cloud/ISIS
    └─► Catches up any Streams messages during recovery
    └─► Ensures no gaps in replication

RESULT: Proxmox has full consistent dataset, rejoins mesh
Total Recovery Time: ~5-10 minutes (depends on data size)

---

Network Connectivity Requirements

┌────────────────────────────────────────────────────────────────────────────────────┐
│                       NETWORK CONNECTIVITY MATRIX                                  │
└────────────────────────────────────────────────────────────────────────────────────┘

FROM → TO                  PORT    DIRECTION   PROTOCOL    LATENCY    NOTES
─────────────────────────────────────────────────────────────────────
Proxmox Apps → Proxmox Valkey
                          6379    Bi-directional  TCP     < 5ms      Local only
                          
Proxmox Valkey → Proxmox Sentinel
                          26379   Bi-directional  TCP     < 5ms      Local monitoring

Proxmox Sentinel → Cloud Sentinel
                          26379   Outbound       TCP     < 50ms     Cross-cluster metadata

Proxmox Replication Agent → Cloud Master
                          6379    Outbound       TCP     < 50ms     Async replication writes

Proxmox Replication Agent → ISIS Master
                          6379    Outbound       TCP     < 50ms     Async replication writes

Proxmox Master → Cloud Master
                          6379    Bidirectional  TCP     < 50ms     RDB snapshot + streams

External DNS → All LoadBalancers
                          53      Query          UDP/TCP  < 100ms    Round-robin resolution

External Client → Proxmox LoadBalancer
                          6379    Inbound        TCP     Variable   Only if client needs direct access

SUMMARY:
├─ Intra-cluster latency (same Kubernetes cluster): < 5ms
├─ Inter-cluster latency (VPN/direct): < 50ms recommended
├─ All ports are TCP except DNS (UDP)
├─ Each cluster is network-isolated from external applications
└─ Only Replication Agents and Sentinels communicate across clusters

---

Key Design Principles

1. Local-First Applications

• Applications always connect to Valkey in their own cluster
• Uses local DNS: valkey.valkey.svc.cluster.local
• No cross-cluster latency for app reads/writes
• Reduces network dependency

2. Asynchronous Replication

• Replication Agent continuously syncs between clusters
• Uses Redis Streams for reliable, durable changelog
• Eventual consistency model (data converges within seconds)
• Tolerates network partitions gracefully

3. Independent Failover

• Each cluster has local Sentinel quorum (3 nodes)
• Failover decisions made locally without quorum across clusters
• No split-brain scenarios possible
• Avoids complex distributed consensus

4. Fixed IPs & DNS

• Each cluster has fixed LoadBalancer IP
• DNS round-robin points to all three fixed IPs
• External clients (if needed) use DNS, get random cluster
• Cluster outage doesn't break DNS

5. RDB Snapshot Recovery

• Failed cluster recovers by downloading snapshot
• Guarantees consistency with peer clusters
• Simple, non-destructive process
• No need for complex incremental sync

[simonfernandes]

@mutambaraf We had to remove the above comment to hide sensitive info like IP addresses etc, feel free to repost without

[mutambaraf]

@mutambaraf We had to remove the above comment to hide sensitive info like IP addresses etc, feel free to repost without

There are not the actual IP Address just examples from my local set up.

[mutambaraf]

Multi-Location Unified Valkey Cluster Architecture - Diagram

System Overview: Single Cluster Spanning 3 Locations

    ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
    │                        UNIFIED VALKEY CLUSTER ACROSS 3 LOCATIONS                                     │
    │                                                                                                      │
    │     Single Logical Cluster with 6 Nodes Distributed Across 3 Physical Locations                     │
    │     All Nodes Communicate via Cluster Gossip Protocol (Ports 6379 + 16379)                          │
    │                                                                                                      │
    └──────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌───────────────────────────────────┐  ┌───────────────────────────────────┐  ┌───────────────────────────────────┐
│       LOCATION: PROXMOX           │  │       LOCATION: CLOUD             │  │   LOCATION: ISIS VMS              │
├───────────────────────────────────┤  ├───────────────────────────────────┤  ├───────────────────────────────────┤
│                                   │  │                                   │  │                                   │
│  LOCAL APPLICATIONS               │  │  LOCAL APPLICATIONS               │  │  LOCAL APPLICATIONS               │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │ App-1                       │  │  │  │ App-1                       │  │  │  │ App-1                       │  │
│  │ App-2 (connect to any node) │  │  │  │ App-2 (connect to any node) │  │  │  │ App-2 (connect to any node) │  │
│  │ ...                         │  │  │  │ ...                         │  │  │  │ ...                         │  │
│  └────────────┬────────────────┘  │  │  └────────────┬────────────────┘  │  │  └────────────┬────────────────┘  │
│               │                   │  │               │                   │  │               │                   │
│               ▼                   │  │               ▼                   │  │               ▼                   │
│  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │  │  ┌─────────────────────────────┐  │
│  │  VALKEY CLUSTER NODE        │  │  │  │  VALKEY CLUSTER NODE        │  │  │  │  VALKEY CLUSTER NODE        │  │
│  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │  │  ├─────────────────────────────┤  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  │ Node-1: PRIMARY             │  │  │  │ Node-3: PRIMARY             │  │  │  │ Node-5: PRIMARY             │  │
│  │ Slots: 0-5460               │  │  │  │ Slots: 5461-10922           │  │  │  │ Slots: 10923-16383          │  │
│  │ Port: 6379 (commands)       │  │  │  │ Port: 6379 (commands)       │  │  │  │ Port: 6379 (commands)       │  │
│  │ Port: 16379 (cluster bus)   │  │  │  │ Port: 16379 (cluster bus)   │  │  │  │ Port: 16379 (cluster bus)   │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  │ ▼ Replica of Node-3 ◄─────┐ │  │  │  │ ▼ Replica of Node-5 ◄─────┐ │  │  │  │ ▼ Replica of Node-1 ◄─────┐ │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  │ Node-2: REPLICA             │  │  │  │ Node-4: REPLICA             │  │  │  │ Node-6: REPLICA             │  │
│  │ (Replicates from Node-3)    │  │  │  │ (Replicates from Node-5)    │  │  │  │ (Replicates from Node-1)    │  │
│  │ Port: 6379                  │  │  │  │ Port: 6379                  │  │  │  │ Port: 6379                  │  │
│  │ Port: 16379                 │  │  │  │ Port: 16379                 │  │  │  │ Port: 16379                 │  │
│  │                             │  │  │  │                             │  │  │  │                             │  │
│  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │  │  └─────────────────────────────┘  │
│               ▲                   │  │                  ▲                │  │                 ▲                 │
└───────────────┼────────────────  ─┘  └──────────────────┼────────────────┘  └─────────────────┼─────────────── ─┘
                │                                       │                                       │
                │  Gossip Protocol                      │  Gossip Protocol                      │  Gossip Protocol
                │  Port 16379                           │  Port 16379                           │  Port 16379
                │  (Cluster State Sync)                 │  (Cluster State Sync)                 │  (Cluster State Sync)
                │                                       │                                       │
                └───────────────────────────┬───────────┴───────────────┬───────────────────────┘
                                            │                           │
                                      All 6 nodes aware of each other
                                      Gossip traffic: < 50ms latency
                                      Cluster is UNIFIED

                                            ▲
                                            │
                              ┌─────────────┴──────────────┐
                              │                            │
                    ┌─────────▼──────────┐    ┌──────────▼─────────┐
                    │   VALKEY ADMIN     │    │  HELM DEPLOYMENT   │
                    ├────────────────────┤    ├────────────────────┤
                    │ Web UI: Port 5664  │    │  valkey-helm repo  │
                    │ Monitors all 6     │    │  with cluster mode │
                    │ nodes (real-time)  │    │  enabled: true     │
                    │                    │    │                    │
                    │ Shows:             │    │  Deployed via:     │
                    │ ├─ Slot layout     │    │  helm upgrade      │
                    │ ├─ Node health     │    │  --values values.  │
                    │ ├─ Replication lag │    │  yaml              │
                    │ └─ Cluster state   │    │                    │
                    └────────────────────┘    └────────────────────┘

[mutambaraf]

Multi-Location Valkey Architecture: Separate Clusters with Inter-Cluster Replication

System Overview: 3 Independent Clusters with Custom Replication Agent

    ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
    │               3 INDEPENDENT VALKEY CLUSTERS ACROSS 3 LOCATIONS                                       │
    │                                                                                                      │
    │     Each Location: Standalone Cluster with Cluster Mode + Internal Failover                          │
    │     Inter-Cluster Sync: Custom Replication Agent (handles data propagation)                          │
    │     Recovery: Cluster comeback triggers auto-sync from either of 2 peer clusters                     │
    │                                                                                                      │
    └──────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────┐  ┌─────────────────────────────────────────┐  ┌─────────────────────────────────────────┐
│       LOCATION: PROXMOX                 │  │       LOCATION: CLOUD                   │  │   LOCATION: ISIS VMS                    │
│       (Cluster #1)                      │  │       (Cluster #2)                      │  │   (Cluster #3)                          │
├─────────────────────────────────────────┤  ├─────────────────────────────────────────┤  ├─────────────────────────────────────────┤
│                                         │  │                                         │  │                                         │
│  LOCAL APPLICATIONS                     │  │  LOCAL APPLICATIONS                     │  │  LOCAL APPLICATIONS                     │
│  ┌─────────────────────────────────┐    │  │  ┌─────────────────────────────────┐    │  │  ┌─────────────────────────────────┐   │
│  │ App-1                           │    │  │  │ App-1                           │    │  │  │ App-1                           │   │
│  │ App-2 (LOCAL READS & WRITES)    │    │  │  │ App-2 (LOCAL READS & WRITES)    │    │  │  │ App-2 (LOCAL READS & WRITES)    │   │
│  │ ...                             │    │  │  │ ...                             │    │  │  │ ...                             │   │
│  └────────────────┬────────────────┘    │  │  └────────────┬────────────────────┘    │  │  └────────────┬────────────────────┘   │
│                   │                     │  │               │                         │  │               │                        │
│                   ▼                     │  │               ▼                         │  │               ▼                        │
│  ┌─────────────────────────────────┐    │  │  ┌─────────────────────────────────┐    │  │  ┌─────────────────────────────────┐   │
│  │   VALKEY CLUSTER (CLUSTER MODE) │    │  │  │   VALKEY CLUSTER (CLUSTER MODE) │    │  │  │   VALKEY CLUSTER (CLUSTER MODE) │   │
│  ├─────────────────────────────────┤    │  │  ├─────────────────────────────────┤    │  │  ├─────────────────────────────────┤   │
│  │                                 │    │  │  │                                 │    │  │  │                                 │   │
│  │ Node-1a: PRIMARY (5461 slots)   │    │  │  │ Node-2a: PRIMARY (5461 slots)   │    │  │  │ Node-3a: PRIMARY (5461 slots)   │   │
│  │ Slots: 0-5460                   │    │  │  │ Slots: 0-5460                   │    │  │  │ Slots: 0-5460                   │   │
│  │ Port: 6379 (commands)           │    │  │  │ Port: 6379 (commands)           │    │  │  │ Port: 6379 (commands)           │   │
│  │ Port: 16379 (cluster bus)       │    │  │  │ Port: 16379 (cluster bus)       │    │  │  │ Port: 16379 (cluster bus)       │   │
│  │                                 │    │  │  │                                 │    │  │  │                                 │   │
│  │ ▼ Replica of Node-1a            │    │  │  │ ▼ Replica of Node-2a            │    │  │  │ ▼ Replica of Node-3a            │   │
│  │                                 │    │  │  │                                 │    │  │  │                                 │   │
│  │ Node-1b: REPLICA                │    │  │  │ Node-2b: REPLICA                │    │  │  │ Node-3b: REPLICA                │   │
│  │ (Replicates Node-1a, read-only) │    │  │  │ (Replicates Node-2a, read-only) │    │  │  │ (Replicates Node-3a, read-only) │   │
│  │ Port: 6379                      │    │  │  │ Port: 6379                      │    │  │  │ Port: 6379                      │   │
│  │ Port: 16379                     │    │  │  │ Port: 16379                     │    │  │  │ Port: 16379                     │   │
│  │                                 │    │  │  │                                 │    │  │  │                                 │   │
│  └─────────────────────────────────┘    │  │  └─────────────────────────────────┘    │  │  └─────────────────────────────────┘   │
│                 ▲                       │  │                  ▲                      │  │                 ▲                      │
│                 │                       │  │                  │                      │  │                 │                      │
│        INTERNAL CLUSTER GOSSIP          │  │       INTERNAL CLUSTER GOSSIP           │  │       INTERNAL CLUSTER GOSSIP          │
│        (6379, 16379 local only)         │  │       (6379, 16379 local only)          │  │       (6379, 16379 local only)         │
│                 │                       │  │                  │                      │  │                 │                      │
└─────────────────┼───────────────────────┘  └──────────────────┼──────────────────────┘  └─────────────────┼──────────────────────┘
                  │                                             │                                           │
                  └──────────────────────┬──────────────────────┴────────────────────────┬───────────────── ┘
                                         │                                               │
                                   INTER-CLUSTER REPLICATION AGENT
                                   ┌─────────────────────────────────────────┐
                                   │                                         │
                                   │  CUSTOM REPLICATION AGENT               │
                                   ├─────────────────────────────────────────┤
                                   │                                         │
                                   │  Responsibilities:                      │
                                   │  • Subscribe to write events (Proxmox)  │
                                   │  • Forward keys to Cloud & ISIS         │
                                   │  • Subscribe to write events (Cloud)    │
                                   │  • Forward keys to Proxmox & ISIS       │
                                   │  • Subscribe to write events (ISIS)     │
                                   │  • Forward keys to Proxmox & Cloud      │
                                   │                                         │
                                   │  Cluster Recovery Mode:                 │
                                   │  • Detect cluster restart/comeback      │
                                   │  • Trigger FULL SYNC from peer cluster  │
                                   │  • Verify data consistency              │
                                   │  • Resume incremental sync              │
                                   │                                         │
                                   └─────────────────────────────────────────┘
                                                       ▲
                                                       │
                                     ┌─────────────────┴──────────────────┐
                                     │                                    │
                           ┌─────────▼──────────┐        ┌──────────▼─────────┐
                           │   VALKEY ADMIN     │        │  HELM DEPLOYMENT   │
                           ├────────────────────┤        ├────────────────────┤
                           │ Web UI: Port 5664  │        │  valkey-helm repo  │
                           │ (Multi-cluster)    │        │  cluster mode:     │
                           │ Monitors all 6     │        │  enabled: true     │
                           │ nodes across 3     │        │                    │
                           │ clusters           │        │  Deployed x3:      │
                           │                    │        │  (one per location) │
                           │ Shows:             │        │  via helm upgrade   │
                           │ ├─ Per-cluster     │        │  --values values.   │
                           │ │  health          │        │  yaml              │
                           │ ├─ Inter-cluster   │        │                    │
                           │ │  sync status     │        │                    │
                           │ ├─ Node replication│        │                    │
                           │ │  lag             │        │                    │
                           │ └─ Recovery status │        │                    │
                           └────────────────────┘        └────────────────────┘

---

Cluster Topology: Independent Clusters with Internal Replication

Cluster Architecture (Each Location)

┌────────────────────────────────────────────────────────────────────┐
│              EACH LOCATION: STANDALONE CLUSTER TOPOLOGY             │
└────────────────────────────────────────────────────────────────────┘

PROXMOX CLUSTER (Cluster #1)
  Primary: Node-1a
    Slots: 0 - 5460 (all slots in this cluster - no sharding between clusters)
    Replicated by: Node-1b
    
  Replica: Node-1b
    Replicates: Node-1a (read-only)
    
  Cluster Mode: ENABLED
    └─► Internal failover: If Node-1a fails → Node-1b promoted to primary
    └─► Gossip protocol: Nodes 1a ↔ 1b on port 16379


CLOUD CLUSTER (Cluster #2)
  Primary: Node-2a
    Slots: 0 - 5460 (same slots as Cluster #1 - independent copy)
    Replicated by: Node-2b
    
  Replica: Node-2b
    Replicates: Node-2a (read-only)
    
  Cluster Mode: ENABLED
    └─► Internal failover: If Node-2a fails → Node-2b promoted to primary
    └─► Gossip protocol: Nodes 2a ↔ 2b on port 16379


ISIS CLUSTER (Cluster #3)
  Primary: Node-3a
    Slots: 0 - 5460 (same slots as Clusters #1 & #2 - independent copy)
    Replicated by: Node-3b
    
  Replica: Node-3b
    Replicates: Node-3a (read-only)
    
  Cluster Mode: ENABLED
    └─► Internal failover: If Node-3a fails → Node-3b promoted to primary
    └─► Gossip protocol: Nodes 3a ↔ 3b on port 16379


KEY DIFFERENCES FROM UNIFIED CLUSTER:
  • Each cluster owns ALL slots (0-5460), not split
  • No cross-cluster gossip (clusters are independent)
  • Each cluster has built-in failover for its own nodes
  • Inter-cluster sync via CUSTOM AGENT (not automatic)

---

Inter-Cluster Replication Agent

Architecture

┌────────────────────────────────────────────────────────────────────────────────┐
│                   INTER-CLUSTER REPLICATION AGENT                              │
│                       (Runs in dedicated container)                             │
└────────────────────────────────────────────────────────────────────────────────┘

AGENT RESPONSIBILITIES:

1. NORMAL OPERATION (Incremental Sync)
   ────────────────────────────────────
   
   [Proxmox Cluster] 
         │
         │ Keyspace notifications / SUBSCRIBE
         │ (watch for write events)
         ▼
   ┌─────────────────────────────────────┐
   │ Replication Agent Instance          │
   │ (deployed in Proxmox)               │
   ├─────────────────────────────────────┤
   │ • Listen to Proxmox writes          │
   │ • Forward to Cloud (Node-2a)        │
   │ • Forward to ISIS (Node-3a)         │
   │ • Track replication lag             │
   │ • Log sync events                   │
   └─────────────────────────────────────┘
         │
         ├──────────────────┬──────────────────┐
         │                  │                  │
         ▼                  ▼                  ▼
    [Cloud Cluster]    [ISIS Cluster]   [Proxmox Replica]
    (via REPLICATE/    (via REPLICATE/   (local repl)
     GETEX commands)    GETEX commands)


2. CLUSTER RECOVERY MODE (Full Sync)
   ────────────────────────────────────
   
   SCENARIO: Proxmox Cluster goes offline, comes back online
   
   T+0: Agent detects Proxmox cluster is BACK ONLINE
        └─► Check: is Proxmox cluster fully synced with Cloud/ISIS?
            └─► NO? → Enter RECOVERY MODE
   
   T+1: Recovery Mode Triggered
        ┌─────────────────────────────────────────────────────┐
        │ FULL SYNC PROCESS                                   │
        ├─────────────────────────────────────────────────────┤
        │                                                     │
        │ 1. Pause normal replication (Proxmox writes)        │
        │    └─► Read-only mode temporarily                   │
        │                                                     │
        │ 2. SCAN all keys in Cloud (SCAN 0 COUNT 100)        │
        │    └─► Get all key names in Cloud cluster           │
        │                                                     │
        │ 3. For each key in Cloud:                           │
        │    ├─► DUMP key from Cloud                          │
        │    ├─► RESTORE key to Proxmox                       │
        │    └─► Compare TTL (respect original expiry)        │
        │                                                     │
        │ 4. Verify data consistency:                         │
        │    ├─► DBSIZE check (Proxmox vs Cloud)              │
        │    ├─► Random key sampling (checksums)              │
        │    └─► Log mismatches (if any)                      │
        │                                                     │
        │ 5. Resume normal mode                               │
        │    └─► Resume event-driven replication              │
        │                                                     │
        └─────────────────────────────────────────────────────┘
   
   T+N: Proxmox cluster fully synced and online
        └─► All 3 clusters have identical data
        └─► Resume normal incremental sync


3. RING REPLICATION PATTERN (High Availability)
   ──────────────────────────────────────────────
   
   Proxmox → Cloud → ISIS → Proxmox (ring topology)
   
   Agent Instance 1 (Proxmox):
     └─► Watches Proxmox writes → sends to Cloud
   
   Agent Instance 2 (Cloud):
     └─► Watches Cloud writes → sends to ISIS
   
   Agent Instance 3 (ISIS):
     └─► Watches ISIS writes → sends to Proxmox
   
   BENEFIT: If one agent dies, ring continues (partial)
            Restart agent to resume full sync


4. HANDLING WRITE CONFLICTS
   ─────────────────────────
   
   SCENARIO: Both Proxmox and Cloud receive writes to same key
   
   Solution: LAST-WRITE-WINS (LWW) with timestamps
   
   ├─► Agent records: timestamp + source cluster for every key
   ├─► On conflict: Keep version with highest timestamp
   ├─► Fallback: Keep version from PRIMARY cluster
   └─► Log conflicts for monitoring/debugging

---

Data Flow Scenarios

Scenario 1: Normal Write Operation

┌────────────────────────────────────────────────────────────────────────────────┐
│                    WRITE OPERATION: LOCAL CLUSTER                              │
└────────────────────────────────────────────────────────────────────────────────┘

Application in Proxmox writes: SET key1 value1

STEP 1: App connects to local cluster (Node-1a or Node-1b)
    [Proxmox App] → SET key1 value1 → [Node-1a] (Proxmox Primary)

STEP 2: Node-1a processes write locally
    Write key1=value1 to local storage
    Replicate to Node-1b (synchronous)
    
STEP 3: Replication Agent detects write event
    [Keyspace notification triggered]
    ├─► Replication Agent Instance (Proxmox) wakes up
    ├─► Reads key1 from Node-1a
    └─► Extracts TTL + value

STEP 4: Inter-cluster replication
    Forward to Cloud:  SET key1 value1 (with TTL)
    Forward to ISIS:   SET key1 value1 (with TTL)
    
    [Proxmox Node-1a] ────┬────► [Cloud Node-2a] ✓ ACK
                          │
                          └────► [ISIS Node-3a] ✓ ACK

STEP 5: Cloud and ISIS replicate internally
    [Cloud Node-2a] ──► [Cloud Node-2b] (local replica)
    [ISIS Node-3a] ──► [ISIS Node-3b] (local replica)

RESULT:
    ✓ All 3 clusters have key1=value1
    ✓ All 6 nodes are synchronized
    ✓ Replication lag: < 100ms (typically < 10ms)
    ✓ Read available from ANY node in any cluster (eventual consistency)

Scenario 2: Node Failure (Internal Cluster Failover)

┌────────────────────────────────────────────────────────────────────────────────┐
│               INTERNAL FAILOVER: NODE FAILURE IN CLUSTER                       │
└────────────────────────────────────────────────────────────────────────────────┘

SCENARIO: Node-2a (Cloud Primary) goes down

BEFORE:
    Cloud Cluster:
    ├─ Node-2a (PRIMARY) [ALIVE]
    └─ Node-2b (REPLICA) [watches Node-2a]

T+0: Node-2a becomes unreachable
     
T+1-3s: Node-2b detects Node-2a failure
     └─► PFAIL (Possible Failure) state

T+3-6s: Cluster consensus (gossip between 2a and 2b)
     └─► Node-2b verifies Node-2a is truly dead
     └─► Cluster state: FAIL

T+6-10s: Failover triggered
     └─► Node-2b (REPLICA) → CANDIDATE for promotion
     └─► Node-2b requests votes (from itself - 2-node cluster)
     └─► Node-2b elected → PROMOTED to PRIMARY

NEW STATE (T+10s):
    Cloud Cluster:
    ├─ Node-2b (PRIMARY) [PROMOTED] ← now primary
    └─ Node-2a (REPLICA) [DEMOTED] ← will rejoin as replica

REPLICATION AGENT:
    • Continues to forward writes to Cloud Node-2b (new primary)
    • Updates monitoring (shows Node-2b as new primary)

RECOVERY (T+N: Node-2a comes back online):
    ├─ Node-2a rejoins cluster
    ├─ Discovers Node-2b is primary
    ├─ Automatically becomes replica of Node-2b
    ├─ Agent triggers recovery sync (DUMP/RESTORE)
    └─ Full consistency restored

RESULT: Zero downtime, automatic failover, data preserved

Scenario 3: Entire Cluster Comes Back Online (Recovery Sync)

┌────────────────────────────────────────────────────────────────────────────────┐
│          CLUSTER RECOVERY: FULL SYNC AFTER EXTENDED OFFLINE PERIOD             │
└────────────────────────────────────────────────────────────────────────────────┘

SCENARIO: Proxmox cluster offline for 30 minutes, comes back online

T-30min: Proxmox cluster goes offline
     ├─ Node-1a [OFFLINE]
     ├─ Node-1b [OFFLINE]
     └─ Replication Agent (Proxmox) [OFFLINE]
     
     During outage:
     ├─ Cloud cluster receives writes (10 GB new data)
     ├─ ISIS cluster receives writes (10 GB new data)
     └─ Proxmox cluster has stale data (out of sync)

T+0: Proxmox cluster comes back online
     ├─ Node-1a [ONLINE]
     ├─ Node-1b [ONLINE]
     └─ Replication Agent detects PROXMOX IS BACK

T+1: Agent enters RECOVERY MODE
     
     STEP 1: Lock Proxmox for writes (read-only mode)
             └─► Prevents new writes during sync
             
     STEP 2: Full data sync from Cloud cluster
             
             ┌──────────────────────────────┐
             │ SYNC ALGORITHM:              │
             ├──────────────────────────────┤
             │                              │
             │ 1. SCAN Cloud (Node-2a)      │
             │    ├─ Get all key names      │
             │    ├─ Partition into 100-key │
             │    │  batches                │
             │    └─ Total keys: ~50 million│
             │                              │
             │ 2. For each batch:           │
             │    ├─ DUMP keys from Cloud   │
             │    ├─ RESTORE to Proxmox     │
             │    ├─ Skip if exists & same  │
             │    │  (optimization)         │
             │    └─ Update Proxmox count   │
             │                              │
             │ 3. Verify after sync:        │
             │    ├─ DBSIZE Proxmox:        │
             │    │  50,000,000 keys        │
             │    ├─ DBSIZE Cloud:          │
             │    │  50,000,000 keys        │
             │    ├─ Match? ✓ Continue      │
             │    └─ Mismatch? Re-sync      │
             │                              │
             │ 4. Random sample check:      │
             │    ├─ Pick 1000 random keys  │
             │    ├─ GET from Proxmox       │
             │    ├─ GET from Cloud         │
             │    ├─ Compare values         │
             │    └─ Detect hash mismatches │
             │                              │
             └──────────────────────────────┘
             
             Time estimate: 
             └─► ~30 GB data = 30-60 minutes
                 (depends on network speed)
     
     STEP 3: Resume normal replication
             └─► Proxmox goes ONLINE (read-write)
             └─► Agent resumes event-driven sync
             
     STEP 4: Monitor consistency
             ├─ Continue sampling for 24 hours
             ├─ Alert if mismatches found
             └─ Auto-heal if agent detects drift

T+60min: Proxmox fully synced and online
     └─► All 3 clusters have identical data (50M keys)
     └─► Proxmox apps can resume normal operation


AGENT BEHAVIOR DURING RECOVERY:

Agent monitors:
  ├─ Is Proxmox cluster reachable? [Node-1a heartbeat]
  ├─ Are both Node-1a and Node-1b healthy?
  ├─ Is replication lag > threshold (e.g., 5 seconds)?
  └─ Are there unsynced keys detected by sampling?

If recovery needed:
  ├─ Pause write propagation to Proxmox
  ├─ Dump all keys from Cloud (SCAN + DUMP)
  ├─ Restore to Proxmox (RESTORE + EXPIRE)
  ├─ Verify DBSIZE matches
  └─ Resume normal sync

Scenario 4: Network Partition

┌────────────────────────────────────────────────────────────────────────────────┐
│                      NETWORK PARTITION: CLUSTER ISOLATION                      │
└────────────────────────────────────────────────────────────────────────────────┘

SCENARIO: Cloud cluster loses network connectivity

T+0: Cloud cluster becomes unreachable

T+1-5s: Agent detects Cloud is down
     ├─ Cannot connect to Node-2a (timeout)
     ├─ Cannot connect to Node-2b (timeout)
     └─ Alert: "Cloud cluster OFFLINE"

BEHAVIOR:

Proxmox Cluster:
  ├─ Continues operating normally
  └─ Agent stops forwarding writes to Cloud
     (waits for recovery)

ISIS Cluster:
  ├─ Continues operating normally
  └─ Agent continues receiving writes from Proxmox
     (ISIS still synced via Proxmox→ISIS path)

Cloud Cluster (Internal):
  ├─ Node-2a and Node-2b still replicate each other
  ├─ Cluster is self-healing (if it's an internal issue)
  └─ New writes to Cloud are local-only (not replicated out)


RECOVERY (Cloud comes back online):

T+N: Cloud cluster is reachable again

Agent action:
  ├─ Detect Cloud is back
  ├─ Assess: Is Cloud in sync?
  │   ├─ Sample 100 random keys from Proxmox & Cloud
  │   ├─ If match: Continue normal sync
  │   └─ If mismatch: Trigger recovery sync
  ├─ If recovery needed: DUMP/RESTORE from Proxmox
  └─ Resume normal replication ring


RESULT: Data loss prevented (Proxmox + ISIS preserved Cloud data)
        Cloud catches up via recovery sync