feat(deployment): Implement zero-downtime blue-green deployment

[larryro]

Summary

Implement zero-downtime deployments for the Tale platform using blue-green deployment strategy with Docker Compose on a single VPS/VM.

Current Pain Points

1. Service restarts cause brief outages (~90 seconds downtime)
2. Database migrations block traffic
3. Long container startup times before traffic resumes

Proposed Solution

Use blue-green deployment with Caddy upstream pools. Two versions of stateless services run simultaneously during deployment, with traffic switched only after the new version is healthy.

How It Works

PHASE 1: Blue serving     → Internet → Caddy → [Blue ✅] 
PHASE 2: Green starting   → Internet → Caddy → [Blue ✅] + [Green ⏳]
PHASE 3: Both healthy     → Internet → Caddy → [Blue draining] + [Green ✅ NEW]
PHASE 4: Cleanup          → Internet → Caddy → [Green ✅]

Colors rotate: blue → green → blue → green...

Implementation Tasks

Files to Create

• scripts/deploy.sh - Deployment orchestration script

  • Detect current color (blue/green)
  • Build and start new color
  • Wait for health checks
  • Switch traffic via Caddy reload
  • Drain and cleanup old containers

• compose.blue.yml - Blue deployment overlay

  • Container names with -blue suffix
  • Network aliases: platform-blue, rag-blue, etc.

• compose.green.yml - Green deployment overlay

  • Container names with -green suffix
  • Network aliases: platform-green, rag-green, etc.

Files to Modify

• compose.yml

  • Remove container_name from stateless services (platform, rag, crawler, search, graph-db)
  • Keep container_name for stateful services (db, proxy)
  • Optimize health checks (5s interval instead of 30s)

• services/proxy/Caddyfile

  • Add upstream pool with both blue and green backends
  • Configure health-based routing (lb_policy first)
  • Add passive health checking (circuit breaker)

• services/platform/docker-entrypoint.sh

  • Add graceful shutdown with connection draining
  • Create shutdown marker file for health check

Database Consistency

Both Blue and Green share the same database. For schema changes, use the expand-contract pattern:

1. EXPAND: Add new columns/tables (backward compatible) - run BEFORE deployment
2. DEPLOY: Code works with both old and new schema
3. CONTRACT: Remove old columns (only AFTER old code is gone)

Safe Migrations (EXPAND)

• ADD COLUMN
• ADD TABLE
• CREATE INDEX CONCURRENTLY
• ADD NULLABLE COLUMN

Unsafe Migrations (CONTRACT - run manually after deploy)

• DROP COLUMN
• RENAME COLUMN
• ADD NOT NULL

Resource Requirements

Running two versions simultaneously requires ~2x memory during deployment:

Phase	Memory Required
Normal operation	~5-6 GB
During deployment	~10-12 GB

Recommendation: VPS should have at least 12-16 GB RAM.

Deployment Commands

# Normal deployment
./scripts/deploy.sh deploy

# Quick rollback (if previous containers still running)
./scripts/deploy.sh rollback

# Check status
./scripts/deploy.sh status

Testing Plan

1. Deploy blue version initially
2. Make code change and deploy (should switch to green)
3. Verify zero dropped requests during switch
4. Make another change and deploy (should switch back to blue)
5. Test rollback scenario
6. Test failed deployment scenario (new version doesn't pass health check)

Alternative: Simpler Rolling Update

If blue-green is too resource-intensive, use rolling updates:

docker compose up -d --no-deps --build platform

Caddy's lb_try_duration handles brief unavailability (~10-30s potential for errors).

---

Detailed Plan

See the comment below for the full implementation plan with diagrams.

[larryro]

Full Implementation Plan

How Zero-Downtime Works (Conceptual)

The Problem Today

Currently, when you run docker compose up -d --build:

Time 0:00  User request arrives
           ↓
Time 0:01  Old container stops (SIGTERM)
           ↓
Time 0:02  ❌ USER SEES ERROR (503 Service Unavailable)
           ↓
Time 0:30  New container starts building
           ↓
Time 1:00  New container starts
           ↓
Time 1:30  Health check passes
           ↓
Time 1:31  ✅ Service available again

DOWNTIME: ~90 seconds

The Solution: Blue-Green Deployment

Blue-green deployment runs TWO versions simultaneously during the switch:

┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 1: Normal Operation (Blue is active)                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Internet ──► Caddy ──┬──► platform-blue:3000 ✅ (healthy)        │
│                        │                                            │
│                        └──► platform-green:3000 ❌ (not running)   │
│                                                                     │
│   Caddy routes ALL traffic to blue (the only healthy backend)       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 2: Deployment Started (Green starting up)                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Internet ──► Caddy ──┬──► platform-blue:3000 ✅ (healthy)        │
│                        │    ↑ ALL TRAFFIC HERE                      │
│                        │                                            │
│                        └──► platform-green:3000 ⏳ (starting)       │
│                             ↑ NO TRAFFIC (health check failing)     │
│                                                                     │
│   Users experience NO interruption - blue handles everything        │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 3: Green Ready (Both healthy, green preferred)               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Internet ──► Caddy ──┬──► platform-blue:3000 ✅ (healthy)        │
│                        │    ↑ DRAINING (finishing existing requests)│
│                        │                                            │
│                        └──► platform-green:3000 ✅ (healthy)        │
│                             ↑ NEW TRAFFIC GOES HERE                 │
│                                                                     │
│   Caddy detected green is healthy, starts routing new requests      │
│   Blue finishes any in-flight requests                              │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 4: Cleanup (Blue removed)                                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Internet ──► Caddy ──┬──► platform-blue:3000 ❌ (stopped)        │
│                        │                                            │
│                        └──► platform-green:3000 ✅ (healthy)        │
│                             ↑ ALL TRAFFIC HERE                      │
│                                                                     │
│   Old blue containers removed, resources freed                      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

RESULT: Zero requests dropped, users never see an error

How Caddy Knows Where to Send Traffic

Caddy's reverse proxy has health checking built in:

reverse_proxy {
    to platform-blue:3000 platform-green:3000   ← Both backends listed

    health_uri /api/health     ← Caddy pings this endpoint
    health_interval 3s         ← Every 3 seconds
    health_status 200          ← Only healthy if returns 200

    lb_policy first            ← Use first healthy backend
}

Key behavior:

• Caddy continuously pings /api/health on both backends
• If a backend fails health check → Caddy stops sending it traffic
• If a backend passes health check → Caddy starts sending it traffic
• This happens automatically, no manual intervention needed

Why We Need Overlay Files

Docker Compose needs to know which containers belong to "blue" vs "green":

# compose.blue.yml
services:
  platform:
    container_name: tale-platform-blue      # Unique name
    networks:
      internal:
        aliases:
          - platform-blue                   # DNS name Caddy uses

When we deploy:

# Start blue version
docker compose -f compose.yml -f compose.blue.yml up -d

# Later, start green version (blue keeps running!)
docker compose -f compose.yml -f compose.green.yml up -d

Both can run simultaneously because they have different names.

---

Color Rotation Logic Explained

The Core Principle

The script maintains two variables:

• CURRENT: The color currently serving traffic (the one to STOP later)
• TARGET: The color we're deploying to (the one to START now)

┌─────────────────────────────────────────────────────────────────────┐
│                     COLOR ROTATION RULE                             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   IF current is BLUE  → target is GREEN → after deploy, stop BLUE  │
│   IF current is GREEN → target is BLUE  → after deploy, stop GREEN │
│   IF current is NONE  → target is BLUE  → first deployment         │
│                                                                     │
│   Simple toggle: CURRENT and TARGET are always opposite colors      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Timeline of Multiple Deployments

════════════════════════════════════════════════════════════════════════════
                          DEPLOYMENT TIMELINE
════════════════════════════════════════════════════════════════════════════

TIME        BLUE STATUS      GREEN STATUS     SERVING TRAFFIC    ACTION
────────────────────────────────────────────────────────────────────────────

Day 1       (not exists)     (not exists)     (none)
  │
  │ ─────── Deploy #1 starts ───────
  │         CURRENT=none, TARGET=blue
  │
  ▼
            Starting...      (not exists)     (none)             Build blue
            ✅ Healthy       (not exists)     ◄─── BLUE          Blue ready
────────────────────────────────────────────────────────────────────────────

Day 2       ✅ Running       (not exists)     ◄─── BLUE          Normal ops
  │
  │ ─────── Deploy #2 starts ───────
  │         CURRENT=blue, TARGET=green
  │
  ▼
            ✅ Running       Starting...      ◄─── BLUE          Build green
            ✅ Running       ✅ Healthy       ◄─── BLUE          Green ready
            ✅ Running       ✅ Healthy       ◄─── GREEN         Traffic switch
            Draining...      ✅ Running       ◄─── GREEN         Wait 30s
            ❌ Stopped       ✅ Running       ◄─── GREEN         Stop CURRENT
────────────────────────────────────────────────────────────────────────────

Day 3       (not exists)     ✅ Running       ◄─── GREEN         Normal ops
  │
  │ ─────── Deploy #3 starts ───────
  │         CURRENT=green, TARGET=blue
  │
  ▼
            Starting...      ✅ Running       ◄─── GREEN         Build blue
            ✅ Healthy       ✅ Running       ◄─── GREEN         Blue ready
            ✅ Healthy       ✅ Running       ◄─── BLUE          Traffic switch
            ✅ Running       Draining...      ◄─── BLUE          Wait 30s
            ✅ Running       ❌ Stopped       ◄─── BLUE          Stop CURRENT
────────────────────────────────────────────────────────────────────────────

Pattern continues: blue → green → blue → green → blue → ...

How the Script Determines What to Stop

┌─────────────────────────────────────────────────────────────────────┐
│ DECISION LOGIC (in deploy.sh)                                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 1: At script START, detect what's running                     │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    Check: docker ps | grep tale-platform                            │
│                                                                     │
│    Result A: tale-platform-blue found                               │
│              → CURRENT = "blue"                                     │
│              → TARGET  = "green"                                    │
│                                                                     │
│    Result B: tale-platform-green found                              │
│              → CURRENT = "green"                                    │
│              → TARGET  = "blue"                                     │
│                                                                     │
│    Result C: Neither found (first deployment)                       │
│              → CURRENT = "none"                                     │
│              → TARGET  = "blue"                                     │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 2: Start TARGET color containers                              │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    docker compose -f compose.yml -f compose.${TARGET}.yml up -d     │
│                                                                     │
│    Now both CURRENT and TARGET are running                          │
│    (or just TARGET if first deployment)                             │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 3: Wait for TARGET to be healthy                              │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    while ! curl -sf http://platform-${TARGET}:3000/api/health; do   │
│        sleep 2                                                      │
│    done                                                             │
│                                                                     │
│    At this point: BOTH are healthy (if not first deployment)        │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 4: Switch traffic to TARGET                                   │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    Caddy reload → prefers TARGET (order in config)                  │
│    Traffic now flows to TARGET                                      │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 5: Stop CURRENT (not TARGET!)                                 │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    if [ "$CURRENT" != "none" ]; then                                │
│        sleep 30  # drain time                                       │
│        docker compose -f compose.yml \                              │
│                       -f compose.${CURRENT}.yml down                │
│    fi                                                               │
│                                                                     │
│    KEY: We stored CURRENT at the START of the script                │
│         We know exactly which one to stop                           │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STEP 6: Record new state                                           │
│  ─────────────────────────────────────────────────────────────────  │
│                                                                     │
│    echo "$TARGET" > /var/run/tale-deployment-color                  │
│                                                                     │
│    Next deployment will read this file:                             │
│    - If file says "green" → CURRENT=green, TARGET=blue              │
│    - If file says "blue"  → CURRENT=blue, TARGET=green              │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Why This Works When Both Are Healthy

The key insight is: the script decides what to stop BEFORE starting the new version.

┌─────────────────────────────────────────────────────────────────────┐
│ EXAMPLE: Both healthy, how does script know what to stop?           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│ Scenario: Green was serving, we're deploying Blue                   │
│                                                                     │
│   At script start:                                                  │
│   ┌──────────────────────┐                                          │
│   │ CURRENT = "green"    │  ← Detected: green is running            │
│   │ TARGET  = "blue"     │  ← Calculated: opposite of current       │
│   └──────────────────────┘                                          │
│         │                                                           │
│         │  These values are LOCKED IN at script start               │
│         │  They don't change even when both become healthy          │
│         ▼                                                           │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │ After Blue starts and becomes healthy:                       │  │
│   │                                                              │  │
│   │   Green: ✅ running, ✅ healthy                              │  │
│   │   Blue:  ✅ running, ✅ healthy                              │  │
│   │                                                              │  │
│   │   Script still knows: CURRENT="green", TARGET="blue"         │  │
│   │                                                              │  │
│   │   So it will:                                                │  │
│   │   1. Switch traffic to TARGET (blue)                         │  │
│   │   2. Stop CURRENT (green)                                    │  │
│   │                                                              │  │
│   │   It doesn't need to "figure out" which to stop.             │  │
│   │   It already decided at the beginning!                       │  │
│   └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

Database Consistency During Deployments

The Shared Database Model

Both Blue and Green services connect to the same database. This is intentional and necessary:

┌─────────────────────────────────────────────────────────────────────┐
│                   SHARED DATABASE ARCHITECTURE                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   ┌─────────────────┐         ┌─────────────────────────────────┐  │
│   │ platform-blue   │────────►│                                 │  │
│   │ (v1.0 code)     │         │      TimescaleDB (tale-db)      │  │
│   └─────────────────┘         │                                 │  │
│                               │   - Same data for both versions │  │
│   ┌─────────────────┐         │   - Users see consistent data   │  │
│   │ platform-green  │────────►│   - No data duplication needed  │  │
│   │ (v1.1 code)     │         │                                 │  │
│   └─────────────────┘         └─────────────────────────────────┘  │
│                                                                     │
│   WHY shared database?                                              │
│   - User data must be consistent (can't have two "truths")          │
│   - During transition, both versions may serve requests             │
│   - Duplicating DB is expensive and complex (replication lag)       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

The Problem: Schema Changes

When your code update requires database changes, you have a problem:

┌─────────────────────────────────────────────────────────────────────┐
│ DANGEROUS SCENARIO: Breaking schema change                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Old Code (v1.0):  SELECT name, email FROM users                  │
│   New Code (v1.1):  SELECT name, email, phone FROM users           │
│                                                                     │
│   If you run: ALTER TABLE users ADD COLUMN phone VARCHAR(20)       │
│   BEFORE deploying v1.1 code...                                     │
│                                                                     │
│   ✅ v1.1 works (phone column exists)                               │
│   ✅ v1.0 works (extra column is ignored)                           │
│                                                                     │
│   But if you run: ALTER TABLE users DROP COLUMN email              │
│   BEFORE old code is gone...                                        │
│                                                                     │
│   ❌ v1.0 CRASHES (SELECT email fails - column doesn't exist)       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Solution: Expand-Contract Pattern

════════════════════════════════════════════════════════════════════════════
                    EXPAND-CONTRACT MIGRATION PATTERN
════════════════════════════════════════════════════════════════════════════

PHASE 1: EXPAND (Add new, don't remove old)
─────────────────────────────────────────────

   SAFE migrations (backward compatible):
   ┌────────────────────────────────────────────────────────────────────┐
   │ ✅ ADD COLUMN         │ Old code ignores it                       │
   │ ✅ ADD TABLE          │ Old code doesn't use it                   │
   │ ✅ ADD INDEX          │ Old code still works                      │
   │ ✅ ADD NULLABLE COLUMN│ Old code can write NULL                   │
   │ ✅ WIDEN column type  │ Old code still fits (VARCHAR(50)→100)    │
   └────────────────────────────────────────────────────────────────────┘

   Example migration for adding phone field:
   ┌────────────────────────────────────────────────────────────────────┐
   │ -- expand.sql (run BEFORE code deployment)                        │
   │ ALTER TABLE users ADD COLUMN IF NOT EXISTS phone VARCHAR(20);     │
   │ -- Use CONCURRENTLY for indexes to avoid locks                    │
   │ CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_phone           │
   │   ON users(phone);                                                │
   └────────────────────────────────────────────────────────────────────┘

════════════════════════════════════════════════════════════════════════════

PHASE 2: DEPLOY (Code that works with old AND new schema)
─────────────────────────────────────────────────────────────

   Deploy v1.1 code that:
   - Writes to new columns (phone)
   - Still handles missing data gracefully
   - Doesn't rely on removed columns yet

   At this point:
   ┌────────────────────────────────────────────────────────────────────┐
   │ Old code (v1.0): SELECT name, email FROM users   ✅ WORKS         │
   │ New code (v1.1): SELECT name, email, phone FROM users  ✅ WORKS   │
   │                                                                   │
   │ Database has: name, email, phone (all columns present)            │
   └────────────────────────────────────────────────────────────────────┘

════════════════════════════════════════════════════════════════════════════

PHASE 3: CONTRACT (Remove old, after all old code is gone)
─────────────────────────────────────────────────────────────

   UNSAFE migrations (only after old code stopped):
   ┌────────────────────────────────────────────────────────────────────┐
   │ ⚠️ DROP COLUMN        │ Only after no code uses it               │
   │ ⚠️ DROP TABLE         │ Only after no code uses it               │
   │ ⚠️ RENAME COLUMN      │ Old code will break                      │
   │ ⚠️ NARROW column type │ Old code may write too-long data        │
   │ ⚠️ ADD NOT NULL       │ Old code may write NULL                  │
   └────────────────────────────────────────────────────────────────────┘

════════════════════════════════════════════════════════════════════════════

Migration Timing Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                  MIGRATION TIMING DIAGRAM                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│ TIME ──────────────────────────────────────────────────────────────►│
│                                                                     │
│ ┌──────────┐                                                        │
│ │ EXPAND   │ Run expand migrations                                  │
│ │ PHASE    │ (Add columns, indexes)                                 │
│ └────┬─────┘                                                        │
│      │                                                              │
│      ▼                                                              │
│ ┌──────────┐ ┌──────────┐                                          │
│ │ Blue v1.0│ │ Blue v1.0│  Both versions can now                   │
│ │ running  │ │ running  │  use the database                        │
│ └──────────┘ └──────────┘                                          │
│              ┌──────────┐                                          │
│              │Green v1.1│  New version starts                      │
│              │ starting │                                          │
│              └────┬─────┘                                          │
│                   │                                                 │
│      ┌────────────┴────────────┐                                   │
│      │ BOTH HEALTHY            │ Traffic switch happens             │
│      │ Both use same schema    │ Users see no difference            │
│      └────────────┬────────────┘                                   │
│                   │                                                 │
│              ┌────┴─────┐                                          │
│              │ Blue v1.0│  Old version stopped                     │
│              │ stopped  │                                          │
│              └────┬─────┘                                          │
│                   │                                                 │
│                   ▼                                                 │
│ ┌──────────┐                                                        │
│ │ CONTRACT │ Now safe to run contract migrations                    │
│ │ PHASE    │ (Remove old columns - v1.0 is gone)                   │
│ └──────────┘                                                        │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Example: Renaming a Column (Requires 3 Deployments!)

════════════════════════════════════════════════════════════════════════════
     EXAMPLE: Renaming column 'username' to 'display_name'
════════════════════════════════════════════════════════════════════════════

This requires THREE deployments, not one!

DEPLOYMENT 1: Add new column (expand)
─────────────────────────────────────────
   Migration:
   ┌────────────────────────────────────────────────────────────────────┐
   │ ALTER TABLE users ADD COLUMN display_name VARCHAR(100);           │
   │ UPDATE users SET display_name = username WHERE display_name IS NULL;│
   └────────────────────────────────────────────────────────────────────┘

   Code v1.1:
   - Writes to BOTH username AND display_name
   - Reads from display_name (fallback to username)

════════════════════════════════════════════════════════════════════════════

DEPLOYMENT 2: Migrate all reads to new column
──────────────────────────────────────────────
   No migration needed.

   Code v1.2:
   - Still writes to BOTH columns (for rollback safety)
   - Reads ONLY from display_name
   - Old 'username' writes kept for v1.1 rollback

════════════════════════════════════════════════════════════════════════════

DEPLOYMENT 3: Remove old column (contract)
────────────────────────────────────────────
   Migration (after v1.1 is gone):
   ┌────────────────────────────────────────────────────────────────────┐
   │ ALTER TABLE users DROP COLUMN username;                           │
   └────────────────────────────────────────────────────────────────────┘

   Code v1.3:
   - Writes ONLY to display_name
   - Reads ONLY to display_name
   - 'username' no longer exists

════════════════════════════════════════════════════════════════════════════

Convex Database Considerations

┌─────────────────────────────────────────────────────────────────────┐
│                    CONVEX SCHEMA CHANGES                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Convex handles schema changes through its deploy process:         │
│                                                                     │
│   1. Schema is defined in convex/schema.ts                          │
│   2. On deploy, Convex validates schema compatibility               │
│   3. For breaking changes, Convex will fail the deploy              │
│                                                                     │
│   For zero-downtime with Convex:                                    │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │ - Add optional fields first (don't add required fields)      │  │
│   │ - Use v.optional() for new fields                            │  │
│   │ - Backfill data before making fields required                │  │
│   │ - Remove fields only after code no longer reads them         │  │
│   └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
│   Example schema evolution:                                         │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │ // v1.0 schema                                               │  │
│   │ users: defineTable({                                         │  │
│   │   name: v.string(),                                          │  │
│   │   email: v.string(),                                         │  │
│   │ })                                                           │  │
│   │                                                              │  │
│   │ // v1.1 schema (add optional phone)                          │  │
│   │ users: defineTable({                                         │  │
│   │   name: v.string(),                                          │  │
│   │   email: v.string(),                                         │  │
│   │   phone: v.optional(v.string()),  // New, optional           │  │
│   │ })                                                           │  │
│   │                                                              │  │
│   │ // v1.2 schema (after backfill, make required)               │  │
│   │ users: defineTable({                                         │  │
│   │   name: v.string(),                                          │  │
│   │   email: v.string(),                                         │  │
│   │   phone: v.string(),  // Now required                        │  │
│   │ })                                                           │  │
│   └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

Detailed Implementation

Step 1: Update compose.yml

Remove container_name from stateless services and optimize health checks:

# For each stateless service (platform, rag, crawler, search, graph-db):
services:
  platform:
    # container_name: tale-platform  # REMOVE
    healthcheck:
      test: ['CMD', 'curl', '-sf', 'http://localhost:3000/api/health']
      interval: 5s        # Was 30s
      timeout: 3s         # Was 10s
      retries: 2          # Was 3
      start_period: 45s   # Was 60s

Step 2: Create compose.blue.yml

services:
  platform:
    container_name: tale-platform-blue
    networks:
      internal:
        aliases:
          - platform-blue

  rag:
    container_name: tale-rag-blue
    networks:
      internal:
        aliases:
          - rag-blue

  crawler:
    container_name: tale-crawler-blue
    networks:
      internal:
        aliases:
          - crawler-blue

  search:
    container_name: tale-search-blue
    networks:
      internal:
        aliases:
          - search-blue

  graph-db:
    container_name: tale-graph-db-blue
    networks:
      internal:
        aliases:
          - graph-db-blue

Step 3: Create compose.green.yml

Same structure as blue, with -green suffix.

Step 4: Update Caddyfile

{$DOMAIN:http://localhost} {
    @needsTLS not host *.test *.local *.localhost localhost 127.0.0.1
    tls {
        issuer acme {
            dir {$ACME_CA:https://acme-v02.api.letsencrypt.org/directory}
            disable_tlsalpn_challenge
        }
    }

    log {
        output stdout
        format console
        level INFO
    }

    # Blue-Green Upstream Pool
    reverse_proxy {
        to platform-blue:3000 platform-green:3000

        lb_policy first

        # Active health checking
        health_uri /api/health
        health_interval 3s
        health_timeout 2s
        health_status 200

        # Passive health checking (circuit breaker)
        fail_duration 10s
        max_fails 3
        unhealthy_status 500 502 503 504

        # Retry on failure
        lb_try_duration 5s
        lb_try_interval 250ms

        header_up Host {host}
        header_up X-Real-IP {remote_host}
    }

    encode gzip zstd

    header {
        -Server
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        Referrer-Policy "strict-origin-when-cross-origin"
    }

    @https {
        protocol https
    }
    header @https Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
}

Step 5: Create scripts/deploy.sh

Key functions:

• get_current_color() - Detect which color is active
• deploy_new_version() - Build and start new color
• wait_for_health() - Poll health endpoints
• switch_traffic() - Reload Caddy config
• cleanup_old() - Drain and remove old containers

Step 6: Update docker-entrypoint.sh

Add graceful shutdown:

shutdown() {
  touch /tmp/shutting_down  # Signal to health check
  sleep 5                    # Let load balancer detect
  kill -TERM $PIDS
  wait $PIDS
}
trap shutdown SIGTERM SIGINT

---

Deployment Workflow

┌────────────────────────────────────────────────────────────────────┐
│ STEP 1: Detect Current State                                       │
├────────────────────────────────────────────────────────────────────┤
│   $ docker ps --filter name=tale-platform                          │
│   If tale-platform-blue is running → deploy green                  │
│   If tale-platform-green is running → deploy blue                  │
│   If neither → this is first deployment, use blue                  │
└────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────┐
│ STEP 2: Build New Version                                          │
├────────────────────────────────────────────────────────────────────┤
│   $ docker compose -f compose.yml -f compose.green.yml build       │
│   The old blue containers keep running, users unaffected           │
└────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────┐
│ STEP 3: Start New Containers                                       │
├────────────────────────────────────────────────────────────────────┤
│   $ docker compose -f compose.yml -f compose.green.yml up -d       │
│   Traffic still goes to blue                                       │
└────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────┐
│ STEP 4: Wait for Health                                            │
├────────────────────────────────────────────────────────────────────┤
│   Script polls: curl http://platform-green:3000/api/health         │
│   Timeout: 120 seconds (configurable)                              │
│   If timeout → abort deployment, blue keeps running                │
└────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────┐
│ STEP 5: Traffic Switches                                           │
├────────────────────────────────────────────────────────────────────┤
│   $ docker exec tale-proxy caddy reload                            │
│   New requests go to green, in-flight requests finish on blue      │
└────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────┐
│ STEP 6: Drain and Cleanup                                          │
├────────────────────────────────────────────────────────────────────┤
│   Wait 30 seconds for in-flight requests to complete               │
│   $ docker compose -f compose.yml -f compose.blue.yml down         │
│   Next deployment will use blue again (alternates)                 │
└────────────────────────────────────────────────────────────────────┘

TOTAL TIME: ~2-3 minutes
USER IMPACT: Zero downtime

---

Summary: Database Safety Rules

Rule	Description
Never drop/rename in same deploy	Old code will break immediately
Add nullable first	Old code can write NULL safely
Use CONCURRENTLY for indexes	Avoid table locks during creation
Contract after old code stops	Only remove columns when no code uses them
Test rollback scenario	What if you need to go back to v1.0?
Same DB for both colors	No data duplication, consistent user experience