Autonomous UX Mapping, Drift Detection & AI-Driven QA
AI agents crawl mobile apps, discover screens/transitions, and build a navigable knowledge graph.
- Autonomous exploration via UI actions
- Real-time UX map generation
- Visual/structural diff detection
- Competitor UX mapping
- Test gap identification
- Analytics overlay on graph
- AI agent context for RAG/planning
- QA: Regression detection, exploratory testing
- Product: UX analysis, journey mapping
- Design: Flow consistency validation
- Founders: Competitive benchmarking
- Engineering: Visual bug replay
- ScreenGraph: Screens (nodes) + actions (edges)
- Flows: Reusable step sequences
- Perceptual Hashing: Screen uniqueness detection
- Policies: Exploration strategies
- Artifacts: Screenshots, XMLs, OCR, traces
┌─────────────────┐ ┌─────────────────┐
│ SvelteKit │────────▶│ Encore Backend│
│ Frontend │ REST │ Services │
│ (Vercel) │ +SSE │ (Encore Cloud)│
└─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ PostgreSQL │
│ + Event Store │
└─────────────────┘
Stack:
- Backend: Encore.ts microservices
- Frontend: SvelteKit 2 (Svelte 5)
- Agent: XState-driven orchestration
- Database: PostgreSQL with event sourcing
- Logging: Structured logs (module/actor)
Single-Sink Architecture:
- Agent writes events to
run_events(append-only log) - Graph projection service derives screen graph
- No dual-writes
Screen Deduplication:
- Identified by structural layout (XML) + visual appearance (perceptual hash)
- Same screen across runs → single canonical record
- Tracks: first seen, last seen, visit count
Navigation Edges:
- Captures: Screen A → Action → Screen B
- Evidence-based traversal counts
- Enables pathfinding + reachability analysis
API Endpoints (Planned):
GET /graph/:app_id- Full graphGET /graph/screens/:screen_id- Screen detailsGET /graph/:app_id/paths- PathfindingGET /graph/:app_id/coverage- Coverage metricsGET /graph/:app_id/stream- Real-time SSE
Install Bun:
curl -fsSL https://bun.sh/install | bashInstall Encore CLI:
# macOS
brew install encoredev/tap/encore
# Linux
curl -L https://encore.dev/install.sh | bashClone:
git clone <repository-url>
cd ScreenGraphInstall Dependencies:
bun install
cd frontend && bun installEnvironment (.env):
# Backend: Managed by Encore (encore secret set)
# Frontend: Create frontend/.env
PUBLIC_API_BASE=http://localhost:4000 # local
# PUBLIC_API_BASE=https://steering-wheel-documentation-65b2.encr.app # prodcd backend && encore run- API:
http://localhost:4000 - Dashboard:
http://localhost:9400
cd frontend && bun run dev- UI:
http://localhost:5173
cd backend && encore testSpecific test:
cd backend && encore test agent/tests/determinism.test.tscd frontend && bun testRun Management:
POST /run- Start agent runGET /run/:id/stream- SSE event streamPOST /run/:id/cancel- Cancel runGET /health- Health check
Event Types:
agent.*- Agent state transitionsscreen.discovered- New screen foundscreen.mapped- Screen added to graphedge.created- New navigation edgeedge.reinforced- Existing edge traversed
ScreenGraph/
├── backend/
│ ├── agent/ # Agent orchestration
│ │ ├── orchestrator/ # XState machine + worker
│ │ ├── nodes/ # Setup/Main/Policy/Recovery/Terminal
│ │ ├── ports/ # Interfaces
│ │ └── adapters/ # Implementations
│ ├── run/ # Run API endpoints
│ ├── graph/ # Graph projection
│ ├── artifacts/ # Screenshot/XML storage
│ └── db/migrations/ # Schema migrations
├── frontend/
│ ├── src/routes/ # SvelteKit routes
│ └── src/lib/ # Components + Encore client
├── .cursor/
│ ├── commands/ # Task automation
│ └── rules/ # Coding standards
└── vibes/ # Engineering personas
Migrations:
cd backend && encore db reset --all # Reset dev cluster
cd backend && encore db shell db # Shell access
cd backend && encore db migrate # Run migrationsTest Cluster:
cd backend && encore db reset --all --test
cd backend && encore db shell db --testPort conflicts:
lsof -ti:4000 | xargs kill # Backend
lsof -ti:5173 | xargs kill # FrontendDatabase issues:
cd backend && encore db reset --allFrontend sync errors:
cd frontend
rm -rf .svelte-kit node_modules
bun install
bunx svelte-kit sync
bun run devBackend (Encore Cloud):
encore auth login
git push encore mainDashboard: https://app.encore.cloud/steering-wheel-documentation-65b2
Frontend (Vercel):
cd frontend && vercel --prodSet env: PUBLIC_API_BASE=https://steering-wheel-documentation-65b2.encr.app
Backend:
encore secret set --type prod SecretName
encore secret listFrontend: Vercel Dashboard → Settings → Environment Variables
Backend:
- Dashboard: https://app.encore.cloud/steering-wheel-documentation-65b2
- Logs:
encore logsorencore logs --env=prod - Metrics: Auto-collected by Encore
Frontend:
- Vercel Dashboard: https://vercel.com/dashboard
- Analytics: Enable in settings
✅ Completed:
- Agent orchestration (XState machine)
- Structured logging (module/actor)
- Type-safe APIs (Encore generated clients)
- Event sourcing + snapshots
- Run lifecycle (start/cancel/stream)
🟡 In Progress:
- Node handler implementation
- Main loop (Perceive → Act → Verify)
- Graph projection service
⏳ Upcoming:
- LLM integration for action selection
- Graph visualization UI
- Multi-policy exploration
- Analytics overlay
Framework Docs:
Project Docs:
CLAUDE.md- Commands + quick reference.cursor/rules/- Coding standardsbackend/agent/CLAUDE.md- Agent architecturebackend/graph/README.md- Graph implementationbackend/API_DOCUMENTATION.md- Complete API reference
External:
Git Workflow:
git checkout -b feature/my-feature
# ... make changes ...
git commit -m "feat: description"
git push origin feature/my-featureStandards:
- Backend: Encore.ts conventions
- Frontend: Svelte 5 runes (
$state,$derived,$effect) - TypeScript: Strict mode, no
any - Testing: All tests passing before PR
[Your License Here]
ScreenGraph - Autonomous UX mapping for mobile apps.
For detailed commands: CLAUDE.md
For coding standards: .cursor/rules/founder_rules.mdc