This repo is in active early-stage development. Expect bugs, breakage, and rapid iteration.
Streetlight API is the public infrastructure layer of the Street Ninja ecosystem — a clean, standards-based API for exposing real-time resource data to support homeless and low-income individuals in Vancouver.
The goal is to make Streetlight API the most accurate and up-to-date source of crisis resources available, built not just for Street Ninja, but for any nonprofit, outreach team, or civic tech project that wants to help.
This app is responsible for:
- Scheduling scraping jobs (HTML, PDF, API)
- Passing raw content to the Python-powered Ninja Crawl scraping engine
- Validating, normalizing, and storing structured resource data (shelters, meals, hygiene, etc.)
- Exposing a clean, public-facing API
Streetlight API delegates all parsing logic to Ninja Crawl, a standalone Python FastAPI service. This app never scrapes HTML or PDFs directly — instead, it handles orchestration:
- Fetches raw content (using Axios, Playwright, or external APIs)
- Sends raw data to Ninja Crawl (
POST /scrape) - Validates the response using Zod schemas
- Saves the result to PostgreSQL via Prisma
- Language: TypeScript (strict)
- Framework: Express
- HTTP Client: Axios
- Browser Automation: Playwright & Chromium
- Database: PostgreSQL (Prisma ORM)
- Scraping Queue: BullMQ (Redis-backed jobs)
- Validation: Zod
- Scheduling: node-cron
- Logging: Pino
- Testing: Jest
Know of a website, PDF, or open data portal that lists services for unhoused or low-income people in BC?
Please reach out. I’m always looking to expand the dataset that powers Street Ninja.
Start a discussion, open an issue, or email me.
- Streetlight API (you are here) — orchestrates scraping and exposes public API
- Ninja Crawl — Python-based scraping engine (HTML/PDF → JSON)
- Street Ninja SMS App — SMS assistant for accessing resources by text
- Street Ninja Website — map and public frontend
If you believe in what I'm building and want to help cover hosting or SMS costs:
