Fluid Forge

Write YAML. Deploy Anywhere. Own Your Data Products.

📖 Live Docs · ⚡ CLI Repository · 🐛 Report Issue · 💬 Discussions

What Terraform did for infrastructure,
Fluid Forge does for data products.

One contract. Every cloud. Zero lock-in.
Define your data product in a single YAML file — schema, quality rules, lineage, governance, SLAs — then deploy it to GCP BigQuery, AWS Athena, Snowflake, or DuckDB with one command. No rewrites. No vendor lock-in. Ever.

Why Fluid Forge?

Data teams are drowning in glue code. Every cloud has its own APIs, its own deploy patterns, its own way of doing quality checks. You end up maintaining N copies of the same logic across N platforms — and it still breaks on Friday at 5pm.

Fluid Forge kills that complexity with 44 purpose-built CLI commands and a declarative contract system.

Problem	Fluid Forge Solution
Cloud-specific deploy scripts everywhere	One declarative YAML contract that runs on any provider
Schema drift breaks pipelines silently	Built-in `validate` + `verify` catches drift before and after deploy
No visibility into data quality	`test` with quality gates, SLA checks & anomaly detection — in the contract
Orchestration is hand-wired spaghetti	`export` generates Airflow, Dagster, Prefect & Step Functions from your contract
Governance is an afterthought	`policy-check` → `policy-compile` → `policy-apply` — governance-as-code, compiled to native IAM
Switching clouds means rewriting everything	Swap one line (`provider: gcp` → `provider: snowflake`) and redeploy
AI/LLM access to data is ungoverned	`agentPolicy` — declarative boundaries on which models can consume your data

🚀 From Zero to Data Product in 30 Seconds

pip install fluid-forge
fluid init my-project --quickstart
fluid apply contract.fluid.yaml --yes

That's it. Your data product is live — running on DuckDB locally, no cloud account needed. When you're ready for production, change one line and push. Same contract, same guarantees, infinite scale.

# contract.fluid.yaml — your entire data product in one file
fluidVersion: "0.7.1"
kind: DataProduct
id: "gold.crypto.bitcoin_tracker_v1"
name: "Bitcoin Price Tracker"

metadata:
  owner:
    team: data-engineering
    email: data-team@company.com
  domain: crypto
  tags: [real-time, pricing]

exposes:
  - exposeId: bitcoin_prices
    title: "Bitcoin Hourly Prices"
    kind: table
    contract:
      schema:
        - name: timestamp
          type: timestamp
          required: true
        - name: price_usd
          type: float64
          description: "Bitcoin price in USD"
    binding:
      platform: local        # swap to gcp, aws, or snowflake anytime
      format: duckdb_table
      location:
        database: crypto_data
        table: bitcoin_prices

dataQuality:
  rules:
    - type: not_null
      field: price_usd
    - type: freshness
      max_age: "1h"

governance:
  accessPolicy:
    - role: analyst
      access: read
    - role: data-engineer
      access: admin

🌍 Deploy Everywhere — From Laptop to Cloud

🏠 Local
(DuckDB)
_{Develop & test instantly
No cloud account needed}

☁️ GCP
(BigQuery + GCS)
_{Production analytics
Cloud Composer, Pub/Sub, IAM}

🔶 AWS
(S3 + Athena + Glue)
_{Serverless queries
on your data lake}

❄️ Snowflake
_{Enterprise warehouse
Snowpark & dbt integration}

Plus: Build your own providers with the Provider SDK — Databricks, Azure, Postgres, anything.

⚡ 44 Commands — Everything You Need

Fluid Forge isn't just apply. It's a complete data product lifecycle toolkit:

Category	Commands	What It Does
Declare & Deploy	`init` · `validate` · `plan` · `apply` · `execute`	Build, validate, and deploy data products from YAML contracts
Test & Verify	`test` · `verify` · `contract-tests` · `diff`	Live resource validation, schema compatibility, drift detection
Orchestration	`export` · `generate-airflow` · `scaffold-composer` · `scaffold-ci`	Auto-generate Airflow DAGs, Dagster graphs, Prefect flows, CI/CD pipelines
Governance	`policy-check` · `policy-compile` · `policy-apply`	Validate policies, compile to native IAM, and enforce — all from the contract
Visualization	`viz-graph` · `viz-plan` · `preview`	Lineage diagrams (SVG/PNG/HTML), execution plan visualization
Publishing	`publish` · `export-opds` · `odcs` · `datamesh-manager`	Register in catalogs, export to OPDS/ODCS, push to Data Mesh Manager
AI & Blueprints	`forge --mode copilot` · `forge --mode agent` · `blueprint` · `marketplace`	Adaptive AI-assisted creation, spec-backed domain agents, blueprint templates, marketplace discovery
Config & Admin	`context` · `providers` · `doctor` · `auth` · `wizard`	Provider management, diagnostics, interactive onboarding

Run fluid doctor to verify your setup, or fluid wizard for interactive onboarding.

🛡 Built-In Governance & Compliance

Governance isn't a plugin — it's a first-class citizen in every contract:

Column-level sensitivity — Tag PII, classify data at the field level
Access policies — RBAC rules that compile to native cloud IAM (BigQuery, Snowflake, AWS)
Data sovereignty — Jurisdiction and residency enforcement baked into the contract
Agentic governance — Control which AI/LLM models can access your data and for what purposes
Quality gates — Anomaly detection, SLA thresholds, and freshness checks that block bad deploys

fluid policy-check contract.fluid.yaml     # Validate policies
fluid policy-compile contract.fluid.yaml   # Compile to native IAM JSON
fluid policy-apply contract.fluid.yaml     # Enforce on infrastructure

📚 Documentation

Everything you need to go from first install to production-grade data products:

Section	Description
Getting Started	Install & build your first data product in under 2 minutes
Walkthroughs	Step-by-step guides — Local, GCP, Airflow, Jenkins CI/CD
CLI Reference	Full command reference for all 45 top-level CLI commands
Providers	Deep dives into GCP, AWS, Snowflake, Local & Custom providers
Advanced	Blueprints, governance, Airflow integration, AI-powered agents
Vision & Roadmap	Where we're headed and how to shape the future

🛠 Running the Docs Locally

git clone https://github.com/Agentics-Rising/forge_docs.git
cd forge_docs
npm install
npm run docs:dev       # Dev server at http://localhost:8080

Build for production:

npm run docs:build     # Output → docs/.vuepress/dist/
npm run docs:preview   # Preview the production build

🏗 Site Structure

docs/
├── README.md                  # Home page (live site landing)
├── vision.md                  # Philosophy & roadmap
├── contributing.md            # How to contribute
├── RELEASE_NOTES_0.7.1.md    # Latest release notes
├── getting-started/           # Installation & first steps
├── walkthrough/               # Step-by-step guides (Local, GCP, Airflow, Jenkins)
├── cli/                       # CLI command reference
├── providers/                 # Provider docs (GCP, AWS, Snowflake, Local, Custom)
├── advanced/                  # Blueprints, governance, Airflow integration, AI agents
└── .vuepress/
    ├── config.ts              # VuePress configuration & navigation
    └── public/                # Static assets (logo, favicon)

🚢 Deployment

Pushes to main automatically build and deploy to GitHub Pages via the workflow in .github/workflows/deploy-docs.yml.

Live docs: https://agentics-rising.github.io/forge_docs/

Note: To enable GitHub Pages, go to your repository Settings → Pages and set the source to GitHub Actions.

🤝 Contributing

We welcome contributions! Whether it's fixing a typo, improving an explanation, or adding a new guide:

Fork this repository
Create a branch (git checkout -b docs/my-improvement)
Install dependencies with npm ci
Make your changes — the dev server hot-reloads on save
Run npm run docs:build
Open a Pull Request

If the docs change accompanies a CLI change, include the related forge-cli PR in the docs PR description. Docs-only PRs are welcome too.

For detailed guidelines, see CONTRIBUTING.md and our Code of Conduct.

🔗 Related Repositories

Repository	Description
`forge-cli`	The Fluid Forge CLI — the core engine
`forge-cli-sdk`	SDK for building custom providers

License

Licensed under the Apache License, Version 2.0. See LICENSE for the full license text and NOTICE for attribution details.

Built with VuePress 2 · Powered by Fluid Forge
_{Declarative Data Products for Modern Data Teams}

Proudly developed by dustlabs.co.za

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
.github		.github
docs		docs
examples		examples
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
SECURITY.md		SECURITY.md
package-lock.json		package-lock.json
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Fluid Forge

Write YAML. Deploy Anywhere. Own Your Data Products.

What Terraform did for infrastructure,
Fluid Forge does for data products.

Why Fluid Forge?

🚀 From Zero to Data Product in 30 Seconds

🌍 Deploy Everywhere — From Laptop to Cloud

⚡ 44 Commands — Everything You Need

🛡 Built-In Governance & Compliance

📚 Documentation

🛠 Running the Docs Locally

🏗 Site Structure

🚢 Deployment

🤝 Contributing

🔗 Related Repositories

License

About

Releases

Packages

Contributors

Folders and files

Latest commit

History

Repository files navigation

Fluid Forge

Write YAML. Deploy Anywhere. Own Your Data Products.

What Terraform did for infrastructure,Fluid Forge does for data products.

Why Fluid Forge?

🚀 From Zero to Data Product in 30 Seconds

🌍 Deploy Everywhere — From Laptop to Cloud

⚡ 44 Commands — Everything You Need

🛡 Built-In Governance & Compliance

📚 Documentation

🛠 Running the Docs Locally

🏗 Site Structure

🚢 Deployment

🤝 Contributing

🔗 Related Repositories

License

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

What Terraform did for infrastructure,
Fluid Forge does for data products.