BUILDING IN PUBLIC

Data platforms that run themselves.

I'm Vijay — 21+ years in data infrastructure, and my work is in the open: a 2M message/sec Apache Fluss benchmark merged upstream, and the Intelligent Data Platform documented layer by layer as I build it. A platform where every component has its own agent: you describe the business need, and the agents build it, run it and keep it cheap — so a small team ships like a much bigger one. Built in public, one layer at a time.

21+ yrsHyperscale data systems
2M msg/secApache Fluss benchmark, merged upstream
100+ daysConsecutive public build log
Open now

The bootcamp

Eight weeks, live and hands-on, building a self-operating data platform one agent layer at a time. Small groups, running now.

See the eight weeks and enrol
01 — What I'm building

The Intelligent Data Platform

A two-to-five person data team can absolutely run hyperscale infrastructure. What they can't do is run it and build on it, when pipeline babysitting, cost reviews and 2am pages eat the week. The platform takes that load: self-managing, self-healing, cost-aware pipelines, so the team's time goes back into data products.

Write
Ingestion agentsBatch & streaming, discovered automatically
Store
Open lakeZoned layers, live schema evolution
Query
Speed layerPluggable, sub-second queries
Serve
Users & appsSub-second analytics
Agent layers Batch orchestration Streaming orchestration SRE FinOps Governance & security Pipeline Designer

Built on open, proven infrastructure — not a black box. Engines and query layers are pluggable by design, so nothing here locks you into one vendor's table format.

One prompt, a live pipeline

A working beta already takes a single prompt and deploys both a streaming and a batch pipeline, writing into the same open lake table.

Agents take the toil

SRE, FinOps and governance agents watch the platform, heal routine failures and keep spend in line — so engineers spend their attention on the decisions only they can make.

Patterns, not demos

Designed architectural patterns — the Data Boundary Layer and the Agent Stateless Bridge — for safe sync and zero-downtime recovery.

This is becoming a product companies can run themselves — see where it's headed.

02 — Open now

AI Agents for Data Engineering

Eight weeks building the platform above, from the inside — one agent layer at a time, until the thing runs itself. The curriculum came straight out of what working data engineers said they were stuck on, not a syllabus I wrote at a whiteboard.

  • W1Core platform & agent foundationA component registry behind one unified interface, and every component gets its own agent. You describe the business need; the agent builds the environment.
  • W2Batch pipelinesThe ingestion agent discovers sources on its own. Schedules, partitions and retries all handled — zero manual config.
  • W3Streaming pipelinesThe streaming agent handles lag, windowing and late data. When a component hits its limit, the agent spawns another in under 60 seconds.
  • W4Data lake, CDC & schema evolutionOpen lake tables join the platform. The evolution agent detects schema drift and migrates it live. Your source database changes at 2am — you sleep through it.
  • W5Reliability & observabilityThe SRE agent reads the logs, finds root cause, quarantines bad records and restores health autonomously. Ask the platform why a pipeline failed, and it answers.
  • W6Cost optimisation & scalingThe FinOps agent spawns components on demand, optimises queries and tiers cold data automatically. Data grows 10×; cost grows 0.5×.
  • W7Governance & securityData lands untouched. The policy agent profiles it, auto-classifies PII, PHI and PCI, and tags it — access locked by tag, nothing masked on write. Masking is applied only when someone requests access, so the original is never lost and every environment enforces the same policy from the same tags.
  • W8The morphing final projectOne prompt, and the platform reconfigures itself — Netflix-style personalisation at scale, Uber-style real-time logistics, Maps-style spatial data.

Join the bootcamp

Groups are kept small and run live. Dates, fee and everything included are on the enrolment page — you'll see it all before you pay anything.

  • Format Live, hands-on
  • Length 8 weeks
  • You build A working platform
  • Final project Morphing platform
  • Group size Small, by design
Enrol now

What you walk out with

Not a certificate and a repo of notebooks. By the end you have built a working, self-operating data platform — and you know how each agent in it was designed, tested and trusted with production.

Design the platform itself — a component registry behind one interface, with an agent owning every component.
Go from sentence to running environment — take a business requirement in plain language and have it provisioned, not ticketed.
Build ingestion agents — sources discovered automatically; scheduling, partitioning and retries owned by the agent, not by you.
Build streaming agents — lag, windowing and late data handled — and capacity added by spawning a component in under a minute.
Run an open lake that evolves itself — CDC in, schema drift detected and migrated live, no downtime and no 2am wake-up.
Build an SRE agent that actually resolves — reads logs, reaches a root cause, quarantines bad records and restores health autonomously.
Make a platform you can interrogate — ask why a pipeline failed and get an answer in words, instead of going hunting through dashboards.
Build FinOps agents — provision on demand, optimise queries, tier cold data — so 10× the data doesn't mean 10× the bill.
Implement governance that holds — auto-classified PII, PHI and PCI, tag-based access, masking at provisioning time so raw data is never lost.
Reconfigure a whole platform from one prompt — the same system retargeted at a different domain — the final project proves you can do it.
03 — Who's building it

Vijay

Twenty-one years in data infrastructure, most of it on systems where a bad design decision shows up as a production incident rather than a slow dashboard. The habit that came out of it: go deep on one thing until it actually holds under load — and publish the numbers, not the adjectives.

I left to build independently. These days that means the Intelligent Data Platform, open-source work in the streaming ecosystem, and writing up every layer publicly as I go — because teaching it is how I find out whether the design is really sound.

  • Apache Fluss contributor. A 2M message/sec IoT benchmark, merged into the official apache/fluss-benchmarks repository — run on EKS via Terraform and Helm.
  • A full agent architecture designed and documented across the batch, streaming, SRE, FinOps and governance layers, with a fintech fraud-detection case study running through all of them.
  • Daily build log. 100+ consecutive days of the platform's design published openly — the dead ends included.
04 — Get in touch

Building something at this scale?

Whether it's the bootcamp, the platform, or a data system that's outgrown its team — I'm reachable.