Whiteboard-proof diagrams for data pipelines

Draw data pipelines
everyone can read.

A handful of simple shapes (circles, squares, triangles and arrows) to design, describe and explain data pipelines. Sketch it on a whiteboard, or describe it in a few lines and let your docs, your CI or your AI agent draw it.

Learn the shapes Diagrams as code Drawing tools
Example pipeline: a 'create report' trigger starts a data collector that reads items, statistics and user data, then a report generator writes to the reports store

Why Pipeline Diagrams

Diagrams help when designing, describing or explaining how systems work. Pipeline Diagrams capture data pipelines (originally on AWS), their components and how they connect.

The notation is whiteboard proof: everything can be drawn by hand in seconds, but it works just as well in Keynote, PowerPoint, OmniGraffle, Visio or draw.io.

Models gradually stiffen during a design. Start at a high level with a couple of simple symbols, and add detail, like processor and data store types, once your architecture settles. That keeps the focus at the right level of abstraction.

The method started in 2016 in a fast-growing data team, and is now revived for the age of AI agents. Read the origin story.

Diagrams as code

Describe a diagram in a few lines, and let it draw itself: in Markdown next to your code, in CI from your own scripts, or in a chat with an AI agent. It runs in your browser with nothing to install, and your diagram never leaves your computer.

trigger report manual "create report"
store items D "items"
processor collector PY "data collector"
lambda generator JS "report generator"
store reports S3 "reports"

report, items -> collector -A-> generator -> reports
note collector "m4.xlarge"
In your browser Nothing to install, fully local. Download the offline viewer, or add one script tag. For AI agents Claude and other agents draw, check and fix pipeline diagrams with an MCP server or a skill. npm install pipeline-diagrams JavaScript, command line, web pages, Markdown, GitHub and Obsidian. Try it in the browser. Language reference Every shape, flow, group and note, with live examples.

What is a pipeline

A pipeline is any path that manipulates and stores data. It starts at a trigger or event, data gets processed, and it ends in a data store. You can often spot pipelines by following the arrows that connect two data stores. A pipeline can consist of many other pipelines.

If you can slap a name on a process between two data stores, it probably is, or should become, a pipeline.

The basic shapes

Processor A circle. Takes input data, transforms it and writes the output.
Data store A square with a bar on top. Any persistent or temporary store.
Trigger A triangle. Kicks off a pipeline: manual, timed or evented.
Flow An arrow. Synchronous, asynchronous, queued or streaming.
Other A rectangle with the name of the system inside.
External A box filled grey. Not the topic now, and you can't change it.
Internal A box with one diagonal line. Not the topic now, but yours: you can change it if you have to.

There are also symbols for user facing apps, humans, annotations, lists and grouping, and a few rules that keep diagrams readable. The notation section has them all.

Training

Teach your team the method in one session of two and a half hours, with pen and paper. The workshop takes groups from the basic shapes to their own pipelines: sketch a full pipeline, name every path, colour in what it really is, and ask what happens at ten times the data. The page doubles as the slides.

Workshop: pipelines on paper A hands-on session for teams: shapes, two sketching exercises, reviews and questions. Read it, or present it as slides.

Draw it in a tool

  • draw.io: open draw.io with the shapes loaded, or add the library to a diagram with File › Open Library from › URL, using https://pipeline-diagrams.com/downloads/pipeline-diagrams.drawio.xml.
  • Excalidraw: a library for the whiteboard feel.
  • PowerPoint, Keynote, Miro, Figma: an icon pack.

All of them are on the Drawing tools page.