How I engineer systems.

Scale is a question about the first thing that breaks, not about the biggest number in the deck. I look for the bottleneck — a partition key, a single writer, a shared lock, a hot node — and design so that adding capacity moves that limit instead of moving the queue. If throughput does not improve when I add a worker, the architecture is wrong, not the hardware.

Every dependency will be unavailable at some point. The design question is which failure mode I want: produce nothing, produce late, or produce something approximate. For data systems the answer is almost always 'nothing or late' — wrong data spreads downstream and is far more expensive than a delay. Idempotent retries, checkpointed state and staged-then-promoted writes are how I make that choice enforceable rather than aspirational.

A green pipeline run means code executed, not that the data is right. Quality has to be asserted: schema contracts at boundaries, referential and volumetric invariants, distribution checks against an expected profile, and business assertions written by the people who understand the domain. Then those results have to be stored, so quality is a trend you can look at rather than an alert you missed.

Job status is the least interesting signal a data platform emits. Freshness, row volume against expectation, distribution drift, and lag on the streaming path tell you something is wrong before a dashboard does. Lineage matters just as much: when a number is questioned, the time it takes to answer 'where did this come from' is a property of the platform's design.

Cost is decided at design time by storage layout, partitioning, file sizing, and how much data a query is forced to read — not by tuning after the invoice arrives. Streaming everything because streaming is interesting is a cost decision. So is keeping raw history forever. I would rather make those decisions explicitly and write down why.

A system only one person can run is a liability with good uptime. Configuration over bespoke code, one obvious way to add a source, runbooks for the failure modes that actually occur, and naming that matches how the business talks. The test is whether someone else can debug it on a Sunday with only what is written down.

Every pipeline has an owner, a consumer and a decision it supports. If I cannot name all three, the right engineering decision is usually to not build it. The most valuable thing I have deleted was a beautifully engineered dataset nobody read.