<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
     xmlns:atom="http://www.w3.org/2005/Atom"
     xmlns:dc="http://purl.org/dc/elements/1.1/"
     xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Shortland · Writing</title>
    <link>https://davidshortland.dev/writing</link>
    <description>Long-form writing on engineering practice, architecture, and the reasoning behind decisions.</description>
    <language>en-GB</language>
    <lastBuildDate>Mon, 31 Aug 2026 14:23:12 GMT</lastBuildDate>
    <atom:link href="https://davidshortland.dev/rss.xml" rel="self" type="application/rss+xml" />
  <item>
    <title>The Last Published Date</title>
    <link>https://davidshortland.dev/writing/the-last-published-date</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/the-last-published-date</guid>
    <pubDate>Wed, 26 Aug 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Two client websites built alongside each other, with opposite architectures, decided by one piece of evidence that was free, public, and sitting under a date stamp before either proposal was written.</description>
    <content:encoded><![CDATA[<p>The last post on newvisionpackaging.co.uk was dated September 2022. I found it by scrolling to the bottom of the blog before I wrote the proposal, which took about fifteen seconds and told me more about what to build than anything else I did that week.</p>
<p>The site was WordPress on shared hosting and cost around £800 a year. Twenty-eight pages, fifty-five blog posts, a contact form plugin, a cookie banner, an Instagram feed, a spam filter. A good share of that money bought the machinery of publishing: a database, an admin area, a theme, plugins to keep patched, and a login my dad had and never used. Four years of paying for a press that printed nothing.</p>
<p>The rest had aged the way an unattended site does. The footer read "© 2020". The header call to action truncated to "EMAIL SALES@" at the width most phones render it. The contact page still listed a colleague who'd left. The Instagram link pointed at a numeric plugin ID rather than a profile. None of that is WordPress's fault. It's what happens to any site nobody opens, and that those defects had survived four years said the same thing the blog date did.</p>
<p>Both clients in this post are my dad. That's the obvious objection to everything below and I'll come back to it at the end rather than pretend it isn't there.</p>
<p>There are project pages for both, with the figures and the stacks: <a href="https://davidshortland.dev/work/new-vision-packaging">New Vision Packaging</a> and <a href="https://davidshortland.dev/work/corby-united">Corby United</a>.</p>
<h2 id="the-cheap-reading">The Cheap Reading</h2>
<p>The easy lesson is that WordPress is heavy and static hosting is cheap, so swap one for the other and pocket the difference. I did swap it. The rebuild is prerendered Angular on S3 behind CloudFront, it costs about £20 a year, and there's no database, no runtime, no admin panel and nothing to patch.</p>
<p>If that were the whole lesson I'd have got my other project wrong, because I built a second site alongside it and gave it a CMS, a server-rendered Lambda, a webhook and a deployment pipeline. Same developer, same stretch of months, opposite answers. What differed was not the technology on offer. It was what I knew about who would touch each site after I stopped.</p>
<h2 id="what-the-date-is-evidence-of">What the Date Is Evidence Of</h2>
<p>A blog with a four-year-old last post is a prediction. Whatever a client believes about publishing, the observed rate is zero and has been zero long enough that the sample means something. Nobody has to tell you this and nobody has an incentive to volunteer it, because it's sitting on the public internet under a date stamp.</p>
<p>So I asked, in the proposal, what he wanted the site to do over the next few years. He said he wanted something that could sit there and only be updated when it absolutely had to be. He wasn't apologising for the four years. He was describing them as the plan.</p>
<p>That reframes the job. The publishing system was not an asset he'd been underusing. It was the recurring cost of a capability he had decided not to use, plus a surface that needed patching whether he used it or not, and the correct move was to take it away. That's an awkward thing to put in a proposal, because it reads as removing a feature and charging for the privilege.</p>
<p>So the rebuild has no CMS. Content lives as TypeScript files in the repo, changing a product description means editing a file and pushing, and the pipeline builds and deploys it. For anyone publishing weekly that's a downgrade. For someone publishing twice a year it means there's no admin surface to attack, no plugin to update, and no quietly rotting installation still charging rent. The old URLs all still resolve, through eighty-one redirects, because a site with fifty-five posts and four years of indexing has links pointing at it from places neither of us knows about.</p>
<h2 id="the-other-site">The Other Site</h2>
<p>Corby United is a walking football club, which is football for people whose knees have opinions. Dad runs that too. There was no old site to inspect, so the last-published date had nothing to say and the evidence had to come from him instead.</p>
<p>He wanted match reports, photographs from Tuesday mornings, news about fixtures. He'd already been sending that material round on WhatsApp for a year, which is the same species of evidence as a blog date and points the other way.</p>
<p>So Corby got the heavier stack, and each piece of it is there because someone publishes. Sanity Studio, so he can write without me. Server-side rendering on a Lambda behind CloudFront, so a post is live the moment he hits publish rather than after a rebuild. A webhook that invalidates the CDN on publish, with a five-minute cache expiry behind it for the times the webhook doesn't fire. A self-mutating CDK pipeline, so deploying isn't something I do by hand at nine at night.</p>
<p>Every one of those is dead weight on a site that publishes twice a year. On New Vision Packaging they would have been an elaborate way of making a brochure harder to change.</p>
<h2 id="what-came-out-of-both">What Came Out of Both</h2>
<p>The other thing the two projects have in common is what I deleted.</p>
<p>Corby started with a .NET backend. Its last remaining job was scoring a questionnaire that helps someone work out whether the club suits them, and I removed the service once I noticed the questions, their point values and the outcome conditions were all public content the page already downloaded. A server that protects nothing is a server you pay to run and patch. The scoring is a few dozen lines of TypeScript in the browser now.</p>
<p>Then the contact form went, and took SES, a reCAPTCHA integration, a Secrets Manager secret and the questionnaire's lead capture with it. Nothing in that stack sends email any more. The pages show a phone number and an address, which is what a 68-year-old thinking about trying walking football was going to use anyway.</p>
<p>New Vision Packaging lost <code>/work</code>, <code>/capabilities</code> and <code>/services</code>, three sections that existed largely because the old theme had slots for them. Every third-party embed went with them. Google Fonts became self-hosted woff2, the Maps embed became a static image linking out, and the Instagram widget became a text link. The site sets no cookies, so it needs no consent banner, so the broken banner that used to render as a narrow column across the hero is gone by deletion rather than by repair.</p>
<p>Both sites are smaller than their briefs implied, and that's the part I'd defend hardest if either client asked what they paid for.</p>
<h2 id="the-bit-i-can-t-prove-yet">The Bit I Can't Prove Yet</h2>
<p>Back to the objection. Both clients are my dad, so I had access no stranger and no agency would get, and I could ask follow-up questions over dinner.</p>
<p>I do not think that rescues the argument, because the evidence that decided the New Vision Packaging design was public. The last-published date on a prospective client's blog is visible to anyone before the first meeting, and so is a copyright notice four years stale. Neither needs a relationship. What the relationship bought was the confidence to act on it early, and to write a proposal whose main recommendation was subtraction.</p>
<p>What I can't prove yet is that I read either client correctly. Both sites went live this month, so here's the prediction while it still costs something to make.</p>
<p>I expect dad to publish to Corby at least twice a month for the first three months and then settle at roughly monthly. I expect New Vision Packaging to need no content change at all before November, and I expect the first request, whenever it lands, to be a new product photograph rather than anything structural. I expect neither site to cost more than £2 a month to run.</p>
<p>If the Corby posts stop after three weeks, I built the wrong thing, and the CMS and the SSR Lambda and the webhook were a stack I picked because I wanted to build it. If the packaging site turns out to need a change the static build can't absorb, I've made his site harder to run than the one I replaced, and £780 a year of saving won't cover that.</p>
<p>Neither site has analytics and neither is getting any. The check is a maintenance log, a hosting bill, and one email in November asking whether the phone rang. I'll post the answer either way.</p>]]></content:encoded>
    <category>architecture</category>
    <category>aws</category>
    <category>angular</category>
    <category>clients</category>
    <category>maintenance</category>
  </item>
  <item>
    <title>Building LapDelta: A Real-Time F1 Replay for a Pound a Month</title>
    <link>https://davidshortland.dev/writing/building-lapdelta</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/building-lapdelta</guid>
    <pubDate>Wed, 05 Aug 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>A 60fps replay of real Formula 1 telemetry that runs for about a pound a month with no request-serving backend, built on a zero-copy data contract shared across three languages and a fair amount of geometry.</description>
    <content:encoded><![CDATA[<p>Playing back live-looking telemetry for twenty cars at sixty frames a second sounds like a job for a streaming backend, a service that holds session state, interpolates positions on request and pushes frames down a socket. That's the obvious shape and it's the wrong one.</p>
<p>LapDelta has no request-serving backend at all. It's an animated, to-scale track map with synchronised telemetry traces and a session-aware timing tower, replaying real Formula 1 sessions in the browser, live at <a href="https://lapdelta.app" target="_blank" rel="noopener">lapdelta.app</a>. The entire running system is an offline build that produces one file per session, a CloudFront distribution serving static bytes, and a browser client that does all the work. It costs roughly a pound a month to run.</p>
<p>I should be plain about what this is and isn't. There are excellent, mature F1 data products out there, built by real teams with real budgets, that do far more than this and do it better. LapDelta doesn't compete with any of them and this isn't the story of a better one. It's a fan project whose one genuinely interesting property is economic: it does a credible version of the thing, real telemetry, twenty cars, sixty frames a second, a timing tower you can trust, on almost no budget, and it stays fast while doing it. That constraint shaped almost every decision below, far more than the feature list did.</p>
<p>This post is about how that's possible, and about the part I enjoyed most along the way: the maths that turns noisy positional data into a track you can recognise and an order you can trust. Some of that maths falls out of the cost constraint and some of it is just the geometry the problem happens to need. I'll try to be honest about which is which.</p>
<p>There are animated plates of the circuit, and of a selection of the other tracks, on the <a href="https://davidshortland.dev/work/lapdelta">project page</a>.</p>
<p>(LapDelta is an unofficial fan project. It is not affiliated with, endorsed by, or connected to Formula 1, the FIA, or any team, it uses no official marks, and it is non-commercial.)</p>
<h2 id="start-with-the-constraint">Start With the Constraint</h2>
<p>I wanted the project to be cheap to run when nobody is watching, simple to operate, and genuinely fast under load. Those three constraints rule out the streaming backend immediately, because a service that interpolates frames per client is a service you pay for per client, babysit, and scale.</p>
<p>So the constraint forces an inversion. With no server to do the compute, the compute moves to the two places that are already free: some of it happens once, offline, when a session is ingested, and the rest happens in the browser on the viewer's own machine. What travels between them is a single precomputed artifact.</p>
<p>That gives three zones with hard boundaries:</p>
<figure><img src="https://davidshortland.dev/writing/images/lapdelta-architecture.png" alt="LapDelta architecture: offline build, static serve, browser client" width="800" height="300" loading="lazy" /><figcaption>LapDelta architecture: offline build, static serve, browser client</figcaption></figure>
<p>An offline build in Python ingests a session and writes one artifact. A dumb static layer, S3 behind CloudFront, serves that artifact as immutable, brotli-compressed bytes. A clever client, Angular with a Rust core compiled to WebAssembly, downloads the artifact once and renders everything from it. There's no <code>/api</code>, no database in the request path, and no request path.</p>
<h2 id="one-contract-three-languages">One Contract, Three Languages</h2>
<p>The artifact is the whole interface between the zones, so its format matters more than any single piece of code. It's a <a href="https://flatbuffers.dev" target="_blank" rel="noopener">FlatBuffers</a> schema called <code>SessionArtifact</code>, and it's the published language of the system: Python writes it, Rust and TypeScript read it.</p>
<p>FlatBuffers earns its place here because it's zero-copy. The browser downloads the bytes, hands the buffer to the WASM module, and Rust reads fields straight out of that memory with no parse step. On the sixty-frames-a-second render path that's literal, and sampling every car's state is a handful of reads straight off the buffer, allocating nothing. The one exception is the one-time build of the running-order model, which does decode every sample into owned arrays up front, a genuine pass over the data that happens once at load rather than per frame. For a file holding tens of thousands of telemetry samples across twenty cars, "no parse step" is the difference between a snappy load and a stuttering one.</p>
<p>This is the one decision in the project I'd call a clean technical tradeoff rather than a preference. Protocol Buffers was the obvious alternative and for most jobs I'd reach for it first, but protobuf has to decode the wire format into language objects before you can read a single field, allocating as it goes, and for a buffer this size on the browser's main thread that decode is exactly the cost I wanted to avoid. FlatBuffers stores its data in a layout you can read in place. The price of reading a field is a pointer offset. I gave up protobuf's friendlier ergonomics and smaller wire size to buy that.</p>
<p>It also collapses three sources of truth into one. There's exactly one definition of what a session is, and the Python writer and the Rust and TypeScript readers are all generated from it, so a field can't mean one thing on the way out and another on the way in, because there's only one field. Anywhere the same concept is defined in several places, the copies eventually drift. A code generator makes that drift impossible here.</p>
<h2 id="the-stack-and-why">The Stack, and Why</h2>
<p>The serialisation format was a technical decision. The languages, mostly, weren't, and it's worth being honest about which is which.</p>
<p><strong>Python, because the data lives there.</strong> The ingestion is Python because <a href="https://docs.fastf1.dev" target="_blank" rel="noopener">FastF1</a>, the library that exposes historical Formula 1 timing and telemetry, is Python. It's also free and open. That's the other half of the cost story: the input side of the pipeline costs nothing because the data source costs nothing. The data arrives already in the shape I need, and writing the ingestion in anything else would have meant reimplementing FastF1 or shelling out to it for no benefit. Use the language your data already speaks.</p>
<p><strong>Rust compiled to WASM, half on purpose and half to learn it.</strong> The compute core has a genuine reason to be Rust. WebAssembly is a first-class target, the tooling is mature with <code>wasm-bindgen</code> and <code>wasm-pack</code>, and there's no garbage collector or runtime to ship, so the module stays small and its performance is predictable. A sixty-frames-a-second hot path wants exactly that. It's also a language I'm deliberately learning, and a self-contained core with a narrow interface, bytes in and frames out, is a forgiving place to do that, because the blast radius of getting it wrong is one module sitting behind a contract.</p>
<p><strong>Angular, because it's what I reach for.</strong> I'm most productive in Angular, it's my daily driver for single-page apps served from CloudFront, and the goal was to spend my thinking on the hard parts, the geometry and the contract and the render loop, rather than on relearning a framework. Comfort is a legitimate input when the framework isn't the interesting problem.</p>
<p>That's the honest throughline. Pick the tool that removes friction everywhere the tool isn't the point, and spend the saved effort where it is.</p>
<h2 id="the-maths-part-one-building-a-track-from-noise">The Maths, Part One: Building a Track From Noise</h2>
<figure><img src="https://davidshortland.dev/writing/images/lapdelta-math-pipeline.png" alt="The maths pipeline: from raw GPS to running order, split across the offline build and the browser" width="800" height="220" loading="lazy" /><figcaption>The maths pipeline: from raw GPS to running order, split across the offline build and the browser</figcaption></figure>
<p>Raw F1 positional data is a stream of <code>(x, y, time)</code> samples in some circuit-specific coordinate system. It isn't a track. It's a smudge of points, rotated arbitrarily, at whatever scale the source happened to use, sampled unevenly, with the occasional wild outlier where the feed dropped and interpolated across a gap.</p>
<p>Turning that into a clean, consistently oriented track map is the first interesting bit of geometry and it happens once, offline.</p>
<p>First, resample a reference lap to a fixed number of evenly spaced points by arc length. Raw samples cluster on the straights, where high speed at the same sample rate means more metres per sample, and thin out in the corners, which is exactly backwards for drawing a smooth outline. Walking the polyline by cumulative chord length and interpolating at even intervals fixes that:</p>
<pre><code class="language-python">deltas = np.diff(closed, axis=0)
seg_lengths = np.hypot(deltas[:, 0], deltas[:, 1])
cumulative = np.concatenate([[0.0], np.cumsum(seg_lengths)])
total = cumulative[-1]
targets = np.linspace(0.0, total, count, endpoint=False)
x = np.interp(targets, cumulative, closed[:, 0])
y = np.interp(targets, cumulative, closed[:, 1])</code></pre>
<p>Then align it. Different circuits arrive at arbitrary rotations, and I want every track to land in the same orientation and the same normalised box so the renderer never has to special-case one. The trick is principal component analysis: take the scatter matrix of the centred points, which is the covariance without the 1/(n−1) and has the same eigenvectors, then take its eigenvectors, and the one with the largest eigenvalue is the track's longest axis, so rotating that onto the horizontal gives a repeatable orientation:</p>
<pre><code class="language-python">covariance = centered.T @ centered
eigenvalues, eigenvectors = np.linalg.eigh(covariance)
major = eigenvectors[:, int(np.argmax(eigenvalues))]
angle = np.arctan2(major[1], major[0])</code></pre>
<p>A final translate-and-scale into a unit-ish bounding box, and every circuit in the calendar comes out centred, level and drawn to the same scale, with the renderer doing nothing more than drawing a polyline. Two honest caveats. The major axis is only defined up to a sign, so this pins the rotation but not a possible 180-degree flip, and a near-square circuit whose two axes are close in length has no stable long axis to align in the first place. Both are cosmetic. The running order comes from arc length, not from which way the map happens to point.</p>
<h2 id="the-maths-part-two-making-it-small">The Maths, Part Two: Making It Small</h2>
<p>The artifact has to be small enough to download fast and sit in memory comfortably, and telemetry is naturally floating point, with speed, throttle, brake and position all arriving as <code>f64</code> from the source. Storing it raw would more than quadruple the file for precision nobody can see at sixty frames a second.</p>
<p>So each channel is quantised. Position and speed go to <code>i16</code> with a per-channel scale and offset chosen to fit the channel's actual range symmetrically into the integer space:</p>
<pre><code class="language-python">offset = (highest + lowest) / 2.0
scale = (span / 2.0) / _I16_HALF_RANGE
raw = np.round((values - offset) / scale)
raw = np.clip(raw, -_I16_HALF_RANGE, _I16_HALF_RANGE).astype(np.int16)</code></pre>
<p>Decoding is the inverse, <code>raw * scale + offset</code>. Because the scale comes from the channel's own range, the reconstruction error is bounded to half a quantum of that range. Invisible on screen, and a quarter of the bytes. Bounded channels that don't need the resolution go further: throttle and brake to a <code>u8</code> percentage, gear clamped to 0 to 8, DRS to a single bit's worth of <code>u8</code>. The schema carries the scale and offset alongside the values, so the reader needs no out-of-band knowledge to decode.</p>
<h2 id="the-maths-part-three-sixty-frames-a-second-in-the-browser">The Maths, Part Three: Sixty Frames a Second in the Browser</h2>
<p>Now the client side, in Rust compiled to WASM. The playback engine runs one clock, and on every tick it asks the core for the state of every car at a given millisecond, and the core has to answer fast enough to hit the frame budget for twenty cars at once.</p>
<p>The samples are timestamped but not on a fixed grid, so "the state at time <code>t</code>" almost never lands exactly on a stored sample. It's a binary search for the bracketing samples followed by a linear interpolation between them:</p>
<pre><code class="language-rust">let lower = last_at_or_before(&amp;times, time_ms);
let (upper, fraction) = if lower + 1 &gt;= count {
    (lower, 0.0)
} else {
    let span = times.get(lower + 1) - times.get(lower);
    if span &gt; GAP_MS {
        return None;
    }
    (lower + 1, (time_ms - times.get(lower)) as f32 / span as f32)
};</code></pre>
<p>The <code>GAP_MS</code> guard is the detail that matters. Real feeds drop out, and if two consecutive samples are more than two seconds apart, interpolating between them would slide a car smoothly across a chunk of track it was never on. So beyond that threshold the core refuses to invent data and reports the car as absent rather than lie convincingly. The interpolation itself dequantises both endpoints and lerps between them in one step, so the integer packing from the build stage gets paid back transparently here.</p>
<h2 id="the-maths-part-four-who-is-actually-winning">The Maths, Part Four: Who Is Actually Winning</h2>
<p>The hardest question turned out to be the simplest to ask. What is the running order? The source data doesn't hand you a clean position-per-car-per-instant you can trust frame by frame, so the order has to be derived from where each car physically is on the track.</p>
<p>That's a projection problem. For each car's <code>(x, y)</code>, find the nearest point on the track polyline, and the cumulative arc length up to that point is how far around the lap the car is. Nearest-point-on-a-segment is a clamped dot product per edge:</p>
<pre><code class="language-rust">let t = if length_squared &lt;= 1e-12 {
    0.0
} else {
    (((point.0 - a.0) * edge_x + (point.1 - a.1) * edge_y) / length_squared).clamp(0.0, 1.0)
};</code></pre>
<p>Run that against the track's cumulative arc-length table, accumulate a lap each time the car crosses the start line, and a noisy cloud of coordinates becomes a single, mostly-monotonic number: total race distance. Mostly, because projection noise can nudge it backward between frames, so the model tolerates small reversals rather than assuming they can't happen. Sort the cars by it and you have the order. A search window keeps the projection local, since a car this frame is near where it was last frame and there's no need to scan the whole circuit each time, the same two-second gap rule prevents a dropout from teleporting a car forward a lap, and once the winner takes the flag the order locks to the official classification so the final result is exact rather than reconstructed. The pretty track map and the timing tower are, underneath, the same arc-length model viewed two different ways.</p>
<p>The obvious objection is why derive the order from positions at all, rather than reading it from timing. Because the timing the source gives you isn't continuous. The structured timing comes per lap: lap start, lap time, sector times, pit in and out, plus a final classification per session. There's no gap-to-the-car-ahead you can resolve at an arbitrary millisecond. A replay you can pause and scrub needs the order as a continuous function of time, and it needs that order to agree with the cars drawn on the map. Position telemetry is the only signal dense and continuous enough. Derive the order from it and the timing tower and the dots on the track can never disagree, because they come from the same samples. The cost is honest. Mid-race gaps are estimates, a fraction of a lap times a representative lap time, and the exact gaps only appear at the finish where the order locks to the official classification. Timing data still does the work positions can't, pinning each car's absolute lap number so a car a lap down sorts correctly even when it's physically ahead on track.</p>
<h2 id="the-bug-that-made-the-phone-go-dead">The Bug That Made the Phone Go Dead</h2>
<p>Everything above worked on a laptop long before it worked on a phone. On iOS you could play a session and watch it run perfectly, and taps on the telemetry just died. No response, no error, nothing.</p>
<p>The cause wasn't in any of the maths. An Angular effect was rebuilding the telemetry chart on every frame, and at sixty rebuilds a second it was starving the main thread so completely that iOS never got the idle moment it needs to promote a touch into a click. The fix was one word, wrapping the chart rebuild in <code>untracked()</code> so the playback clock stops triggering a reconstruction it was never meant to trigger. The render loop kept running, the main thread got its breath back, and taps came back to life.</p>
<p>It's the most ordinary kind of performance bug, the cost of a thing you're doing far more often than you need to, and a useful reminder that on the client the frame budget is a shared resource and the renderer doesn't own all of it.</p>
<h2 id="what-it-reveals">What It Reveals</h2>
<p>The thing I'd want a reader to take from this isn't the F1 data. It's the shape.</p>
<p>Starting from constraints, cheap and simple and fast, rather than from a stack, forced an architecture where the expensive work happens exactly once and the per-viewer work happens on the viewer's own hardware. A single generated contract made it safe to split that work across Python, Rust and TypeScript without the three drifting apart. And a surprising amount of what looks like product, the recognisable track and the smooth motion and the trustworthy order, is geometry done carefully at the right stage of the pipeline.</p>
<p>The running cost is about a pound a month. There's no server to patch, no scaling to plan and no request path to fall over. The compute didn't disappear; it moved to where it was free, and the contract is what let it move safely.</p>
<p>The mature tools I mentioned at the start got a lot of their polish from having a budget. This got its shape from not having one. That isn't a complaint and it isn't a humblebrag either. It's just the most interesting thing about the project, and the reason it was worth writing down.</p>]]></content:encoded>
    <category>rust</category>
    <category>wasm</category>
    <category>flatbuffers</category>
    <category>python</category>
    <category>angular</category>
    <category>architecture</category>
  </item>
  <item>
    <title>The Second Reader</title>
    <link>https://davidshortland.dev/writing/the-second-reader</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/the-second-reader</guid>
    <pubDate>Wed, 24 Jun 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>University taught me every practice that defines good engineering, then removed the one condition under which any of them make sense. What&apos;s left over is a lesson about knowledge itself.</description>
    <content:encoded><![CDATA[<p>For two years I lived in two versions of software at once.</p>
<p>By day I was an engineer on systems that had to keep running, writing code other people would read, depend on, and change long after I'd forgotten why I wrote it. By night I was a computer science student, handing in code that would be read exactly once, by one person, and then never run again.</p>
<p>I did well at the degree. I say that not as a boast but because it's the load-bearing fact of what follows: the marks were real, they measured something real, and what they measured was far narrower than it looked. Topping a module never answered the question the two years kept putting to me. It sharpened it. The same work, judged by the standard I used at my desk during the day, would not always have passed.</p>
<h2 id="the-reader-who-never-comes">The Reader Who Never Comes</h2>
<p>Start with the difference that turns out to explain everything else.</p>
<p>A university assignment is read once, by a marker, who never has to change it. They open it, judge it against a rubric, record a number and move on. The code's life ends at the moment it is understood.</p>
<p>Production code has a second reader. Someone comes later, a colleague or you in eight months, holding less context than you hold right now, and has to understand what you left well enough to change it without breaking it. Almost everything we call good engineering is, underneath, the discipline of writing for that person. Clear names, small units, decoupled parts, a test that explains its own intent: every one of them is a message to a reader who isn't in the room yet.</p>
<p>A degree has no second reader, so it never asks you to practise the one thing the job is mostly made of.</p>
<p>This isn't a complaint about lecturers or syllabi, and I want to be careful not to let it sound like one. It's structural. You cannot simulate a second reader inside an artefact that is handed in once and discarded, because the format forbids it. Whatever else a degree can do, and it can do a great deal, it can't make you write for someone who is never going to arrive.</p>
<h2 id="suspended-not-absent">Suspended, Not Absent</h2>
<p>The obvious objection is that this is plainly false. Universities <em>do</em> teach maintainability. They mark for decoupling, they want tests, they ask for documentation, and, we'll come to this, they ask for rather a lot of comments. All true. The practices are on the rubric.</p>
<p>But watch what happens to a practice when the code is never touched again. It arrives as a rule with nothing underneath it. You're told to separate concerns and you're never made to feel the afternoon you'd lose if you didn't. You learn <em>that</em> you should decouple, and you're never once burned by the coupling, because nothing you submit ever has to survive a change. The practice is present and the reason is missing.</p>
<p>I've written before about <a href="https://davidshortland.dev/writing/knowing-how-isnt-knowing-why">the gap between knowing how and knowing why</a>, about how a principle only sticks once you've felt the pain it answers and how before that it's just noise with good posture. A write-once degree is a machine for producing exactly that: knowing-how with the why suspended. Not wrong, not absent, but inert, a hook with nothing to catch on, because the experience that would fire it can't occur in a thing that's read once and thrown away.</p>
<p>And here's the part I'd ask any frustrated student to sit with, because it's the generous truth and the honest one at the same time. The same distance from consequence that stops a university teaching maintainability is exactly what lets it teach the theory. The deep, slow, immediately useless material, the proofs and the complexity and the foundations, is precisely what you'll never sit still for on the job, because the job rewards what pays off this week. A university's removal from consequence isn't only a weakness; it's the thing that buys the theory room to exist. Its great strength and its great blind spot are the same fact about it.</p>
<h2 id="what-nothing-checks">What Nothing Checks</h2>
<p>Comments are the cleanest illustration, so let me use them, because the obsession is real and the lesson hides inside it.</p>
<p>There's a distinction the rubric rarely draws. A comment that restates what the code does is noise, and worse than noise, because it rots. A comment that captures <em>why</em>, the intent, the constraint that isn't visible, the reason the obvious approach was wrong here, is irreplaceable, because code can only ever say what it does. It can never say what it was permitted to do, or what it deliberately doesn't.</p>
<p>The trouble is that nothing checks a comment. The compiler runs the code; nothing runs the prose beside it. Code has a kind of gravity, in that it has to work, so reality keeps tugging it back toward the truth. A comment is under no such force. It can say one thing while the code three lines below it does another, and nothing anywhere will complain. So comments drift, not through carelessness but through the absence of anything that would catch the drift.</p>
<p>In a write-once world that drift never has time to bite. You comment the code, it's graded, it dies. You never meet the comment that has quietly lied for two years until it sends someone confidently in the wrong direction at the worst possible moment. So you leave with the habit and none of the scar, and you carry it into production, where the scar is waiting.</p>
<p>Hold onto the shape of that, because it's the whole essay: <em>nothing checked the comment, so the comment drifted.</em></p>
<h2 id="the-thing-that-has-to-be-there">The Thing That Has to Be There</h2>
<figure><img src="https://davidshortland.dev/writing/images/what-checks-it.png" alt="What keeps a claim honest: code, design, and understanding each have something that tests them; a comment has nothing" width="800" height="470" loading="lazy" /><figcaption>What keeps a claim honest: code, design, and understanding each have something that tests them; a comment has nothing</figcaption></figure>
<p>Now stand back and look at code, comments and your own understanding side by side, and ask what keeps each of them honest.</p>
<p>The code is kept honest by the compiler and the tests, which run it, so a lie stops the build. The design is kept honest by the second reader and by years of change, which exercise it, so a bad decision surfaces as pain. Your understanding is kept honest by doing, by shipping, by maintaining, by being wrong somewhere it costs you and having to look at why. In every case the thing stays true only because something forces it to. Remove the check and it drifts, quietly, without anyone noticing. The noticing was the check's whole job.</p>
<p><strong>A claim that nothing ever tests drifts from the truth.</strong> That isn't a fact about university. It's a fact about knowledge. A belief you never expose to anything that could refute it isn't knowledge, it's a guess you've grown comfortable with. University is simply the purest specimen, because it removes every check at once, the future, the second reader, the test of running again, the cost of being wrong, and then prints a certificate saying the work is finished.</p>
<h2 id="not-behind-elsewhere">Not Behind, Elsewhere</h2>
<p>I should answer the fair rebuttal, because it's a good one. Isn't this just what being a beginner is? Don't junior engineers also follow rules before they feel them?</p>
<p>They do. The difference is that the graduate isn't behind on the same road; they were standing somewhere the road doesn't reach. The junior on a team is inside the loop from the first week. They write the coupling on Monday and watch it cascade on Friday, and the rule attaches itself to fresh, specific pain. The student's environment can't produce that pain at all, because the code is never touched again. It isn't a slower version of the same process. It's a different place, missing the part that does the teaching.</p>
<p>I want to be precise about my own footing, because it would be easy to turn this into a flattering story and it isn't one. I only saw the gap because I happened to be standing in both places at once. Most students aren't, and that's no failing of theirs. My classmates weren't worse engineers waiting to catch up; they were doing the same work without the half of the loop that I, by the luck of having a job, got to supply from somewhere else. Take the job away and I'd have learned exactly what they learned. The vantage was circumstantial. The lesson isn't.</p>
<h2 id="why-the-door-is-closing">Why the Door Is Closing</h2>
<p>For a long time none of this mattered very much, because the industry had a place to close the gap: the first job. You hired a graduate, and the early years were where the hooks got fired, through the ordinary repetitive work that teaches by repetition and consequence the things a degree structurally can't. The degree was never meant to produce a finished engineer. It produced foundations and the on-ramp did the rest.</p>
<p>Two things are wearing that on-ramp away at once. The repetitive work that used to be a junior's training ground is increasingly handed to machines, which means the at-bats that fired the hooks are vanishing. And tighter budgets have made firms less willing to pay for the stretch of time in which a graduate is becoming useful rather than being useful.</p>
<p>Here's the uncomfortable part, and I'd rather name it than smooth it over. The usual prescription is that universities should teach strong fundamentals and employers should recognise the degree for what it is. I believe the first half completely. The second half doesn't rescue the graduate; it more or less explains their predicament. If an employer correctly reads a degree as evidence of fundamentals and not of readiness, then in a market where the on-ramp is disappearing, the rational move is to hire fewer of the people whose hooks aren't fired yet. Recognising what the degree is doesn't open the door. It's part of why the door is closing.</p>
<p>So the responsibility moves onto the graduate. Not fairly, but really. The firing has to happen somewhere, and if the on-ramp won't supply it, you have to.</p>
<h2 id="build-the-loop">Build the Loop</h2>
<p>The good news is genuine: a loop is a thing you can build, and you don't need anyone's permission to start.</p>
<p>What you need is code that outlives the moment you wrote it. Something you keep running. Something you're forced to come back to and change, and live with long enough to be embarrassed by. Go back to your own work from six months ago and feel, specifically, what your past self cost you. That flinch is the whole education, and a degree is the one place engineered to spare you from it. Put your code in front of a second reader, or contribute to something that already has one. The artefact barely matters. The loop is the part the degree had to leave out, and it was always the part doing the teaching.</p>
<p>The lesson reaches past software and past university, which is why I think it's worth writing down. Wherever you need something to stay true, whether that's a comment, a design, a belief, or a claim you make about yourself, find the thing that checks it. Where you can't find one, assume the drift has already started, because it has.</p>
<p>The second reader is the person the whole craft is quietly written for, the one who comes after you holding less than you hold now, who finds out whether what you left behind was true. A degree is the one place you never have to meet them, and everything afterwards is learning to write for them anyway.</p>]]></content:encoded>
    <category>philosophy</category>
    <category>engineering</category>
    <category>learning</category>
    <category>career</category>
  </item>
  <item>
    <title>Let It Stabilise First</title>
    <link>https://davidshortland.dev/writing/let-it-stabilise-first</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/let-it-stabilise-first</guid>
    <pubDate>Sun, 07 Jun 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Don&apos;t commit to a structural contract before you know what you&apos;re agreeing to. On deferring irreversible decisions until real usage has revealed the domain.</description>
    <content:encoded><![CDATA[<blockquote><p>One of four essays drawn from my final-year dissertation on agile and lean delivery in operational start-up contexts. The project is a real system I built and ran; I can't name the client, so treat the figures as reported rather than as something you can go and check.</p></blockquote>
<p>Midway through building an API, I made a decision I initially read as a concession to time pressure. Several of the data shapes weren't settled yet. Stakeholders were reviewing the system regularly and finding things to adjust. I didn't want to be changing typed contracts every sprint, so I left certain endpoints returning dynamic types, with no enforced schema.</p>
<p>This felt like the wrong thing to do. The standard discipline says type your interfaces early: it catches contract violations at compile time, it documents your expectations for callers, it makes refactoring safe. Leaving things untyped is the kind of shortcut that creates technical debt.</p>
<p>About four months later, the API had a properly typed surface. The migration was clean because it happened after the shapes had settled under real operator use.</p>
<p>Looking back, I don't think the untyped period was a shortcut. It was the correct sequencing.</p>
<h2 id="the-cost-of-premature-contracts">The Cost of Premature Contracts</h2>
<p>A typed contract is a commitment. Once you define the shape of a request or response, you have stated what the API expects. Callers build against that statement. Changing it later means changing them too.</p>
<p>This is exactly why typed interfaces are valuable in stable, well-understood domains. The type system encodes domain knowledge, and it propagates corrections when that knowledge changes. The types are trustworthy because the domain is understood well enough to write them correctly.</p>
<p>But early in a system built against real operational workflows, you often don't have that. You have a first approximation. The stakeholders think they know what they need; the developer has a model of those needs; and both will be revised as soon as real usage begins. Requirements gathering and upfront design can get you close. What they cannot do is expose the edge cases, the workflow details, and the inconsistencies that only become visible when people use the system for real work.</p>
<p>If you commit to a typed contract based on your first approximation, you're locking in a design you know will be revised. Every revision means breaking the contract, notifying callers, and updating both ends, and the more structural the original contract the more each revision costs. What you pay for a premature contract isn't the original typing. It's every update after it, multiplied by the number of things that depended on the shape you guessed.</p>
<h2 id="what-stabilisation-looks-like">What Stabilisation Looks Like</h2>
<p>The useful question is not "should I type this?" but "do I know enough about this domain to commit to a shape?"</p>
<p>For several months in this project, the answer was no. Operators were discovering their own workflows as they used the system. Features that seemed well-defined in sprint planning became ambiguous when someone tried to use them for a real task. Response shapes that made sense at design time turned out to omit data that was needed in practice, or to structure data in a way that didn't match how the frontend needed to consume it.</p>
<p>The dynamic types weren't a retreat from discipline. They were an acknowledgment that the domain knowledge was still arriving.</p>
<p>When things did settle, the migration was straightforward. Each endpoint had been used enough times that I could look at the actual consumption patterns, understand what the system expected, and define a type that matched reality rather than a projection of what reality might be. The type I ended up with was different from what I would have written four months earlier: not dramatically, but in ways that would have made the earlier version wrong.</p>
<p>One outcome that surprised me: several query endpoints that handled freeform operator-defined metadata stayed dynamic by design even after the migration. The metadata was open-ended by nature. A typed shape would have been either too rigid, excluding legitimate values, or too permissive, accepting anything, which gives you the same behaviour with more syntax. Keeping those endpoints dynamic was itself a product of the stabilisation process: it showed that the right answer for that part of the API was never "type it later", it was "this doesn't have a stable shape, by design."</p>
<h2 id="the-lean-framing">The Lean Framing</h2>
<p>Lean software development has a principle about this: decide as late as responsibly possible. The point is to delay the irreversible decisions until you have the information needed to make them well, which is not the same as delaying everything.</p>
<p>A typed API contract is, in principle, reversible. You can migrate it. But migration has a cost, and that cost scales with how much has been built against the original contract. The later you revise it, the more expensive the revision. So "as late as responsibly possible" means: delay until the domain is understood, but not so long that the cost of revision has become prohibitive.</p>
<p>The right window is during active development, after a few real iterations with users but before the API surface has become a dependency for more than you can easily update.</p>
<h2 id="the-broader-application">The Broader Application</h2>
<p>The same logic applies beyond API contracts. It applies to database schemas, to the boundaries between services, to the naming conventions you encode in test fixtures, to any structural decision that will be expensive to change once other things depend on it.</p>
<p>The discipline is to identify the decisions that will be expensive to revise and make sure you have real usage data before you make them. If a decision is easy to revise, make it now and update it when you learn more. If it's hard to revise, wait until the domain has shown you enough to make it correctly, the way those query endpoints eventually showed me they were never going to hold a shape at all.</p>]]></content:encoded>
    <category>engineering</category>
    <category>architecture</category>
    <category>pragmatism</category>
    <category>refactoring</category>
  </item>
  <item>
    <title>Testing to the Risk, Not the Checklist</title>
    <link>https://davidshortland.dev/writing/testing-to-the-risk-not-the-checklist</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/testing-to-the-risk-not-the-checklist</guid>
    <pubDate>Wed, 03 Jun 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Deliberate exclusions are engineering decisions. Coverage isn&apos;t a number; it&apos;s a resource allocation problem, and the resource is finite.</description>
    <content:encoded><![CDATA[<blockquote><p>One of four essays drawn from my final-year dissertation on agile and lean delivery in operational start-up contexts. The project is a real system I built and ran; I can't name the client, so treat the figures as reported rather than as something you can go and check.</p></blockquote>
<p>On one project I worked on, three constraints decided the entire testing strategy before I wrote a single test: a small team where testing time competed directly with delivery time, no integration test environment for the third-party services the system depended on, and a small group of domain experts available for structured testing of the finished thing.</p>
<p>Tests are only valuable if they represent genuine attempts to disprove an assumption, which raises the question of which assumptions, and at which boundaries. A project has finite time and you cannot test everything at the boundary, so the decision about where to spend the effort is a resource allocation question rather than a testing one, and it needs a different analysis from "what are the most important features?"</p>
<h2 id="the-coverage-trap">The Coverage Trap</h2>
<p>Coverage as a primary metric creates a specific failure mode. You optimise for the number, so you write tests for the code that is easiest to test: utility functions, simple transformations, cases that are already well understood. The number goes up. The confidence it represents does not.</p>
<p>This is the inverse of what you want. The parts of a system that are easiest to test are usually the parts with the least risk. Pure functions with simple inputs and outputs behave predictably. The code that integrates with external systems, that depends on user behaviour, that sits at the junction of multiple domain concepts: that is where the risk lives. That code is harder to test, and the coverage metric gives you no pressure to go there. Test to a coverage target and you build a suite that protects you from the problems you already understood, while the problems you didn't understand stay invisible.</p>
<h2 id="a-constraint-driven-strategy">A Constraint-Driven Strategy</h2>
<p>From those three constraints, the strategy took three layers, each chosen for the specific kind of confidence it could provide.</p>
<p>The first layer was unit testing for the small number of pure, algorithmically substantive functions: code that took clearly defined inputs, applied non-trivial logic, and returned clearly defined outputs with no dependency on infrastructure or external state. These were the right target for automated unit tests because they were deterministic, isolated, and complex enough that a test told you something you didn't already know. The tests were written at the boundary conditions and failure modes rather than confirming the happy path: the Popperian principle applied directly.</p>
<p>The second layer was controlled-data testing for the integration points with external services. No sandbox environment existed for those services. Testing against them with live data was not safe. The approach instead was to wrap each integration behind an interface, build test fixtures from production-shaped data, and test the integration logic against those fixtures. This is not as strong a guarantee as a real integration test environment. It was the strongest guarantee available given the constraints, which is the relevant comparison.</p>
<p>The third layer was operator-led user acceptance testing. Domain experts ran scripted test cases against a production-shaped environment. This layer did what neither of the others could: it validated whether the system fit the actual workflows of the people who would use it, including the workflows that no written requirement had anticipated.</p>
<h2 id="deliberate-exclusions">Deliberate Exclusions</h2>
<p>Two layers were explicitly excluded: an automated front-end unit suite and an end-to-end browser-driver suite.</p>
<p>Both were deliberate decisions, recorded with their justifications. The front-end unit suite would have tested HTML structure and component wiring rather than business logic, at a maintenance cost disproportionate to the risk it mitigated for an internal tool with a small, available user group. The browser-driver suite would have required infrastructure and ongoing maintenance that the risk profile didn't warrant given that the same operators who would notice regressions were available for direct testing.</p>
<p>The exclusions matter as much as the inclusions.</p>
<p>A testing strategy with only inclusions is a list. A real strategy records what you are not testing and why, because that's where the resource allocation decision becomes visible, and if you never record what you chose to leave uncovered you can't defend the allocation later or notice when the risk profile shifts enough to warrant revisiting it.</p>
<p>A deliberately excluded test is an engineering decision. An accidentally excluded test is a gap. The difference is whether you can explain it.</p>
<h2 id="sizing-to-the-risk">Sizing to the Risk</h2>
<p>What makes this a risk allocation problem rather than a testing problem is that risk is not uniform across a codebase. Some code failing causes a minor inconvenience. Some code failing causes incorrect results that flow into every downstream workflow, data loss, or a user-facing outage that takes a day to diagnose.</p>
<p>Testing effort should track that distribution, not the distribution of how easy the tests are to write.</p>
<p>The useful question before writing any test is: what is the cost if this code is wrong? If the answer is "a minor UI inconsistency that a user notices and reports," the test is probably not a good use of time. If the answer is "incorrect categorisation of data that every downstream report depends on," you want that test, and you want it designed to probe the conditions where the logic is most likely to fail, not to confirm the case you already know works.</p>
<p>Coverage is a side effect of writing that kind of test. It is not the goal.</p>
<p>The goal is a suite where every test encodes a genuine risk: something that, if broken, would cause a real problem. A suite like that does not need to be large to be valuable. It needs to be honest about what it is and is not covering, and why.</p>]]></content:encoded>
    <category>engineering</category>
    <category>testing</category>
    <category>quality</category>
    <category>agile</category>
  </item>
  <item>
    <title>The Sprint Review Isn&apos;t the Feedback Loop</title>
    <link>https://davidshortland.dev/writing/the-sprint-review-isnt-the-feedback-loop</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/the-sprint-review-isnt-the-feedback-loop</guid>
    <pubDate>Sun, 31 May 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Formal ceremonies create the conditions for feedback. The actual signal comes from elsewhere. Sprint reviews tell you what people decided to say; informal corrections tell you what they actually noticed.</description>
    <content:encoded><![CDATA[<blockquote><p>One of four essays drawn from my final-year dissertation on agile and lean delivery in operational start-up contexts. The project is a real system I built and ran; I can't name the client, so treat the figures as reported rather than as something you can go and check.</p></blockquote>
<p>Every sprint in a project I worked on ended with a stakeholder review. I would demonstrate the features shipped that sprint. The stakeholders would confirm they worked. Occasionally someone asked a question. The session would end and I would plan the next sprint based on what had been confirmed.</p>
<p>The sprint reviews were not where the useful feedback came from.</p>
<h2 id="the-pattern-i-noticed">The Pattern I Noticed</h2>
<p>About halfway through the project, I looked back at the changes that had most substantially improved the system. Almost none of them had originated in a sprint review. They came from a message a few days after the review. A passing comment when someone was using the system for something real. A note raised mid-sprint because an operator had tried to do something and found the system didn't support it the way they expected.</p>
<p>Three changes in particular stand out. An edge case that nobody had documented during requirements gathering, surfaced informally when an operator encountered it while testing against real data. A workflow redesign that came from a brief message explaining that the way I had structured a particular interaction didn't match how the operator actually thought about the task. A shortcut feature suggested in a casual aside that turned out to be one of the things operators valued most in the finished system.</p>
<p>None of these came through the structured sprint review. All of them came through channels with lower stakes.</p>
<p>I've had to describe those three at a level of abstraction that this essay's own argument says is the weak kind of account: tidied up, generalised, stripped of the specifics that would let you judge them. That's a cost of writing them up without naming the system, not a property of the finding, and it's worth saying plainly rather than hoping you don't notice.</p>
<h2 id="why-sprint-reviews-produce-the-wrong-signal">Why Sprint Reviews Produce the Wrong Signal</h2>
<p>The sprint review is not a natural environment for surfacing uncertainty. It's a presentation: the developer has prepared something to show, the stakeholder is there to evaluate it, and the implicit social contract is that this is a checkpoint, a moment of confirmation or redirection rather than an open exploration.</p>
<p>This shapes what people say.</p>
<p>A stakeholder in a formal review will raise concerns they have thought through: concerns they can articulate clearly, concerns that feel substantial enough to justify interrupting a structured session. They will not, generally, raise a vague unease about a workflow they haven't fully thought through yet. They will not flag a minor inconsistency they noticed while doing something else. They will not surface the thing that is a bit wrong but they are not sure how to describe.</p>
<p>These are exactly the signals that matter most in early-stage product development: the things people notice but don't say in a formal setting, the half-formed observations that haven't yet resolved into a proper complaint, the edge cases that emerge from use rather than from hypothetical review. A formal review tells you what people decided to say. The informal correction tells you what they noticed.</p>
<h2 id="why-the-low-stakes-channel-works">Why the Low-Stakes Channel Works</h2>
<p>The asynchronous message works because it removes the performance element from the exchange. The stakeholder isn't in a meeting, nobody is waiting on their response, and there's no implicit pressure to have thought it through. They can say "this is probably nothing, but I noticed that..." and the message is complete. That conversational register is not available in a formal review.</p>
<p>It also decouples the timing of feedback from the review cadence. Sprint reviews happen every two or three weeks. An operator who hits a problem on a Tuesday doesn't file it away for the Thursday review, they send a message, and the feedback arrives while they can still describe exactly what they were doing and what felt wrong.</p>
<p>The three changes above were precisely the kind of correction that a specification-first approach would have missed. They arose from real use, in real contexts, often at the edges of the documented requirements. No amount of upfront requirements-gathering would have surfaced them. They emerged from iteration, and the iteration was made useful by the informal channel, not the formal one.</p>
<h2 id="what-the-sprint-review-is-for">What the Sprint Review Is For</h2>
<p>This is not an argument against sprint reviews. They serve a real function: demonstrating that committed work was delivered, confirming that the direction is correct at a structural level, and giving stakeholders a regular touchpoint for higher-level planning.</p>
<p>What they don't do is substitute for the informal channel. The sprint review is the formal record; the asynchronous message is where the feedback loop actually runs.</p>
<p>Running one without the other leaves you with either undocumented direction (informal only) or a false confidence that the sprint review captures everything worth capturing (formal only). The formal ceremony creates the conditions in which stakeholders feel engaged. That engagement is what makes them send the message at nine on a Wednesday that tells you the thing you need to change.</p>
<p>Optimising for ceremonies optimises for the signal people are willing to produce in a formal setting. Optimising for the informal channel optimises for the signal they produce when the cost of saying something half-formed is low, which is where the workflow redesign and the shortcut feature both came from.</p>]]></content:encoded>
    <category>agile</category>
    <category>engineering</category>
    <category>process</category>
    <category>stakeholders</category>
  </item>
  <item>
    <title>The Pipeline Is the Product</title>
    <link>https://davidshortland.dev/writing/the-pipeline-is-the-product</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/the-pipeline-is-the-product</guid>
    <pubDate>Wed, 27 May 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Spending a sprint on deployment infrastructure before writing any application code looks like deferring value. The evidence pointed the other way.</description>
    <content:encoded><![CDATA[<blockquote><p>One of four essays drawn from my final-year dissertation on agile and lean delivery in operational start-up contexts. The project is a real system I built and ran; I can't name the client, so treat the figures as reported rather than as something you can go and check.</p></blockquote>
<p>Early in a project I worked on, I spent an entire sprint building a deployment pipeline before writing a single line of application code.</p>
<p>The sprint produced nothing a stakeholder could click on. No screens. No features. No visible progress. What it produced was a working CI/CD pipeline: a mechanism that would take any future commit to the protected branch, build it, and deploy it to the production environment automatically, within about four minutes.</p>
<p>At the time, this felt like a straightforward application of continuous delivery principles. In retrospect, I had underestimated how completely it would shape everything that followed.</p>
<h2 id="the-conventional-reading">The Conventional Reading</h2>
<p>The standard argument against spending time on infrastructure before product is that it delays value delivery. The framing is usually opportunity cost: every sprint on pipeline work is a sprint not spent on features. There's a version of this argument that's difficult to dismiss. Early-stage projects should be wary of over-engineering their infrastructure. Building for scale before you have users is a well-documented failure mode.</p>
<p>But this reading misses something specific about what a pipeline enables.</p>
<h2 id="what-the-pipeline-does">What the Pipeline Does</h2>
<p>Without a deployment pipeline, every deployment is a manual operation. You build locally, move artefacts, watch logs, verify. That takes time, it introduces inconsistency between environments, and it's a context switch: you stop working on the problem to manage the mechanics of shipping the solution.</p>
<p>A working pipeline removes that cost permanently. From the moment it's in place, every commit flows to production automatically. The feedback cycle that iterative development depends on, the loop from change to evidence of change, compresses from hours to minutes.</p>
<p>In this project, 152 production deployments happened across the subsequent sprints. The median time from commit to a deployed change in the production environment was four minutes twelve seconds. The change failure rate across those deployments was 2.0 percent. Mean time to recovery from a failure was twenty-three minutes. Lead time, change failure rate and recovery time all sit inside DORA's elite band; deployment frequency, at roughly one a day, sits at the top of high rather than in it.</p>
<p>When I looked back at where that performance came from, the answer was uncomfortable in its simplicity. It traced to the pipeline decision more than to any discipline applied in later sprints. The pipeline was the precondition for everything that came after.</p>
<h2 id="the-inversion">The Inversion</h2>
<p>Here's what I'd have said before: the pipeline is infrastructure. Infrastructure enables product. Product creates value. Therefore the question is how quickly you can build the product, and "when should I build the pipeline?" has the answer "when the absence of one starts to create problems."</p>
<p>Here's what I'd say now: in the early sprints of a codebase you intend to operate in production, the pipeline is not infrastructure. It is the primary means by which you can demonstrate anything at all. The ability to commit a change and show it live to a stakeholder within five minutes is not a nice-to-have. It is the feedback loop that iterative development requires to function.</p>
<p>The second thing the pipeline does is change the risk profile of every subsequent decision. When deployment is a manual, infrequent event, you tend to batch changes and ship large updates. When deployment is automatic and takes four minutes, you ship small changes constantly. Small changes fail less, and when they do fail they're easy to diagnose and revert. The high-performing DORA numbers are not a consequence of later discipline; they are a direct consequence of making deployment so easy that frequent, small deployment becomes the path of least resistance.</p>
<h2 id="the-surprise">The Surprise</h2>
<p>What changed my thinking wasn't the metrics. It was a reflection partway through the project: every piece of measurable quality I could point to, the delivery cadence, the ability to show working software at every stakeholder review, the low change failure rate, traced back to a decision made before any user-facing feature was written.</p>
<p>Before this project I'd have read "spend a sprint on CI/CD before building features" as deferring value creation. What I found instead was that the pipeline was the value-creating artefact of those early sprints. Not because the pipeline itself was the deliverable, but because it made everything that followed a different kind of work: work where you could verify, demonstrate, and recover.</p>
<h2 id="the-qualification">The Qualification</h2>
<p>This is not a universal argument for infrastructure over product. There are contexts where the right move is to build something fast, validate it, and retrofit delivery automation later. A prototype is not a production system, and a proof of concept doesn't need DORA metrics.</p>
<p>The argument is narrower: in a project where you intend to operate the system in production, where you expect to iterate over months, and where stakeholder feedback will shape the design, the pipeline is not a supporting element. It is the asset that makes iteration meaningful.</p>
<p>The cost of building it in sprint one instead of sprint five is one sprint. The cost of building it in sprint five instead of sprint one is four sprints of deployment friction, inconsistent environments, and feedback cycles that are hours long instead of minutes. The arithmetic is straightforward, and the intuition points the other way, which is why the sprint felt like a waste for most of the time I was spending it.</p>]]></content:encoded>
    <category>engineering</category>
    <category>devops</category>
    <category>ci-cd</category>
    <category>delivery</category>
    <category>agile</category>
  </item>
  <item>
    <title>Make the Easy Thing the Right Thing</title>
    <link>https://davidshortland.dev/writing/make-the-easy-thing-the-right-thing</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/make-the-easy-thing-the-right-thing</guid>
    <pubDate>Wed, 20 May 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>When you start copying infrastructure code for the third time, you&apos;ve already waited too long. A note on extracting shared patterns into a library, and being kind to future-you.</description>
    <content:encoded><![CDATA[<blockquote><p>Note, August 2026. This opens by describing my own site as Angular SSR on Lambda, which was true when it was published. The site has since been rebuilt with every route prerendered to static HTML on S3 and no origin server at all. What follows about the shared library is unchanged, and one of the client sites still runs the SSR path it provides.</p></blockquote>
<p>A while ago I deployed my own site. The architecture was deliberate: Angular SSR running on Lambda behind CloudFront, a CDK pipeline that deploys itself, S3 for static assets, the works. It took longer than a static site would have, every decision had a reason, and the result was something I was happy to maintain.</p>
<p>Then I started building sites for people in my life, and I caught myself doing what software engineers always seem to do when they think nobody's watching. I copied files.</p>
<h2 id="the-first-sign">The First Sign</h2>
<p>For the second site I copied the Lambda configuration from my own. Same Web Adapter layer, same memory size, same env vars. Then the CloudFront distribution, then the Route 53 records, then the bucket deployments with their cache-control strings. By the time I had a working second deploy I had three hundred lines of nearly-identical CDK code sitting in two places.</p>
<p>It's tempting to feel productive about this. Look how fast the second one went, and how it already worked because the first one already worked. This is the kind of productivity that looks great today and becomes a liability tomorrow. The cost of copy-paste isn't the second instance. It's the moment the two start to drift.</p>
<p>A third site was on the horizon. If I copied again I'd end up with three subtly different versions of the same infrastructure, and every fix I made in one place would have to be remembered and re-applied in the other two. AWS keeps moving. A Lambda Function URL permission requirement changes here, a Web Adapter layer version bumps there. Three near-twins cost three times what one twin costs to maintain.</p>
<p>So I stopped and extracted.</p>
<h2 id="what-goes-into-the-library">What Goes Into the Library</h2>
<p>This is where the design lives. Get it wrong and you either build something nobody else can use, or something so flexible it doesn't save anyone any work.</p>
<p>What ended up shared:</p>
<ul>
<li>The full request flow: CloudFront, S3 origin with origin access control, Lambda Function URL origin with the IAM dance that makes it actually work in late 2025 after AWS changed how OAC permissions are granted, the viewer-request function that injects <code>x-forwarded-host</code>.</li>
<li>The cache strategy. Hashed assets get one-year immutable. Unhashed assets get a day with must-revalidate. SSR HTML defers to whatever the origin emits in <code>Cache-Control</code>.</li>
<li>The security headers. HSTS, framing, referrer policy, the Permissions-Policy header. Everything that should be the same on every site I run.</li>
<li>The Route 53 alias records and the apex-to-www redirect handled at the edge by a CloudFront Function.</li>
</ul>
<p>What stayed in the consumer:</p>
<ul>
<li>The Content Security Policy. Every site has different third parties. One needs an image CDN, another needs reCAPTCHA, another needs a newsletter embed. You can't predict a CSP in a library, so the library takes it as a string and trusts the caller to construct it.</li>
<li>Additional behaviours. One site has an API behind the same distribution on the <code>/api/*</code> path. The library accepts a map of extra behaviours and slots them in before the static catch-alls.</li>
<li>Additional domain aliases. One site has a future subdomain that needs to point at the same distribution.</li>
<li>The pipeline glue. Each project has its own source repos and its own build commands. The library exposes a helper for the post-deploy invalidation step and doesn't try to own the pipeline itself.</li>
</ul>
<p>The line I drew: anything that should be the same on every site goes in the library, and anything that's a property of the site, meaning what data sources it connects to, what other behaviours it has, what domains it serves, stays in the calling code.</p>
<p>The library has knobs. Not many. Enough that the variations between three real-world sites are expressible, and few enough that the configuration isn't harder to write than the inline code was.</p>
<h2 id="the-migration-was-different-from-the-greenfield">The Migration Was Different From the Greenfield</h2>
<p>The two new sites adopted the library straight away. There was nothing deployed to conflict with, so CFN created the resources cleanly with the library's natural logical IDs. The interesting case was the original, a site that was already live, with a Distribution under a CFN logical ID derived from where it sat in the construct tree at the time, and a cache policy with an account-global name attached to it.</p>
<p>CFN doesn't follow rename instructions. Change a logical ID and it sees a new resource being created and an old resource being deleted, and it does the create first. If the new resource has an account-global name like a CloudFront cache policy, or if the new distribution wants the same alias as the existing one, the create fails and the changeset rolls back.</p>
<p>That left me two options for the migration. I could pin the old logical IDs in the new code so CFN treated the library's resources as in-place updates rather than create-then-delete, or I could delete the existing stack first and let the next pipeline run recreate everything fresh.</p>
<p>I went with the second. The first approach worked, and it left a residue in the code, a permanent block of "this resource was named differently before we used the library." I'd rather take an hour of downtime on a personal site than carry that residue forever. The library-based code is now the same on every site, the migration was a one-time event, and the result is consistent.</p>
<p>This is a small principle I keep coming back to. One-time costs are cheap and permanent costs compound. Given the choice between an hour of downtime and a permanent block of migration code that future-me has to understand every time he touches this file, the hour is the better trade.</p>
<h2 id="the-reward">The Reward</h2>
<p>The boring, useful kind.</p>
<p>When AWS releases a new version of the Lambda Web Adapter, I bump the default in the library, run the build and publish a patch release, and every site picks it up on its next deploy. When I notice that the static behaviour list doesn't include <code>.webp</code>, I add it once and every site starts caching <code>.webp</code> correctly.</p>
<p>There's a smaller and more subtle reward too. I can reason about the system now. With three copies of inline CDK code, "is the cache policy the same on every site?" was a question I had to answer by reading three files and squinting. With one library the answer is yes, by definition. The shape of the system is in one place and the variations are explicit.</p>
<p>I don't believe in "always extract on the second use" or "always inline until the third". Rules like that are recipes. The actual question is whether the duplication is going to cost you: two copies of a five-line helper that's never going to change is fine, and two copies of three hundred lines of infrastructure code that talks to a moving cloud provider isn't.</p>
<p>The signal I trust is noticing myself fix something in one place and make a mental note to remember to fix it in the other. At that point the maintenance burden has overtaken the convenience of copy-paste, and the longer I wait the more painful the extraction gets.</p>
<p>You're not building a public library when you do this, and you're not optimising for hypothetical users. You're being kind to future-you, who's going to want to make a small change six months from now and would prefer to make it in one place. That's the entire goal. Make the easy thing the right thing.</p>]]></content:encoded>
    <category>cdk</category>
    <category>aws</category>
    <category>architecture</category>
    <category>refactoring</category>
    <category>engineering</category>
  </item>
  <item>
    <title>Building a Mathematics Interpreter in F#: From Parser to Symbolic Calculus</title>
    <link>https://davidshortland.dev/writing/building-a-mathematics-interpreter</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/building-a-mathematics-interpreter</guid>
    <pubDate>Fri, 17 Apr 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>How we built a full interpreter with symbolic differentiation, six number types, and interactive graph plotting, and what the architectural decisions reveal about building extensible systems.</description>
    <content:encoded><![CDATA[<p>For the Advanced Programming module at UEA we built a mathematics interpreter in F#, from scratch. You give it a string like <code>d/dx(sin(x^2) + 3/4)</code> and it parses that into a tree, differentiates the tree symbolically, simplifies what comes back, and plots it on a graph you can pan around.</p>
<p>This post is about the decisions that shaped it, and about how much of what the system ended up doing follows from one choice we made at the start.</p>
<h2 id="the-pipeline">The Pipeline</h2>
<p>Every interpreter has roughly the same shape. Raw text in, structured meaning out. Ours has four stages:</p>
<figure><img src="https://davidshortland.dev/writing/images/interpreter-pipeline.png" alt="Interpreter pipeline diagram" width="800" height="200" loading="lazy" /><figcaption>Interpreter pipeline diagram</figcaption></figure>
<p>The lexer breaks the input into tokens, the parser arranges those tokens into a tree that represents the mathematical structure, and the evaluator walks the tree and computes a result. Each stage knows nothing about the others.</p>
<p>What makes this more than a calculator is the feedback arrows at the bottom. The AST doesn't have to travel to the evaluator and stop there. It can be transformed by the differentiator into a new AST, or handed to the plotter and re-evaluated hundreds of times at different x-values. That reuse turned out to be the most important property of the architecture, and it isn't the one we started with.</p>
<h2 id="what-the-parser-returns">What the Parser Returns</h2>
<p>The stub we were given combined parsing and evaluation into a single pass. The parser would see <code>3 + 4</code> and, rather than build a tree node, compute <code>7</code> on the spot. That works for simple arithmetic and it caps everything above it. You can't differentiate a number and you can't plot <code>7</code>. What the rest of the system needs is the structure of the expression, which the single pass throws away at exactly the moment it has it.</p>
<p>So our first real decision was to separate the two completely. The parser returns an abstract syntax tree:</p>
<pre><code class="language-fsharp">type Expr =
    | Number of NumberType
    | Variable of string
    | BinaryOp of BinaryOperator * Expr * Expr
    | UnaryOp of UnaryOperator * Expr
    | FunctionCall of string * Expr
    | VectorLiteral of Expr list
    | MatrixLiteral of Expr list list</code></pre>
<p>That's a discriminated union in F#, a type that says an expression is one of these seven things. A <code>BinaryOp</code> holds an operator and two sub-expressions, which are themselves <code>Expr</code> values, so it's trees all the way down.</p>
<p>Here's what the tree looks like for <code>2 + 3 * x</code>:</p>
<figure><img src="https://davidshortland.dev/writing/images/ast-example.png" alt="AST example diagram" width="800" height="380" loading="lazy" /><figcaption>AST example diagram</figcaption></figure>
<p>Operator precedence is baked into the shape. Multiplication binds tighter than addition, so <code>Mul</code> sits lower. Walk the tree depth-first and you compute <code>3 * x</code> before adding <code>2</code>. The mathematics lives in the structure. Nothing downstream has to reconstruct it.</p>
<p>Everything else in the project depends on this. Symbolic differentiation is a transformation from one AST into another. Integration evaluates the same AST at hundreds of points. The GUI keeps the AST around and re-evaluates it whenever the user pans or zooms the graph. The general form of the principle is to keep what something means separate from what you do with it. Domain models in backend systems are the same idea.</p>
<h2 id="two-phase-lexing">Two-Phase Lexing</h2>
<p>The lexer has one awkward problem. Is <code>-</code> subtraction or negation?</p>
<p>In <code>3 - 5</code> it's subtraction. In <code>3 * -5</code> and in <code>(-5)</code> it's negation. The character is identical and the meaning depends entirely on what came before it, so we handled it in two passes.</p>
<p>The first pass tokenises naively and every <code>-</code> becomes a <code>Sub</code> token. The second pass walks the token stream and reclassifies: a <code>Sub</code> becomes a <code>UnaryMinus</code> if it appears at the start of the input, after an opening parenthesis, after an operator, or after an assignment. Splitting it this way means the first pass never has to track state and the second never has to think about characters.</p>
<p>The same two-pass idea handles rational numbers. When the lexer sees <code>3/4</code> it has to decide whether that's the rational three-quarters or an integer division, and the rule we settled on is to treat <code>/</code> as a rational separator only when the accumulator has no decimal point and the denominator is a valid integer. <code>3/4</code> becomes <code>Rational(3, 4)</code>. <code>3.0/4</code> becomes <code>Float(3.0) Div Int(4)</code>, because the decimal point has already given up the exactness a rational is there to preserve.</p>
<h2 id="parsing-getting-precedence-right">Parsing: Getting Precedence Right</h2>
<p>The parser uses recursive descent with precedence climbing. Each precedence level gets its own function, and the lower-precedence functions call the higher-precedence ones:</p>
<pre><code class="language-text">parseExpression  -&gt;  handles + and -    (lowest precedence)
parseTerm        -&gt;  handles * / %      (medium)
parseFactor      -&gt;  handles ^          (highest binary)
parsePrimary     -&gt;  handles atoms      (numbers, variables, functions, parens)</code></pre>
<p><code>parseExpression</code> calls <code>parseTerm</code> for its operands, which calls <code>parseFactor</code>, which calls <code>parsePrimary</code>. The correct tree shape falls out of the call order. <code>2 + 3 * 4</code> parses as <code>Add(2, Mul(3, 4))</code>, because <code>parseTerm</code> has already claimed the multiplication by the time <code>parseExpression</code> gets to look at the <code>3</code>.</p>
<p>Exponentiation is the one place the pattern breaks. Most operators are left-associative, so <code>2 - 3 - 4</code> means <code>(2 - 3) - 4</code>, but exponentiation associates rightward and <code>2^3^2</code> has to evaluate as <code>2^(3^2) = 512</code> rather than <code>(2^3)^2 = 64</code>. Both readings produce a plausible-looking number, which is why that class of bug survives casual testing and shows up in production.</p>
<p>The fix is one line, in how the parser recurses:</p>
<pre><code class="language-fsharp">| Pow :: tail -&gt;
    let tokens', rightExpr = parseFactor tail   // recurse on parseFactor, not parseFactorRest
    (tokens', BinaryOp(Exponentiation, leftExpr, rightExpr))</code></pre>
<p>Left-associative operators recurse on their own "rest" function and build the tree leftward. Right-associative operators recurse on the base function and build it rightward.</p>
<h2 id="evaluation-and-the-symbol-table">Evaluation and the Symbol Table</h2>
<p>Once the parser hands over an AST, the evaluator walks it depth-first. The interesting question there is state.</p>
<p>Mathematical expressions live in a context. After <code>x = 5</code>, the expression <code>x + 3</code> should evaluate to <code>8</code>, and that context is the symbol table, a map from variable names to values. The mutable approach stores the table as a shared object that the evaluator reads and writes. It works, but it makes evaluation order matter in ways that are hard to see from the call site, and it makes tests harder to isolate. We used F#'s immutable maps instead, so the evaluator takes a symbol table in and returns one out:</p>
<pre><code class="language-fsharp">let evaluateStatement (statement: Statement) (symbolTable: SymbolTable)
    : NumberType * SymbolTable =
    match statement with
    | ExpressionStmt expr -&gt;
        let value = evaluateExpr expr symbolTable
        (value, symbolTable)                          // table unchanged
    | Assignment(varName, expr) -&gt;
        let value = evaluateExpr expr symbolTable
        let newTable = Map.add varName value symbolTable
        (value, newTable)                             // new table returned</code></pre>
<p>Evaluating an expression leaves the table alone. An assignment returns a new table with the binding added, and the old table still exists, unchanged. That's what lets the plotter evaluate <code>y = x^2</code> against hundreds of different symbol tables, one per x-value, with no interference between them. Threading state through as <code>(input, state) -> (output, newState)</code> is the same shape you find in Redux reducers, event sourcing and database transactions.</p>
<h2 id="six-number-types">Six Number Types</h2>
<p>The interpreter supports integers, floats, rationals, complex numbers, vectors and matrices, each a case in a single discriminated union:</p>
<pre><code class="language-fsharp">type NumberType =
    | Int of int
    | Float of float
    | Rational of int * int
    | CustomComplex of float * float
    | Vector of float list
    | Matrix of float list list</code></pre>
<p>The interesting problem is what happens when an integer meets a rational, or a float meets a complex number. We implemented automatic type promotion, where the less general type promotes to the more general one when the two meet in an operation.</p>
<figure><img src="https://davidshortland.dev/writing/images/type-promotion.png" alt="Type promotion hierarchy" width="800" height="320" loading="lazy" /><figcaption>Type promotion hierarchy</figcaption></figure>
<p>The rules preserve precision wherever there's still precision to preserve. <code>Int + Rational</code> stays <code>Rational</code>, since both are exact. <code>Float + Rational</code> converts the rational to a float, because the float has already lost exactness and there's nothing left to protect. <code>Anything + Complex</code> promotes to complex. So <code>5 + 1/2</code> produces <code>Rational(11, 2)</code> and not <code>Float(5.5)</code>.</p>
<p>Rationals simplify themselves through GCD:</p>
<pre><code class="language-fsharp">let simplifyRational (num: int) (den: int) : int * int =
    let g = gcd num den
    let newNum = num / g
    let newDen = den / g
    if newDen &lt; 0 then (-newNum, -newDen) else (newNum, newDen)</code></pre>
<p>A rational with denominator 1 collapses back to an integer, so <code>6/3</code> evaluates to <code>Int(2)</code> rather than <code>Rational(2, 1)</code>, and the system always settles on the most specific type that can represent the result.</p>
<p>At 908 lines it's the largest module in the project, and most of that is the combinatorial expansion of every operation across every pair of types. Unglamorous code. The edge cases buried in it (division by zero in rationals, negative denominators, complex division by conjugate) are most of what makes the rest of the system trustworthy.</p>
<h2 id="symbolic-differentiation">Symbolic Differentiation</h2>
<p>This is the part of the project I'm most pleased with.</p>
<p><code>computeDerivative</code> takes an AST and a variable name and returns a new AST representing the derivative. The rules you learn in calculus are implemented as recursive tree transformations.</p>
<p><strong>Constant rule:</strong> the derivative of a number is zero.</p>
<pre><code class="language-fsharp">| Number _ -&gt; Number(Int 0)</code></pre>
<p><strong>Variable rule:</strong> the derivative of <code>x</code> with respect to <code>x</code> is 1, and any other variable is treated as a constant.</p>
<pre><code class="language-fsharp">| Variable name when name = varName -&gt; Number(Int 1)
| Variable _ -&gt; Number(Int 0)</code></pre>
<p><strong>Product rule:</strong> <code>d/dx[f * g] = f' * g + f * g'</code></p>
<pre><code class="language-fsharp">| BinaryOp(Multiplication, left, right) -&gt;
    BinaryOp(Addition,
        BinaryOp(Multiplication, computeDerivative left varName, right),
        BinaryOp(Multiplication, left, computeDerivative right varName))</code></pre>
<p><strong>Chain rule:</strong> <code>d/dx[f(g(x))] = f'(g(x)) * g'(x)</code></p>
<pre><code class="language-fsharp">| FunctionCall(funcName, argExpr) -&gt;
    let innerDerivative = computeDerivative argExpr varName
    let outerDerivative = match funcName.ToLower() with
        | "sin" -&gt; FunctionCall("cos", argExpr)
        | "cos" -&gt; UnaryOp(Negation, FunctionCall("sin", argExpr))
        | "exp" -&gt; FunctionCall("exp", argExpr)
        | "ln"  -&gt; BinaryOp(Division, Number(Int 1), argExpr)
        ...
    BinaryOp(Multiplication, outerDerivative, innerDerivative)</code></pre>
<p>Between them these cover 11 mathematical functions, the product rule, the quotient rule, the power rule with both constant and variable exponents, and the chain rule composing all of it. The output is an AST throughout, never a number.</p>
<p>Raw symbolic derivatives are ugly, though. Put <code>x^2</code> through the power rule and you get <code>2 * (x^(2-1) * 1)</code>, technically correct and not something any human would write. So there's a simplification pass:</p>
<pre><code class="language-fsharp">| BinaryOp(Multiplication, Number(Int 1), right) -&gt; simplifyExpr right
| BinaryOp(Addition, Number(Int 0), right) -&gt; simplifyExpr right
| BinaryOp(Exponentiation, base_, Number(Int 1)) -&gt; simplifyExpr base_
| BinaryOp(Exponentiation, _, Number(Int 0)) -&gt; Number(Int 1)</code></pre>
<p>These rules apply recursively until the expression stops changing, which turns <code>2 * (x^(2-1) * 1)</code> into <code>2 * x</code>. The simplifier folds constants too. <code>3 + 4</code> becomes <code>7</code>, and nested multiplications flatten, so <code>2 * (3 * x)</code> becomes <code>6 * x</code>.</p>
<p>None of this works without the AST decision from earlier, because differentiation is tree transformation and a parser that evaluated as it went would have left nothing to transform.</p>
<h2 id="root-finding-newton-raphson">Root Finding: Newton-Raphson</h2>
<p>The interpreter finds roots with Newton-Raphson. Start with a guess, evaluate the function and its derivative at that point, then step in the direction the derivative suggests.</p>
<pre><code class="language-text">x_next = x_current - f(x_current) / f'(x_current)</code></pre>
<p>Convergence is quadratic, so each iteration typically doubles the number of correct digits. Our implementation runs up to 500 iterations per starting point with a tolerance of <code>1e-10</code>.</p>
<p>The awkward part is that a single starting point might find only one root, or converge to the wrong one, or diverge entirely. Our answer is unsubtle: generate 1000 evenly spaced initial guesses across the search interval, run Newton-Raphson from every one of them, discard the failures and de-duplicate what's left. Brute force in the search space, precise once it converges.</p>
<p>This reuses the symbolic differentiation. The derivative Newton-Raphson needs is computed from the AST rather than approximated numerically, so convergence is exact to the precision of floating-point arithmetic instead of being limited by the size of a finite-difference step.</p>
<h2 id="the-gui">The GUI</h2>
<p>The WPF GUI is where the architecture becomes tangible. A user types <code>y = sin(x)</code>, hits Plot, and the curve appears. They click Derivative and the orange <code>cos(x)</code> overlay draws on top. They type bounds and the integral shades blue beneath the curve. They click Find Roots and markers appear at the zeros.</p>
<p>Behind every one of those interactions is the same AST being read again. The plot evaluates it at hundreds of x-values, the derivative button calls <code>computeDerivative</code> on it and plots the AST that comes back, the integral evaluates it at the trapezoidal quadrature points, and the root finder passes it and its symbolic derivative to Newton-Raphson.</p>
<p>Deferred evaluation is what makes that feel smooth. When the user types <code>y = x^2 + 3</code>, the interpreter detects that <code>x</code> is a free variable and stores the AST without evaluating it. There's no error and no prompt for a value. The expression waits until something supplies a context, either the Plot button handing it hundreds of x-values or a later definition of <code>x</code>.</p>
<p>Panning and zooming re-evaluate across the new viewport bounds. That's fast enough to feel instantaneous because the AST is a lightweight data structure rather than a closure or a string that has to be re-parsed, so the user is manipulating the tree directly without knowing it.</p>
<p>Separating parsing from evaluation is what turned a calculator into a computer algebra system. Symbolic differentiation, the integration visualisation, root finding and interactive plotting were all out of reach while the parser was collapsing expressions into values, and all of them were straightforward once it stopped. The abstractions you choose early decide what's easy later and what's impossible. You generally have to choose them before you know which is which.</p>
<p>The interpreter came to about 4,700 lines of code across F# and C#, and the most important line in it is probably <code>type Expr =</code>.</p>]]></content:encoded>
    <category>fsharp</category>
    <category>interpreters</category>
    <category>architecture</category>
    <category>functional-programming</category>
  </item>
  <item>
    <title>Building an Event-Driven Health Tracker with Three Lambda Functions</title>
    <link>https://davidshortland.dev/writing/building-an-event-driven-health-tracker</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/building-an-event-driven-health-tracker</guid>
    <pubDate>Mon, 13 Apr 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>How we built a health tracking platform with event-driven notifications, scheduled jobs, and auto-completing goals, and why decoupling what happens from when it happens made everything simpler.</description>
    <content:encoded><![CDATA[<p>The brief was a health and fitness tracker. Log exercises, record meals, track weight, set goals. Standard full-stack coursework.</p>
<p>We could have built it as a monolith, one Express server handling requests and sending emails and checking for overdue goals in the same process, and it would have worked. Monoliths that send emails in the request path are fragile, though. If the email service is slow the user waits, and if it fails the request fails, and a user's goal achievement notification shouldn't be coupled to whether AWS SES responded in time.</p>
<p>So we split the system into three independently deployable Lambda functions connected by events. The API handles requests, a notification service sends emails asynchronously, and a scheduled function checks for overdue goals every morning. Each one does its job without knowing how the others work.</p>
<p>This post is about that architecture: why we split it, how the pieces connect, and what the separation made possible.</p>
<h2 id="the-architecture">The Architecture</h2>
<figure><img src="https://davidshortland.dev/writing/images/health-tracker-architecture.png" alt="Health Tracker architecture diagram" width="800" height="420" loading="lazy" /><figcaption>Health Tracker architecture diagram</figcaption></figure>
<p>The system has two paths. The synchronous path handles user requests, where the Angular frontend calls the API Gateway, which invokes the API Lambda, which reads and writes to MongoDB. That part is straightforward.</p>
<p>The asynchronous path is where it gets interesting. When something notable happens in the API, a user registering or a goal being achieved or a group invitation going out, the API publishes a message to an SNS topic and moves on. It doesn't send an email. It doesn't know that emails exist.</p>
<p>The SNS topic delivers messages to an SQS queue, and the queue triggers a second Lambda that reads the message, renders an HTML email from a Liquid template and sends it through SES. If that Lambda fails, the message stays in the queue and gets retried, the API never knows, and the user's request was never blocked in the first place.</p>
<p>A third Lambda runs on a CloudWatch Events schedule, once per day at 9 AM. It queries the API for goals past their target date and publishes overdue notifications to the same SNS topic, where the notification Lambda picks them up like any other event.</p>
<p>Three functions, one SNS topic, one queue, and the event types distinguished by their subject line rather than by separate infrastructure.</p>
<h2 id="separating-what-happens-from-when-it-happens">Separating What Happens from When It Happens</h2>
<p>The key design decision is that the API's job is to record <em>what happened</em> and not to decide <em>what to do about it</em>.</p>
<p>When a user logs a weight measurement that happens to hit a goal target, the API saves the measurement and marks the goal complete, then publishes a <code>GoalAchieved</code> event and returns the response. The API is done.</p>
<p>The notification Lambda picks up that event independently and decides what to do about it. It checks whether the user has verified their email, and if they have it renders a congratulations email with a suggested next goal, the target incremented by a sensible amount. If they haven't verified, it does nothing.</p>
<pre><code class="language-typescript">if (subj === goalAchievedSnsSbj) {
    const {email, htmlContent, emailVerified} = await render_goal_achieved(message, baseWebsiteUrl, engine);
    if (emailVerified) {
        await sendEmail(sender, email, "Goal achieved! Way to go!", htmlContent);
    }
}</code></pre>
<p>This separation is what keeps the API controller code clean. The registration endpoint publishes a <code>Registered</code> event and contains no email rendering logic. The goal completion code publishes <code>GoalAchieved</code> and knows nothing about suggested next goals. Each concern lives in exactly one place.</p>
<p>It also means the notification logic can change without touching the API. We added the "suggest a new goal" feature entirely within the notification Lambda, and the API never had to be redeployed.</p>
<h2 id="auto-completing-goals">Auto-Completing Goals</h2>
<p>The most interesting behaviour in the system is reactive: goals that complete themselves when the user logs data.</p>
<p>When a user creates a health metric, meaning a weigh-in, the database service saves the record and then checks every open weight goal for that user:</p>
<pre><code class="language-typescript">async createHealthMetricAsync(healthMetric: HealthMetricDocument) {
    let healthMetricDocument = await HealthMetrics.create(healthMetric);

    const weightGoals = await Goal.find({
        userId: healthMetric.userId,
        type: GoalType.WEIGHT,
        completed: false
    });

    const goals = await Promise.all(weightGoals.map(async goal =&gt; {
        goal.currentValue = lastHealthMetric.weight;
        if (goal.currentValue &lt;= goal.targetValue) {
            goal.completed = true;
            goal.completedDate = new Date();
        }
        return await goal.save();
    }));

    return {healthMetric: healthMetricDocument, completedGoals: goals.filter(g =&gt; g.completed)};
}</code></pre>
<p>The same pattern applies to exercise goals. Logging a run updates every open distance goal for that exercise type, logging a workout updates every open duration goal, and the controller then publishes <code>GoalAchieved</code> events for any goals that were completed, which flow through SNS to the notification Lambda.</p>
<p>From the user's perspective they log a run and a few seconds later get an email saying they hit their 100km goal. From the system's perspective five things happened in sequence: the exercise was saved, matching goals were queried, progress was updated, the controller published events, and the notification Lambda rendered and sent an email asynchronously and separately. Each step knows only about itself and the next.</p>
<p>The calorie calculation is a nice detail too. When an exercise is logged, the system pulls the exercise type's MET value and the user's most recent weight and calculates the calories burned automatically:</p>
<pre><code class="language-typescript">const caloriesBurned = exerciseType.mET * 3.5 * lastHealthMetric.weight * exerciseDocument.duration / 200;</code></pre>
<p>The user logs "30 minutes of running" and the system returns the exercise record with calories already worked out, nothing entered by hand. It's the standard MET formula, which is a population-average estimate rather than a measurement of what this person actually burned, but it's the same estimate every fitness tracker is quoting you.</p>
<h2 id="the-scheduled-lambda">The Scheduled Lambda</h2>
<p>The third Lambda runs on a cron schedule. Every day at 9 AM UTC, CloudWatch Events triggers it:</p>
<pre><code class="language-typescript">const overdueGoalsRule = new cdk.aws_events.Rule(this, `OverdueGoalsCheckRule-${stage}`, {
    schedule: cdk.aws_events.Schedule.expression('cron(0 9 * * ? *)'),
    description: `Trigger overdue goals check daily at 9 AM UTC`
});
overdueGoalsRule.addTarget(new cdk.aws_events_targets.LambdaFunction(checkOverdueGoalsLambda));</code></pre>
<p>The Lambda itself is small. It calls the API's internal endpoint to find overdue goals and publishes notifications for each one, and the notification Lambda handles the rest.</p>
<p>What makes that work is the internal API key. The three Lambdas share a secret generated by CDK and stored in Secrets Manager, and the API authenticates requests two ways: JWT tokens for user requests, and the raw API key for Lambda-to-Lambda communication. The overdue goals Lambda uses the API key to call the same API the frontend calls, with elevated access.</p>
<p>This keeps the overdue-checking logic in the API where it belongs and leaves the scheduled Lambda as nothing more than a trigger. If the business rules for "overdue" change, only the API needs updating.</p>
<h2 id="group-goals">Group Goals</h2>
<p>Groups add a social dimension. Users create groups, invite members via join codes and set shared goals, and the interesting part is how group goals work at the data level.</p>
<p>When someone sets a group goal, the system creates a separate goal record for every member:</p>
<pre><code class="language-typescript">async createGroupGoal(creatorId: string, groupId: string, goalData: {...}) {
    const group = await Group.findById(groupId);
    const groupGoalLink = new Types.ObjectId().toHexString();

    return await Promise.all(group.members.map(async memberId =&gt; {
        let currentValue = await this.calculateCurrentProgress(memberId, goalData);

        return await Goal.create({
            ...goalData,
            userId: memberId,
            isGroupGoal: true,
            groupId,
            groupGoalLink,
            currentValue,
            completed: currentValue &gt;= goalData.targetValue
        });
    }));
}</code></pre>
<p>Each member gets their own goal document with the same target and independent progress, and a <code>groupGoalLink</code> ties them together so the UI can show group-wide progress. From the database service's perspective group goals then work identically to personal goals. The auto-completion logic needs no special case for them. When a group member logs an exercise that completes their goal, the same <code>GoalAchieved</code> event fires and a <code>GroupGoalCompleted</code> notification goes out.</p>
<p>The alternative would have been a single shared goal document tracking multiple users' progress, which sounds simpler until you need to handle a member leaving the group, or partial completion, or showing individual progress in the UI. Denormalising into one-goal-per-member made every downstream query simpler.</p>
<h2 id="composing-the-infrastructure">Composing the Infrastructure</h2>
<p>The CDK stack defines all three Lambdas, the SNS topic, the SQS queue and their permissions in a single TypeScript file, and the pipeline builds all three services in parallel:</p>
<pre><code class="language-typescript">const buildApi = new pipelines.ShellStep(`BuildApi-${stage}`, {
    commands: ['cd HealthTrackerAPI', 'npm install', 'npm run build', 'npm run zip']
});

const buildNotificationLambda = new pipelines.ShellStep(`BuildNotificationLambda-${stage}`, {
    commands: ['cd HealthTrackerAPI.NotificationsLambda', 'npm install', 'npm run build', 'npm run zip']
});

const buildGoalOverdueLambda = new pipelines.ShellStep(`BuildGoalOverdueLambda-${stage}`, {
    commands: ['cd HealthTrackerAPI.OverdueGoalsLambda', 'npm install', 'npm run build', 'npm run zip']
});</code></pre>
<p>Three independent builds feed into a single CDK synth step that composes them into one CloudFormation stack, and the infrastructure references between them, the API Gateway URL passed to the overdue Lambda and the SNS ARN passed to the API, are wired through CDK constructs rather than hardcoded strings.</p>
<p>The system deploys to two stages, dev and prod, with isolated resources. Each stage gets its own SNS topic, its own SQS queue, its own set of secrets and its own MongoDB database, so a bug in the notification template in dev can't send emails to prod users. The stages share nothing except the pipeline that deploys them.</p>
<p>The three-Lambda split was never about scalability, since a monolith would have handled the load of a university project comfortably. It was about keeping each concern in its own box.</p>
<p>The API doesn't know how emails are sent. The notification Lambda doesn't know how goals are tracked. The scheduled Lambda doesn't know how overdue goals are identified. Each function has a single reason to change, and when it does change the blast radius is limited to itself. The event bus, SNS plus SQS, is the contract between them, and as long as the message format stays stable any of the three can be rewritten, redeployed or replaced independently. That's the practical benefit here, being able to change one part of the system without coordinating with every other part.</p>]]></content:encoded>
    <category>aws</category>
    <category>lambda</category>
    <category>sns</category>
    <category>event-driven</category>
    <category>angular</category>
    <category>architecture</category>
  </item>
  <item>
    <title>Building WeatherWise: A Weather Platform That Tells You What to Do</title>
    <link>https://davidshortland.dev/writing/building-weatherwise</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/building-weatherwise</guid>
    <pubDate>Sat, 04 Apr 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Building a weather platform that transforms raw API data into prioritised, actionable recommendations, and the full-stack architecture that supports it.</description>
    <content:encoded><![CDATA[<p>Every weather app shows you the same thing: temperature, humidity, wind speed, a little cloud icon. You look at the number, you decide what it means for your day, you close the app. The interpretation is your problem.</p>
<p>For the Advanced Web Development module at UEA I built WeatherWise. It does the interpretation for you. Instead of showing "UV index: 9" and leaving you to work out what that means, it tells you to wear SPF 50+, stay out of direct sun between 10 and 4, and bring a hat. Instead of "wind: 45 km/h" it tells you to secure outdoor furniture and drive carefully.</p>
<p>The project scored 95%, in a module that came out at 89% overall, and I think the reason is that the interesting engineering isn't in fetching weather data, which is an API call, but in what happens between the data arriving and the user seeing it.</p>
<h2 id="the-insight-engine">The Insight Engine</h2>
<p>The core of the application is a rule-based recommendations system that evaluates raw weather data against a set of conditions and produces prioritised, actionable insights.</p>
<figure><img src="https://davidshortland.dev/writing/images/weatherwise-insights.png" alt="Insights engine diagram" width="800" height="400" loading="lazy" /><figcaption>Insights engine diagram</figcaption></figure>
<p>Each insight has a category (safety, health, travel, activity, clothing, business), a priority level, a description of the condition, and an action. The action is the specific thing the user should do.</p>
<p>The rules are layered by severity:</p>
<p><strong>High priority</strong> covers safety-critical conditions. Visibility below 1 km triggers fog driving advice, temperature above 35 degrees triggers heat warnings with hydration targets, and UV index 8 or above triggers specific sunscreen SPF recommendations and time windows to avoid.</p>
<p><strong>Medium priority</strong> covers preparation. Rain in the forecast triggers umbrella and waterproof advice, and the engine also checks the time of day, so rain between 6 and 9 AM adds a commute-specific recommendation to leave 20 minutes early, check traffic apps and consider public transport. High humidity combined with high temperature triggers hydration alerts that wouldn't fire for either condition alone.</p>
<p><strong>Low priority</strong> captures opportunities. Temperature between 20 and 28 degrees with UV below 6 and no rain gets you a suggestion to go outside, and it only fires when none of the higher-priority conditions are active.</p>
<pre><code class="language-typescript">interface Insight {
  category: 'clothing' | 'activity' | 'travel' | 'health' | 'business' | 'safety';
  priority: 'high' | 'medium' | 'low';
  icon: React.ReactNode;
  title: string;
  description: string;
  action: string;
}</code></pre>
<p>The insights are sorted by priority before rendering. High-priority items appear first with red indicators, medium items are amber, low items are green. If nothing triggers at all, the component shows a positive "all clear" message instead of an empty state.</p>
<p>What makes this more than a series of if-statements is that the rules compose. The commute recommendation checks for rain <em>and</em> a specific time window. The humidity alert checks humidity <em>and</em> temperature, because 85% humidity at 15 degrees isn't a health concern and 85% humidity at 30 degrees is. The rules encode domain knowledge about when weather conditions actually matter to a person's day.</p>
<h2 id="the-stack">The Stack</h2>
<p>This project uses a different stack from everything else in my portfolio, which was part of the point. The work projects and my other university work are Angular. WeatherWise is Next.js 15 with React 19, PostgreSQL with Drizzle ORM, and Zustand for state management.</p>
<p>Next.js gave me the App Router for file-based routing with server components, API routes co-located with the pages that use them, and middleware for authentication guards. Drizzle gave me type-safe database queries that infer their types from the schema definition, so the TypeScript compiler catches query errors at build time rather than at runtime. Zustand gave me a lightweight store without Redux's boilerplate.</p>
<p>Between them the type safety runs from the database schema through the API routes to the React components. Change the schema and the compiler errors appear everywhere that data is used.</p>
<h2 id="authentication-no-passwords">Authentication: No Passwords</h2>
<p>WeatherWise uses Google OAuth exclusively. There's no registration form, no password field, no forgot-password flow. Users click "Continue with Google" and they're in.</p>
<p>This was a deliberate design decision rather than a shortcut. Password authentication means storing hashed passwords, building reset flows, handling rate limiting, dealing with weak passwords, and accepting liability for credential storage, and OAuth delegates all of it to Google. The database stores a user's name, email and profile image. No secrets.</p>
<p>The NextAuth callback chain handles user creation automatically:</p>
<pre><code class="language-typescript">async signIn({ user, account, profile }) {
    const existingUser = await db.select().from(users)
        .where(eq(users.email, user.email));

    if (existingUser.length === 0) {
        await db.insert(users).values({
            name: user.name,
            email: user.email,
            image: user.image,
            preferences: { temperatureUnit: 'celsius', windUnit: 'kmh' }
        });
    } else {
        await db.update(users)
            .set({ name: user.name, image: user.image, updatedAt: new Date() })
            .where(eq(users.email, user.email));
    }
    return true;
}</code></pre>
<p>First sign-in creates the user with sensible defaults, and subsequent sign-ins update the profile image and name in case the user changed them on Google's side. The JWT session lasts 30 days. The schema took three migrations to reach this design: the first version had a password field, the second added OAuth, and the third removed passwords entirely.</p>
<h2 id="state-management-and-caching">State Management and Caching</h2>
<p>The Zustand store manages the weather data cache, user preferences, saved locations and loading state. The interesting part is how it handles multiple locations.</p>
<p>The dashboard loads weather for up to four saved locations in parallel:</p>
<pre><code class="language-typescript">const weatherPromises = locations.map(async (location) =&gt; ({
    locationId: location.id,
    weather: await fetch(`/api/weather/current?location=${lat},${lon}`).then(r =&gt; r.json())
}));

const results = await Promise.all(weatherPromises);</code></pre>
<p>Each result is stored in a <code>locationWeatherCache</code> object keyed by location ID, and when the user removes a location its cached weather is pruned with it:</p>
<pre><code class="language-typescript">removeLocation: (locationId) =&gt; set((state) =&gt; ({
    locations: state.locations.filter(loc =&gt; loc.id !== locationId),
    locationWeatherCache: Object.fromEntries(
        Object.entries(state.locationWeatherCache)
            .filter(([key]) =&gt; key !== locationId)
    )
}));</code></pre>
<p>The store also handles unit conversion. Converting at the component level would scatter the conversion logic across the codebase, so the store provides helper methods instead:</p>
<pre><code class="language-typescript">getTemperatureInUnit: (tempC, tempF) =&gt; {
    return get().preferences.temperatureUnit === 'celsius' ? tempC : tempF;
}</code></pre>
<p>Components call the helper, so the preference propagates from one place. Add a Kelvin option tomorrow and only the store changes.</p>
<h2 id="location-comparison">Location Comparison</h2>
<p>The comparison feature lets users place up to four saved locations side by side. Each location loads its weather in parallel and the UI highlights the best and worst values for each metric.</p>
<p>The highlighting logic is context-aware. For temperature, higher isn't necessarily better or worse, so it's left neutral. For UV and wind, lower is better. For visibility, higher is better. The system determines best and worst per metric, and it only applies colour highlighting when three or more locations are being compared, since with two it's obvious.</p>
<p>This feature reuses the same weather API calls and Zustand cache as the dashboard, so a location that was already loaded on the dashboard doesn't get fetched again. The comparison just reads from the store.</p>
<h2 id="geolocation-and-fallbacks">Geolocation and Fallbacks</h2>
<p>When the dashboard loads it requests the browser's geolocation with a 5-second timeout:</p>
<pre><code class="language-typescript">navigator.geolocation.getCurrentPosition(
    (position) =&gt; {
        const { latitude, longitude } = position.coords;
        if (savedLocations.length === 0) {
            loadWeatherForLocation(latitude, longitude);
        }
    },
    (error) =&gt; {
        if (savedLocations.length === 0) {
            loadWeatherForLocation(51.5074, -0.1278); // London fallback
        }
    },
    { enableHighAccuracy: false, timeout: 5000, maximumAge: 0 }
);</code></pre>
<p>The fallback strategy has two layers. If the user has saved locations, those take priority over geolocation entirely, because the user has already told the system what they care about. If they have no saved locations and geolocation fails, whether through denied permissions or a timeout or unavailability, it falls back to London. The user always sees weather data and never an empty screen.</p>
<p>Locations are stored and queried by latitude and longitude, not by city name. "Portland" could be Oregon or Maine. Coordinates avoid the ambiguity and give precise results from the weather API.</p>
<h2 id="the-data-model">The Data Model</h2>
<p>The database has two tables. Users store authentication data and preferences as a JSON column:</p>
<pre><code class="language-typescript">export const users = pgTable('users', {
    id: uuid('id').defaultRandom().primaryKey(),
    name: varchar('name', { length: 255 }),
    email: varchar('email', { length: 255 }).unique().notNull(),
    image: text('image'),
    preferences: json('preferences').$type&lt;{
        temperatureUnit: 'celsius' | 'fahrenheit';
        windUnit: 'mph' | 'kmh';
    }&gt;().default({ temperatureUnit: 'celsius', windUnit: 'kmh' }),
    createdAt: timestamp('created_at').defaultNow(),
    updatedAt: timestamp('updated_at').defaultNow()
});</code></pre>
<p>Locations use decimal precision to seven places, which is roughly 1 centimetre of accuracy, and cascade-delete with their user:</p>
<pre><code class="language-typescript">export const locations = pgTable('locations', {
    id: uuid('id').defaultRandom().primaryKey(),
    userId: uuid('user_id').references(() =&gt; users.id, { onDelete: 'cascade' }).notNull(),
    name: varchar('name', { length: 255 }).notNull(),
    latitude: numeric('latitude', { precision: 10, scale: 7 }).notNull(),
    longitude: numeric('longitude', { precision: 10, scale: 7 }).notNull(),
    isDefault: boolean('is_default').default(false),
    createdAt: timestamp('created_at').defaultNow()
});</code></pre>
<p>Preferences live in a JSON column rather than in separate columns because they're always read and written as a unit, so adding a new preference like a pressure unit means updating the TypeScript type and the default value, with no migration needed.</p>
<p>API inputs are validated with Zod schemas at the route boundary. The preferences endpoint rejects anything that isn't a valid unit combination:</p>
<pre><code class="language-typescript">const preferencesSchema = z.object({
    temperatureUnit: z.enum(['celsius', 'fahrenheit']),
    windUnit: z.enum(['mph', 'kmh'])
});</code></pre>
<p>Invalid input gets a 400 before it reaches the database, and valid input is type-safe from that point forward.</p>
<p>The weather data itself is free. WeatherAPI.com gives you temperature, wind, UV, humidity, visibility, pressure, forecasts and alerts, and any developer can display that in a grid.</p>
<p>The value is in the layer between the data and the user. The insight engine is only about 200 lines of code. It encodes the domain knowledge that most weather apps leave to the user: what UV 9 means for your skin, what 0.8 km visibility means for your drive, what rain at 7 AM means for your commute.</p>]]></content:encoded>
    <category>nextjs</category>
    <category>react</category>
    <category>postgresql</category>
    <category>drizzle</category>
    <category>zustand</category>
    <category>architecture</category>
  </item>
  <item>
    <title>Alive to Guess Again</title>
    <link>https://davidshortland.dev/writing/alive-to-guess-again</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/alive-to-guess-again</guid>
    <pubDate>Tue, 24 Mar 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Karl Popper argued that a theory which can&apos;t be proven wrong isn&apos;t really saying anything. The same is true of engineering practices: if you aren&apos;t actively trying to break them, you don&apos;t know whether they&apos;re working.</description>
    <content:encoded><![CDATA[<p>There's a problem with justification that I haven't addressed. You can justify almost anything if you're allowed to be vague enough. We do standups because they improve communication. We write tests because they improve quality. We use microservices because they improve scalability. These sound like reasons and they have the shape of reasons, and something important is missing from all of them.</p>
<p>Nobody is trying to prove them wrong.</p>
<h2 id="popper-s-razor">Popper's Razor</h2>
<p>Karl Popper was a philosopher of science who spent most of his career on a single question: what separates real science from things that merely look like science? His answer was falsifiability, and the idea goes deeper than most people realise when they first meet it.</p>
<p>Popper wasn't only saying that theories should be testable. He was saying that science progresses by actively trying to destroy its own theories. You accept a theory provisionally, as the best available explanation, and then you do everything you can to break it. You don't test it in the easy cases; you test it at the extremes, in the conditions where it's most likely to fail. If it survives serious attempts at refutation it earns its place for now. The moment it does fail, you discard it and move on.</p>
<p>The distinction matters. Proving gravity by dropping a ball is trivial, because everyone already knows the ball will fall. The real test is at the boundaries, near a black hole or at quantum scales, in the conditions where the theory might break down. Easy confirmations tell you nothing, and hard tests are where knowledge lives.</p>
<p>Popper's classic examples were astrology and certain readings of Freudian psychoanalysis. An astrologer can explain any outcome after the fact, and if a prediction was wrong there's always a reason: another planet was in retrograde, the birth time was imprecise, the subject wasn't receptive. The theory never fails because it can absorb any result. Einstein's general relativity made a specific, testable prediction about how light bends around massive objects, and if the 1919 eclipse observations had shown no bending the theory would have been wrong. That vulnerability is what made it valuable.</p>
<p>Or as Popper put it: good tests kill flawed theories; we remain alive to guess again.</p>
<p>I came to Popper through a mentor's recommendation, and the moment I understood the argument I started seeing unfalsifiable claims everywhere in software engineering. Worse, I started seeing them in my own work.</p>
<figure><img src="https://davidshortland.dev/writing/images/active-refutation.png" alt="Falsifiable vs unfalsifiable practices" width="800" height="420" loading="lazy" /><figcaption>Falsifiable vs unfalsifiable practices</figcaption></figure>
<h2 id="the-unit-test-problem">The Unit Test Problem</h2>
<p>Here's something I did that taught me this concretely.</p>
<p>I was working on a system and decided it needed better test coverage. That felt like an obviously good decision. Tests improve quality, everyone knows this, so I went through the existing codebase and wrote unit tests for the code that was already there.</p>
<p>The tests passed, coverage went up, and it felt productive. What I was doing was the equivalent of dropping a ball and confirming that gravity works. Every test I wrote verified that the code did what the code already did. I'd look at an implementation, understand its behaviour, then write an assertion confirming it. Easy confirmations, all of them, testing the theory that the code is correct in the most comfortable conditions available: the normal inputs, the happy path, the cases I already knew worked.</p>
<p>What I never did was try to break it. I never asked what the boundary conditions were where this logic might fall apart, what inputs would expose a flaw in my assumptions, what the black hole for this function looked like. I was accumulating confirmations instead of attempting refutations.</p>
<p>The coverage number looked good and the test suite was unfalsifiable in practice. It couldn't fail in a way that told me anything I didn't already know. If a test broke it was because someone had changed the implementation, never because it had caught a genuine behavioural problem. The tests were a mirror held up to the code, reflecting it back at itself.</p>
<p>What I should have done is what TDD actually intends. Define the expected behaviour first, write code to satisfy it, and include the edge cases and boundary conditions where the behaviour might break. A test that says "when a driver completes a session, their lap times are ranked and the fastest is marked" is testing a business rule at its core. The Popperian step is the next one. What happens when two lap times are identical? When the session has zero laps? When it has exactly one? Those are the hard tests, and they're the ones that kill flawed implementations.</p>
<p>In a small team you can't afford to write tests for the sake of coverage. Every test should encode a business rule that would cause a real problem if it were violated, and the most valuable ones test that rule in the conditions where it's most likely to break.</p>
<h2 id="the-pattern-is-everywhere">The Pattern Is Everywhere</h2>
<p>Once I started looking for practices that had never survived a serious attempt at refutation, I couldn't stop finding them.</p>
<p><strong>Standups.</strong> Most teams justify standups as improving communication or keeping everyone aligned, and most have never tried to falsify the claim. Defining what success looks like and then passively waiting to see whether it happens isn't enough. The Popperian approach is to go looking for the failure. Ask the team whether anyone had a coordination problem this week that the standup should have caught and didn't. Whether anyone sat through it already knowing everything that was said. Whether anyone withheld a problem because the format didn't make it safe to raise.</p>
<p>If you go looking for failure and can't find it, the practice has survived a genuine test. If you find failure immediately, you've learned something valuable. Either way you know more than you did, and most teams never ask. The standup continues, provisionally accepted and never tested at the extremes, until it becomes a ritual that can't fail because nobody is trying to make it fail.</p>
<p><strong>Code reviews.</strong> The justification is usually catching bugs or sharing knowledge. Track what actually happens in your reviews and you might find that 90% of the comments are about formatting, naming or style, and almost none of them catch logic errors. "Do reviews happen?" is the easy test. The hard one is whether a code review has ever caught a bug that would have reached production, how often, and what kind. If you go looking for that evidence and can't find it, the practice has been falsified. It isn't doing what you claimed it does. It might well be doing something else that's valuable, and the original justification is dead, so you should either update it or drop the practice.</p>
<p><strong>Retrospectives.</strong> Teams run retrospectives to continuously improve. A serious attempt at refutation would be to pull up the action items from the last three retrospectives and count how many were completed and how many led to a measurable change in how the team works. If the answer is that you don't track that, the practice has been insulated from failure. You've been dropping the ball and confirming that it falls.</p>
<h2 id="provisional-acceptance">Provisional Acceptance</h2>
<p>There's a subtlety in Popper's thinking that changes how I approach all of this. He never said that unfalsified theories are true. He said they're provisionally accepted: they've survived testing so far, they're the best explanation available, and new evidence could overturn them tomorrow. The provisionality is the whole point. Treat a practice as permanently justified and you stop testing it.</p>
<p>That's the difference between "we do standups because they work" and "we do standups because they've survived our attempts to find evidence that they don't, and we'll keep looking." The first is a settled belief that can't be wrong. The second is a living hypothesis. Being wrong is how you learn.</p>
<p>The same principle applies to architectural decisions, technology choices, team structures and deployment processes. Hold them provisionally. Test them where they're weakest, and drop them when the evidence turns against them.</p>
<h2 id="why-this-is-hard">Why This Is Hard</h2>
<p>Unfalsifiable practices survive because actively trying to break your own processes is uncomfortable. Define what failure looks like, go looking for it, and you might find it. Then comes the admitting, the changing course, and possibly some difficult conversations. Keeping the justification vague and the testing gentle is much easier.</p>
<p>Popper noticed the same dynamic in science. Unfalsifiable theories are popular because they're safe: they explain everything, predict nothing, and never require their proponents to change their minds. Falsifiable theories are dangerous. They put themselves on the line, and the danger is what makes them useful.</p>
<p>The connection to the pragmatist's razor is direct. That post argued that every deviation from a principle needs a specific justification. This one adds the extremes. Test a justification where it might fail, and when it fails, let it go. Good tests kill flawed practices. We remain alive to guess again.</p>
<p>I approach testing differently now. Before writing a test I ask what business rule it encodes and what inputs would break it. Not the happy path. The edge cases, the boundary conditions, the black holes. Coverage as a metric has become almost irrelevant to me, and what matters is whether each test represents a genuine attempt to falsify the assumption that the code is correct.</p>
<p>More broadly, I've started treating every practice as a provisional hypothesis. Standups, reviews, architectural patterns: they're all theories about what works, and most of them have only ever been confirmed gently.</p>
<p>I don't always get this right. The pull toward easy confirmation is strong and it takes discipline to go looking for evidence that you're wrong. That discipline is what Popper was arguing for. Testability as a habit of mind, and the honesty to update a belief when it breaks.</p>]]></content:encoded>
    <category>philosophy</category>
    <category>engineering</category>
    <category>testing</category>
    <category>agile</category>
  </item>
  <item>
    <title>The Pragmatist&apos;s Razor</title>
    <link>https://davidshortland.dev/writing/the-pragmatists-razor</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/the-pragmatists-razor</guid>
    <pubDate>Tue, 17 Mar 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Cargo cult engineering is adopting practices without understanding. But there&apos;s an equal and opposite failure: the engineer so principled they forget they&apos;re building software for people, not for architecture diagrams.</description>
    <content:encoded><![CDATA[<p>The cargo cult problem is easy to describe. You adopt a practice you don't understand, and when it fails you don't know why. There's an equal and opposite failure that's much harder to see: understanding a practice perfectly, and applying it in every context regardless of whether the problem it solves exists there.</p>
<p>The same standard of justification that applies to adopting a practice applies to how rigidly you apply it. Relaxing a principle in a given context isn't an absence of rigour, and if you do it properly it's a higher form of it.</p>
<h2 id="two-failure-modes-not-one">Two Failure Modes, Not One</h2>
<p>The cargo cult post described a single failure, practices without understanding. There are two distinct ways to misapply knowledge of a practice.</p>
<p><strong>Failure mode one: applying a practice you don't understand.</strong> This is the cargo cult problem. You adopt microservices because successful companies use microservices. You don't know what problem microservices solve, so you can't evaluate whether you have that problem, and when the practice fails you have no way to find out why.</p>
<p><strong>Failure mode two: applying a practice you do understand, in a context where the problem it solves doesn't exist.</strong> You understand that microservices solve independent deployability at scale, and you understand the trade-offs. Your team is five people, your deployment pipeline is simple, and you have no scaling pressure. You apply the practice anyway, because the principle says you should, and principles are principles.</p>
<p>The first failure is ignorance and the second is rigidity. They produce different symptoms and the same outcome: wasted effort, and systems that don't serve their users well.</p>
<p>The cargo cult test asks whether you can explain why you're doing this. The pragmatist's test extends it, and asks whether you can explain why you're doing it <em>here</em>, given <em>these</em> constraints.</p>
<h2 id="defining-pragmatism">Defining Pragmatism</h2>
<p>Pragmatism gets confused with two other things. The first is recklessness: taking shortcuts without any awareness of what you're giving up, making decisions where you can't name the trade-off because it never occurred to you that there was one. The second is the absence of principles altogether. Pragmatism is neither. It's the application of an additional principle, that the value of any practice is conditional on context, and that context has to be evaluated rather than assumed.</p>
<p>A pragmatic decision has three properties:</p>
<ol>
<li>You can name the principle you're choosing not to follow.</li>
<li>You can explain why the problem that principle addresses is either absent or less important than a competing concern in this specific situation.</li>
<li>You can describe the conditions under which you would revisit that decision.</li>
</ol>
<p>Miss any of them and the decision isn't pragmatic. It's reckless if you can't name the trade-off, and arbitrary if you can't explain the reasoning.</p>
<figure><img src="https://davidshortland.dev/writing/images/pragmatism-spectrum.png" alt="The pragmatism spectrum" width="800" height="300" loading="lazy" /><figcaption>The pragmatism spectrum</figcaption></figure>
<h2 id="the-spectrum">The Spectrum</h2>
<p>The reckless engineer and the purist are mirror images of each other. The reckless engineer makes decisions without reference to principles at all, taking shortcuts because they're faster rather than because they've been evaluated, and never asking what's being given up. The purist is the inverse, applying principles uniformly regardless of context because they're treated as unconditional laws rather than as responses to specific problems, and never asking whether the problem exists here either.</p>
<p>Pragmatism sits between them. That positioning is misleading if you read it as "easier". It's the most demanding position on the spectrum. The purist can apply the same rules everywhere without thinking, the reckless engineer can ignore rules everywhere without thinking, and the pragmatist has to think every time. Thinking takes more effort than either consistency or indifference.</p>
<h2 id="when-to-hold-and-when-to-relax">When to Hold and When to Relax</h2>
<p>Three things tend to determine it.</p>
<p>The first is the cost asymmetry of being wrong. Some principles protect against failures that are cheap to fix and others protect against failures that aren't. Input validation, authentication and data integrity all cost very little to apply correctly and can cost enormously if you skip them. When the downside of relaxing a principle significantly outweighs the cost of following it, that asymmetry does most of the reasoning for you.</p>
<p>The second is whether the shortcut is local or structural. Some deviations affect one file, one function, one component, and if you're wrong you fix it in an afternoon. Others create coupling that compounds, so that changing the database schema now requires changing the API, the frontend and the deployment pipeline. The first kind is often pragmatic. The second rarely is, because the cost isn't borne at the moment of the decision; it's deferred, and deferred costs grow.</p>
<p>The third is whether you can articulate the trade-off precisely. Not vaguely. "I'm choosing not to do X because the problem X addresses doesn't apply here, and applying it anyway would cost Y." If you can't get to that sentence, you're skipping something because it's inconvenient, which is recklessness with a better vocabulary.</p>
<h2 id="the-symmetry">The Symmetry</h2>
<p>There's a symmetry here between the cargo cult problem and the purism problem that I didn't fully see when I wrote the first post.</p>
<p>The cargo cult test says every practice should have a specific, articulable reason for being followed. The pragmatist's extension says every deviation from a practice should have a specific, articulable reason for being made. Those are the same test applied in opposite directions, and together they make one standard: every engineering decision, whether it follows a principle or deviates from it, requires a justification that references the specific context.</p>
<p>The cargo cult engineer fails the first test by following practices without reasons. The purist fails the second by refusing to deviate without acknowledging that reasons could exist. The pragmatist passes both.</p>
<h2 id="pragmatism-as-the-harder-skill">Pragmatism as the Harder Skill</h2>
<p>This framing explains why pragmatism is harder to develop than either purism or recklessness.</p>
<p>Recklessness requires no knowledge of principles at all. You do what seems easiest.</p>
<p>Purism requires knowledge of principles without any judgement about their applicability. You learn the rules and you apply them, which feels rigorous, in the same way that applying a formula without checking whether its assumptions hold is rigorous. It's consistent without being correct.</p>
<p>Pragmatism requires knowledge of principles <em>and</em> the ability to evaluate their relevance to a specific context. You have to understand what problem a principle solves well enough to recognise when that problem is absent, which means understanding it more deeply than the purist does.</p>
<p>The purist knows <em>that</em> you should separate concerns. The pragmatist knows <em>why</em>, that different rates of change in the same unit create cascading modifications, and can therefore spot the situations where the rates of change are actually the same and separation would add complexity for nothing. That's why I think pragmatism is better understood as a deeper engagement with principles than a looser one. The pragmatist doesn't care less about good engineering; they care enough to distinguish between the principle and the context that gives it value.</p>
<h2 id="a-decision-framework">A Decision Framework</h2>
<p>When I'm evaluating a decision like this, three questions do most of the work. What specifically am I trading off, not vaguely "code quality" but precisely which property of the system, which future capability, am I choosing to forgo? If this turns out to be wrong, how expensive is it to reverse, an afternoon's work or something baked into the architecture? And can I explain it in six months, not just to a colleague but to my future self who has forgotten the context?</p>
<p>If all three answers are clear, the decision is defensible whether it follows the principle or deviates from it. If any is vague, that's a signal to think harder before committing.</p>
<p>Which raises the obvious next question. If every engineering decision requires a justification that references context, how do you know whether the justification is actually correct? It isn't enough to have a reason. The reason has to be testable, and that's where the series goes next.</p>]]></content:encoded>
    <category>philosophy</category>
    <category>engineering</category>
    <category>pragmatism</category>
    <category>architecture</category>
  </item>
  <item>
    <title>Knowing How Isn&apos;t Knowing Why</title>
    <link>https://davidshortland.dev/writing/knowing-how-isnt-knowing-why</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/knowing-how-isnt-knowing-why</guid>
    <pubDate>Sun, 15 Mar 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>I started code reviewing for a junior developer a few months ago. It taught me more about my own understanding than I expected, and revealed a model of learning that changes how I think about software knowledge.</description>
    <content:encoded><![CDATA[<p>A few months ago I started code reviewing for a junior developer on our team. I expected it to be straightforward: read the code, spot the issues, talk them through on a call. I'd been writing production software for a while, and how hard could it be to explain what I already knew?</p>
<p>It turned out to be one of the most revealing experiences of my career so far, and not because the code was difficult. The act of teaching exposed gaps in my own understanding that I hadn't known were there.</p>
<h2 id="the-gap-between-doing-and-explaining">The Gap Between Doing and Explaining</h2>
<p>There's a specific moment that stays with me. I was reviewing some code where the junior had written a service that mixed data fetching with business logic. I knew instinctively that this was wrong, I could feel it, and if I'd been writing the code myself I'd have separated those concerns without thinking about it at all.</p>
<p>Then we got on a call to discuss it and I froze. "Because it's better" isn't feedback. "Because separation of concerns" just names the principle without explaining it. What I needed was the <em>specific reason</em> the separation mattered here: what would go wrong without it, what it would cost us later, what it would make easier.</p>
<p>That moment taught me that there's a difference between being able to apply a rule and being able to explain why the rule exists.</p>
<figure><img src="https://davidshortland.dev/writing/images/stages-of-understanding.png" alt="The stages of understanding" width="800" height="380" loading="lazy" /><figcaption>The stages of understanding</figcaption></figure>
<h2 id="how-understanding-develops">How Understanding Develops</h2>
<p>Through code reviewing, and through a lot of conversations with my mentor about this exact problem, I've come to think about software knowledge as moving through stages that are easier to describe than to name.</p>
<p>The first is learning by accumulation. You watch someone more experienced, you see them make a choice, and later you see them make the same kind of choice in a different context, and your brain starts to pattern-match without anyone ever stating the rule. This is how my mentor taught me. Not by explaining SOLID principles, but by walking through specific instances of applying them in real code, so the learning stayed implicit: here's how I structured this service, here's why I split this module, here's what I changed.</p>
<p>At some point you can apply the pattern yourself. You write code that separates concerns, keeps functions small and names things well, and if someone asks <em>why</em>, you struggle to get much past "it feels right" or "that's how it should be done." You've internalised the rule without being able to externalise it. Most competent developers sit here for years and it's a perfectly functional place to be.</p>
<p>The jump is when you can apply the rule, explain the principle behind it, describe the contexts where it does and doesn't apply, and generate new instances that illustrate it. You've moved from knowing-how to knowing-that. You can defend it, and, more usefully, you know when to break it.</p>
<p>That jump is what code reviewing forced on me.</p>
<h2 id="the-feedback-loop">The Feedback Loop</h2>
<p>I originally had a simpler model in my head. You do first, then you read to understand what you did. Practice, then theory. I've come to think that's too linear.</p>
<p>What actually happens is more like a feedback loop. You see instances and develop tacit knowledge. Then you read something, an article about dependency injection or a chapter on domain-driven design, and it <em>clicks</em>, because you've already felt the problem it solves. That reading reshapes how you see the next instance. You apply the refined understanding, hit a new edge case, and go back to reading with a sharper question.</p>
<p>Reading without doing produces cargo cult understanding. You can recite the principles and you've never felt the pain they address. You know you should favour composition over inheritance, and you've never been burned by a deep inheritance hierarchy that turned a simple change into a cascade through twelve files.</p>
<p>Doing without reading produces superstition. You know <em>that</em> something works and you might well attribute it to the wrong cause. You always write small functions because a senior once told you to, and you think it's about readability when it's actually about testability. The practice is correct while the mental model underneath it is wrong, and wrong mental models eventually lead you to apply the rule where it doesn't help, or miss it where it would.</p>
<p>The strongest developers alternate between the two quickly. They try something, read about why it worked, try a variation, read a different perspective. The theory and the practice aren't sequential; they're interleaved, each one sharpening the other.</p>
<h2 id="what-code-reviewing-taught-the-reviewer">What Code Reviewing Taught the Reviewer</h2>
<p>The irony is that reviewing code for a junior developer pushed <em>me</em> to a deeper understanding of several concepts I thought I already had.</p>
<p>When I had to explain why we inject dependencies rather than instantiate them directly, I realised my own understanding of dependency injection was more mechanical than principled. I knew the pattern. Articulating the specific benefit required me to think about it more carefully than writing the code ever had.</p>
<p>When I had to explain why a certain function should be extracted, "it's too long" wasn't going to do. I had to identify the actual reason: this function is doing two things with different rates of change, and when one changes, the other shouldn't have to.</p>
<p>Each of those explanations forced a precision of thought that writing code alone never demanded. The junior's questions were the best forcing function I'd come across, not because they were sophisticated but because they were honest. "Why?" is the most powerful question in software development, and it's the one we stop asking once we reach unconscious competence.</p>
<h2 id="the-implication-for-how-we-teach">The Implication for How We Teach</h2>
<p>This has practical consequences for how learning gets structured in software teams.</p>
<p>Don't start with the textbook. If someone hasn't felt the pain of tightly coupled code, explaining the dependency inversion principle to them is noise, theoretically correct and practically meaningless. Let them write the tightly coupled code, let them experience the change that cascades everywhere, and <em>then</em> show them the principle, at the point where it has a hook to attach to.</p>
<p>Don't stop at the doing either. A team that only learns through osmosis, watching seniors and picking up habits, will produce capable practitioners who can't explain their decisions, which is fine right up until someone has to make a decision in unfamiliar territory where there's no pattern to match against. That's when the rationalised understanding matters.</p>
<p>The opportunities for that kind of deeper learning are obvious once you look: code reviews, pair programming where the more experienced person narrates their thinking, architecture decision records where you have to write down <em>why</em> you chose an approach, and team discussions where practices get questioned rather than assumed.</p>
<h2 id="the-test">The Test</h2>
<p>Here's a test I now apply to myself. For any practice I follow, can I explain what I do, why I do it, and when I would stop doing it?</p>
<p>"I write unit tests" is a what. "Because they let me refactor with confidence" is getting somewhere. "When the cost of maintaining them exceeds the confidence they provide, which happens with highly volatile UI code, I'd stop" is the full thing.</p>
<p>The junior developer doesn't know it, but their code reviews have been a great learning experience for me. They didn't teach me new techniques. They made me understand the ones I already had.</p>]]></content:encoded>
    <category>philosophy</category>
    <category>mentoring</category>
    <category>learning</category>
    <category>engineering</category>
  </item>
  <item>
    <title>Cargo Cult Software Engineering</title>
    <link>https://davidshortland.dev/writing/cargo-cult-software-engineering</link>
    <guid isPermaLink="true">https://davidshortland.dev/writing/cargo-cult-software-engineering</guid>
    <pubDate>Sat, 14 Mar 2026 12:00:00 GMT</pubDate>
    <dc:creator>David Shortland</dc:creator>
    <description>Richard Feynman warned about scientists who follow the form of science without the substance. The same problem is everywhere in software: teams adopting practices they don&apos;t understand, hoping the results will follow.</description>
    <content:encoded><![CDATA[<p>Last year a mentor of mine recommended I read Richard Feynman's 1974 commencement address at Caltech, the one about what he called "cargo cult science". It changed how I think about work.</p>
<p>During the Second World War, Pacific islanders had watched military planes land on improvised airstrips, delivering cargo: food, equipment, supplies. After the war ended the planes stopped coming, and some islanders built replica runways out of bamboo, lit signal fires, and carved wooden headphones to wear while sitting in control towers they'd built from straw. They'd replicated the form perfectly and no planes came.</p>
<figure><img src="https://davidshortland.dev/writing/images/cargo-cult-runway.png" alt="The form is perfect. But no planes land." width="800" height="420" loading="lazy" /><figcaption>The form is perfect. But no planes land.</figcaption></figure>
<p>Feynman's point wasn't about the islanders. It was about scientists who follow the rituals of scientific inquiry, the conferences and the papers and the methodology sections, without the intellectual honesty that makes science actually work. They do everything that looks right, and the planes still don't land.</p>
<p>I think about this regularly in software engineering.</p>
<h2 id="the-rituals-we-perform">The Rituals We Perform</h2>
<p>Watch a typical software team and you'll see rituals everywhere. Daily standups where everyone recites what they did yesterday without anyone actually listening. Sprint retrospectives that produce "fugazi" action items nobody follows up on. Code reviews that check formatting but not logic. Architecture decision records that get written after the decision is already made. Story points that get reported up to management as if they were units of measurement.</p>
<p>Each of these practices exists because someone, somewhere, did it for a real reason and it worked. Standups originated in teams that genuinely needed to coordinate across dependencies every morning. Retrospectives were invented by teams that took continuous improvement seriously. Code reviews catch real bugs, when the reviewer actually reads the code. Adopt any of them without understanding the reason underneath and what you get is the bamboo runway: the ritual, and none of the result.</p>
<h2 id="why-this-happens">Why This Happens</h2>
<p>The pattern is predictable. A successful company publishes a blog post about how they work. This is how Spotify organises engineering teams. This is how Google does code review. This is how Netflix handles deployments. The industry reads it and copies the form: the squad model, the review checklist, the deployment pipeline.</p>
<p>What they don't copy is the context. Spotify's squad model emerged from specific scaling challenges with specific people and specific technical constraints. Google's code review culture is embedded in decades of institutional knowledge and tooling. Netflix's deployment confidence comes from years of investment in chaos engineering and observability.</p>
<p>Lift a practice out of one context and drop it into another without understanding why it works and you have the cargo cult problem exactly. You've built the runway, and the planes aren't coming, because the planes were never about the runway. They were about the logistics network and the supply chain and the war effort behind them.</p>
<h2 id="what-it-looks-like-in-practice">What It Looks Like in Practice</h2>
<p>I see this most clearly with Agile. The Agile Manifesto was written by people who valued <em>individuals and interactions over processes and tools</em>. Twenty years later, "being Agile" mostly means buying Jira licences and having a certified Scrum Master run your ceremonies. The form is immaculate. What goes missing is the substance, which was always the willingness to adapt, to communicate honestly, and to deliver working software frequently because you care about the outcome.</p>
<p>Microservices are the other obvious example. Amazon and Netflix decomposed their monoliths into services because they had specific scaling and organisational problems that monoliths couldn't solve, and they did it gradually, painfully, over years. The industry cargo-culted the result instead: successful companies use microservices, therefore we should use microservices. Teams of five people split their simple CRUD application into twelve services, added a message broker, a service mesh and distributed tracing, then spent the next year debugging network issues that didn't exist when it was one application.</p>
<p>The same thing happens with infrastructure as code, with test-driven development, with domain-driven design, with every practice that has a name. Having a name is what makes the form easy to adopt. The understanding is the part that takes effort.</p>
<h2 id="feynman-s-antidote">Feynman's Antidote</h2>
<p>Feynman's prescription was simple: intellectual honesty. He called it "a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty." The first principle is that you must not fool yourself, and you are the easiest person to fool.</p>
<p>In software terms that means asking uncomfortable questions. Instead of "are we doing standups?", ask whether the standups are actually helping anyone coordinate. Instead of "do we have microservices?", ask whether the service boundaries are in the right places, and how you'd find out if they weren't. Instead of "are we Agile?", ask when the team last changed its process because the process wasn't working.</p>
<p>It also means being honest about what you don't understand. Adopting a practice because someone you respect recommended it is fine, as long as you know that's what you're doing and you're watching for signs that it isn't working in your context.</p>
<p>And it means measuring outcomes rather than activities. A team that ships reliable software and responds quickly to what its users need is engineering well, whether or not it has sprints and standups and story points. A team that performs every ceremony perfectly and ships late and buggy has built a beautiful bamboo runway.</p>
<h2 id="the-practices-i-ve-kept">The Practices I've Kept</h2>
<p>I'm not arguing against practices; I'm arguing against unreflective adoption.</p>
<p>I try to make sure that every practice I follow has a specific, articulable reason behind it. When I can't explain why I do something, I take that as a signal that I might be cargo-culting it.</p>
<h2 id="the-test">The Test</h2>
<p>Here's a simple test for whether a practice is genuine or cargo cult in your team. Can the people performing it explain why they're doing it, in terms of the specific problem it solves for them?</p>
<p>Answers like "we do standups because Scrum says so", "we use microservices because that's the modern architecture" and "we write tests because best practices" don't count.</p>
<p>Compare those with this: "We do a quick sync each morning because the data pipeline team and the frontend team keep stepping on each other's database migrations, and this catches it before it becomes a merge conflict." That's a real reason, and it describes a practice that solves a problem the team actually has.</p>
<p>If you can't articulate the reason, you have two options. Work out what the reason is, or stop doing the thing. Either one beats carving more wooden headphones and waiting for planes that aren't coming.</p>]]></content:encoded>
    <category>philosophy</category>
    <category>engineering</category>
    <category>agile</category>
    <category>architecture</category>
  </item>
  </channel>
</rss>
