Testing to the Risk, Not the Checklist

On one project I worked on, three constraints decided the entire testing strategy before I wrote a single test: a small team where testing time competed directly with delivery time, no integration test environment for the third-party services the system depended on, and a small group of domain experts available for structured testing of the finished thing.

Tests are only valuable if they represent genuine attempts to disprove an assumption, which raises the question of which assumptions, and at which boundaries. A project has finite time and you cannot test everything at the boundary, so the decision about where to spend the effort is a resource allocation question rather than a testing one, and it needs a different analysis from "what are the most important features?"

The Coverage Trap

Coverage as a primary metric creates a specific failure mode. You optimise for the number, so you write tests for the code that is easiest to test: utility functions, simple transformations, cases that are already well understood. The number goes up. The confidence it represents does not.

This is the inverse of what you want. The parts of a system that are easiest to test are usually the parts with the least risk. Pure functions with simple inputs and outputs behave predictably. The code that integrates with external systems, that depends on user behaviour, that sits at the junction of multiple domain concepts: that is where the risk lives. That code is harder to test, and the coverage metric gives you no pressure to go there. Test to a coverage target and you build a suite that protects you from the problems you already understood, while the problems you didn't understand stay invisible.

A Constraint-Driven Strategy

From those three constraints, the strategy took three layers, each chosen for the specific kind of confidence it could provide.

The first layer was unit testing for the small number of pure, algorithmically substantive functions: code that took clearly defined inputs, applied non-trivial logic, and returned clearly defined outputs with no dependency on infrastructure or external state. These were the right target for automated unit tests because they were deterministic, isolated, and complex enough that a test told you something you didn't already know. The tests were written at the boundary conditions and failure modes rather than confirming the happy path: the Popperian principle applied directly.

The second layer was controlled-data testing for the integration points with external services. No sandbox environment existed for those services. Testing against them with live data was not safe. The approach instead was to wrap each integration behind an interface, build test fixtures from production-shaped data, and test the integration logic against those fixtures. This is not as strong a guarantee as a real integration test environment. It was the strongest guarantee available given the constraints, which is the relevant comparison.

The third layer was operator-led user acceptance testing. Domain experts ran scripted test cases against a production-shaped environment. This layer did what neither of the others could: it validated whether the system fit the actual workflows of the people who would use it, including the workflows that no written requirement had anticipated.

Deliberate Exclusions

Two layers were explicitly excluded: an automated front-end unit suite and an end-to-end browser-driver suite.

Both were deliberate decisions, recorded with their justifications. The front-end unit suite would have tested HTML structure and component wiring rather than business logic, at a maintenance cost disproportionate to the risk it mitigated for an internal tool with a small, available user group. The browser-driver suite would have required infrastructure and ongoing maintenance that the risk profile didn't warrant given that the same operators who would notice regressions were available for direct testing.

The exclusions matter as much as the inclusions.

A testing strategy with only inclusions is a list. A real strategy records what you are not testing and why, because that's where the resource allocation decision becomes visible, and if you never record what you chose to leave uncovered you can't defend the allocation later or notice when the risk profile shifts enough to warrant revisiting it.

A deliberately excluded test is an engineering decision. An accidentally excluded test is a gap. The difference is whether you can explain it.

Sizing to the Risk

What makes this a risk allocation problem rather than a testing problem is that risk is not uniform across a codebase. Some code failing causes a minor inconvenience. Some code failing causes incorrect results that flow into every downstream workflow, data loss, or a user-facing outage that takes a day to diagnose.

Testing effort should track that distribution, not the distribution of how easy the tests are to write.

The useful question before writing any test is: what is the cost if this code is wrong? If the answer is "a minor UI inconsistency that a user notices and reports," the test is probably not a good use of time. If the answer is "incorrect categorisation of data that every downstream report depends on," you want that test, and you want it designed to probe the conditions where the logic is most likely to fail, not to confirm the case you already know works.

Coverage is a side effect of writing that kind of test. It is not the goal.

The goal is a suite where every test encodes a genuine risk: something that, if broken, would cause a real problem. A suite like that does not need to be large to be valuable. It needs to be honest about what it is and is not covering, and why.