Skip to content

← Archive

Reqnroll at Insignis · Part 1 of 2

Who Decided This Rule?

Reqnroll gave my team tests that product can read, and Claude made them cheap to build. That moved the hard part to a question the feature file doesn't answer.


19 November 202612 min read

The first of two essays. This one is about why my team writes Reqnroll tests and what changes when Claude writes them. How We Write Reqnroll Tests is my account of how we write them, with the code.

In my second week at Insignis every unit test I'd written for a feature was green, and I still couldn't vouch for it. The feature crossed several parts of the system, and each test checked one piece with everything around it mocked. None of them ran the journey from the input arriving to the outcome going back out. Someone who'd been there for years could have filled that gap from memory. I'd barely started.

My engineering manager suggested behaviour-driven tests, and we landed on Reqnroll, the open-source fork of SpecFlow for .NET.11 Tricentis ended SpecFlow on 31 December 2024. Reqnroll was forked from it in early 2024, is BSD-licensed, and is still actively developed. I started on it that week, and by my fourth the rest of my team was using it. I wrote down how we build these tests as a skill the team now uses, and Claude does most of the building. That's made them cheap enough that writing the scenarios isn't what slows us down any more. The hard part now is a question the tests don't answer: who decided the rules they check? A feature file reads the same whether a person agreed its rules or a model made them up.

Two Ways to Be Wrong§

A test suite can be wrong in two ways. The code can fail to do what a test says, or the test can say the wrong thing.

Any test that runs the real journey can catch the first kind, Reqnroll or not. The first application I brought Reqnroll to had a four-eyes approval process, where a change needs a second person to approve it, and the scenarios showed that the person who made a change could approve it themselves. Their edit set off further edits that ran asynchronously through Kafka and moved the object's last modifier away from them, so by the time they came to approve it, the check no longer saw them as its editor. That only shows up when the whole journey runs with the consumers catching up behind it, which no unit test does. It was fixed before the software was released.

The second kind is worse, because nothing goes red. If I misunderstand a rule, I write the test from my misunderstanding and the code from the same one, and the two agree with each other for as long as the suite exists. No database catches that. Only someone who knows the rule does, and often that's the product owner, who doesn't read C#. It's also the kind of wrong that's easiest to miss when a model writes the rules as well as the tests.

What catches each kind of wrong: a unit test catches code wrong in one class, the real journey catches code wrong across systems, product reading the scenario catches a rule I wrote wrongly, and nothing catches a rule a model wrote that nobody decided
What catches each kind of wrong: a unit test catches code wrong in one class, the real journey catches code wrong across systems, product reading the scenario catches a rule I wrote wrongly, and nothing catches a rule a model wrote that nobody decided

A Rule Product Could Read§

Reqnroll is for the second kind. A scenario is a few lines of plain English in a .feature file, each bound to a C# method that runs it, so the rules come out as sentences product can say yes or no to, and QA can read without opening the code.

I was new to the domain, so I checked each rule with product as I went. When a case was unclear, I sent a screenshot of one scenario and got a yes or no. One had something moving from one state to another, and the answer was no. That transition wasn't allowed at all, and my scenario said it was.

I was very glad when that came back. If it hadn't, the code and my test would both have allowed a transition the business doesn't, and the suite would have stayed green while saying so.

To show the same thing in code you can run, I've published a small library holds API to go with this essay, library-holds. I settled its four rules before any Gherkin was written, and Claude built the app and its tests from them.

  • A returned copy goes to the member who has waited longest for that title and format.
  • A copy only fills a hold for the same title in the same format.
  • The member can collect until closing time, 20:00 UK time, on the 7th day after the copy reaches the hold shelf.
  • A copy that fills no hold goes back on the shelf and nobody is told anything.

In the library, my mistake would look like this (it isn't in the repo, which doesn't model holds running out). Say I'd assumed that a member who misses the collect-by date keeps their place in the queue.

Scenario: A copy that isn't collected in time goes to the next member
    Given Ana placed a hold on "The Hobbit" in hardback at 2027-03-01T09:00:00Z
    And Ben placed a hold on "The Hobbit" in hardback at 2027-03-02T09:00:00Z
    And a copy matching Ana's hold was returned at 2027-03-10T14:00:00Z
    When closing time passes on 2027-03-17 without Ana collecting the copy
    Then the copy fills Ben's hold
    And Ben is told the copy is ready
    And Ana's hold is waiting again

A librarian would pass every line but the last. A hold that runs out is finished, and if Ana still wants the book she places a new one at the back of the queue.

As a test method, that scenario would be called something like Uncollected_copy_goes_to_the_next_member, which describes only the part that's right, and the test behind it would stay green for as long as the code agreed with me. The scenario spells out every outcome, and at work the one product said no to was among them.

Why Not Plain xUnit?§

A Gherkin sceptic would ask why these aren't xUnit tests calling the same code, with well-named methods. The compiler would check every call, which it can't do for step text. Even as a developer, though, I find the scenarios easier to read than the C#, and the second essay puts one rule both ways, side by side. Aslak Hellesøy, who created Cucumber, has argued that it's a collaboration tool first. We don't do the full version he describes, where product write scenarios with us before the code exists, but the conversation he's arguing for is the one that caught my state transition.

The stronger opponent is the setup most teams already have, acceptance criteria agreed in a ticket with integration tests behind them. Product can say yes or no to a ticket as easily as to a scenario. The difference is that the ticket and the test are two artifacts, and once the ticket closes, nothing makes them agree. In a feature file the sentence product agreed is the test that runs. When someone changes a rule later, the pull request shows a changed English sentence that a reviewer can read and send back to product, where a changed integration test shows only changed C#.

Cheap to Build§

The app under test, the containers, the stand-ins for external systems, the drivers and the bindings are what used to make tests like these expensive enough that teams didn't write them. Claude is very good at all of it. The sample's whole history, from the first commit to the experiment below, fits in one morning.

The same sentences product reads are what I hand to Claude. The Rule: lines are a compact statement of what was decided, and a feature file the model wrote is far quicker to review than the C# behind it.

The first rule in the skill is that the model doesn't invent business rules. It gets them from a person, or from a ticket a person points to. Only once those are agreed does it propose edge cases, as a list to accept or reject one at a time. It also says to read the production code to understand the behaviour, but never to turn what the code does into a rule without someone confirming it's intended, because the code may be the bug.

The risk is the flip side of how easy it has become. Ask for scenarios and you get a tidy feature file, with rules and boundary cases, all green. If nobody decided those rules, you're covered by what the model decided. That's the same false confidence as the mocked unit tests I started with, only harder to spot. The feature file is the readable part, so it's the part a reviewer trusts and the easiest to wave through.

The sample has an example. While building it, Claude made "only a waiting hold can be cancelled, otherwise the API returns 409" part of the behaviour, a reasonable choice that isn't one of the four rules I settled, and no scenario covers it.

An Experiment§

To see what happens without agreed rules, I planted a bug in the sample. The query that finds waiting holds for a returned copy forgets about format.

.Where(h => h.Status == HoldStatus.Waiting && h.Title == title)

If Ana is waiting for The Hobbit in hardback and a paperback comes back, the paperback fills her hold. All 22 unit tests still pass, because they mock the repository the query lives in.

I gave a fresh Claude, with no knowledge of the rules or the planted bug, a single-commit repository holding that code and its test infrastructure, with the feature file, the step definitions and any list of rules removed. It had the skill my team uses. The request was the one I'd type: /reqnroll-scenarios, then add Reqnroll scenarios covering how returned copies fill holds, and make sure they pass. Nobody was there to answer its questions. What it wrote is on the experiment branch, exactly as it left it.

It didn't write a green spec with the bug in it, though that's what I'd set the experiment up to show. Comments and a unit test name said holds are matched on title and format, so it wrote the format scenarios for the correct behaviour and marked them as a known bug.

# Known defect, ignored until it is fixed (no ticket yet: raise one and name it here).
# HoldRepository.FindWaitingHoldsAsync filters waiting holds on title only, not format, so the
# paperback fills Ana's hold for the hardback.
@ignore
Scenario: A copy in a format nobody is waiting for goes back on the shelf
    Given Ana placed a hold on the hardback of "The Hobbit" at 2027-01-04T09:00:00Z
    When a paperback copy of "The Hobbit" is returned at 2027-03-01T09:00:00Z
    Then the copy is back on the shelf
    And nobody is told anything
    And Ana's hold is still waiting

It left the production code alone. Its run was 16 passed and 2 skipped. With the one-line fix, all 18 pass.

The skill stops for a person twice, to agree the Rule: lines and to review the feature file before any bindings. With nobody there, it wrote down what it would have asked at each point and carried on, with these rules.

Rule: A returned copy fills the hold of the member who has waited longest for that title in that format
Rule: A returned copy that no waiting hold is for goes back on the shelf, and nobody is told anything
Rule: The member whose hold is filled is told it is ready, and can collect it until 20:00 UK time on the seventh day after the copy reaches the hold shelf

They're close to my four, and nobody agreed any of them. It took them from the code's comments and its own reading, and its account of where they came from began "Nobody could confirm these, so none of them is an agreed rule."

That account ends with the session, though. What gets merged is the feature file, and its Rule: lines read exactly like decisions. Nothing in it records which rules a person agreed and which the model inferred. A developer who accepted the green run without reading the report would merge a spec nobody decided, and the next person to open it couldn't tell.

Even when the model gets a rule right, the feature file looks just as decided as it would if the rule were wrong. To see whether it needed those comments, I ran the experiment again with every comment and test name that mentioned format taken out, and it caught the bug anyway. It wrote the same kind of @ignore scenario and Rule: lines close to my four, and nothing in the file it left on the second experiment branch says nobody agreed them.

Its clues were a format parameter the query never uses, a database index on title and format, and a README that says a hold is on a title in a format. All three were written from the rules I'd agreed, so the model still depended on them. It just found them in quieter places than a comment.

Across both runs it listed the same edge cases it would have asked about: two holds placed at the same instant, titles that differ only in case, a copy returned again while it's on the hold shelf, and the notification provider being down. None of my four rules settles them.

The change I'm proposing for the skill is that a rule nobody confirmed goes into the feature file with a tag saying so, the way @ignore keeps a known bug in view, and the tag only comes off once a person agrees the rule. Taking it off would be a one-line change in a pull request, where a reviewer would see it.

What It Costs§

Scenarios are slow next to unit tests. In the sample, the 22 unit tests take about 150 milliseconds and the 24 scenarios take about nine seconds, and CI needs Docker. Most of those nine seconds are start-up, though. A single scenario on its own takes about seven, nearly all of it starting Postgres in a container and the app on top of it, and each scenario after that adds about a tenth of a second. The start-up happens once per feature, not once per scenario, so the suite gets slower only slowly as it grows. That's the price of running the real journey, and any integration suite built on Testcontainers pays it, whether what it starts is a database, Kafka or both.

Reqnroll's own share is another layer for a team to learn, and step text that nothing checks. A sentence can be bound to code that checks something else and it'll still pass, so keeping the bindings honest costs review attention for as long as the suite exists. The second essay covers how we do it.

What I Have Now§

When I change that first feature now, the suite either stays green or tells me which rule I broke, in a sentence I could show product. It's only as good as its bindings and stand-ins. But that's the check someone with years in the system does from memory, and it no longer depends on anyone's memory, mine included.

The sample's README lists four gaps that nobody has decided. The one I'd decide first is a copy that's scanned in twice. Returned while it's already on the hold shelf for Ana, it fills Ben's hold as well, and both of them are told the same book is waiting. The code already does something there, and a scenario written from the code would say the same thing and pass. Whether that's right is a librarian's call, and nobody's made it yet.

Notes

  1. Tricentis ended SpecFlow on 31 December 2024. Reqnroll was forked from it in early 2024, is BSD-licensed, and is still actively developed. ↩︎