Skip to content

Testing#

Tests are the executable specification. They define the behavior we want, prove we built it, and catch the day it breaks. They are written as part of the work — not bolted on afterward.

Test-first#

Define the test when you define the behavior. Writing the test first forces you to design the interface from the caller's side, and it gives you an unambiguous definition of done: the test passes.

For logic with branches, edge cases, or anything involving money, time, or security — red-green-refactor is the default. For trivial glue code, judgment applies; dogma does not.

See Test-Driven Development.

Testable locally#

A developer or an agent must be able to run the full suite on their own machine — no cloud resources, no special access, no secrets that cannot be mocked. If you cannot test it locally, you cannot reason about it in your editor, and the cost of every change goes up.

This is a design constraint, not a nicety. When building anything new, ask early: can someone run this locally? If the answer is no, the design is wrong.

The test pyramid#

Most tests should be fast and narrow; few should be slow and broad.

  • Unit — the foundation. Fast, isolated, no I/O. The bulk of the suite.
  • Integration — components together, real boundaries. Fewer, slower, higher-value.
  • End-to-end — the whole system. Fewest of all; reserved for critical paths.

Inverting the pyramid — leaning on slow, brittle end-to-end tests — produces a suite that is painful to run and quick to be ignored.

Don't mock what you don't own#

Mock the boundary you control, not someone else's surface. Wrap a third-party API or SDK behind your own thin interface and fake that. Mocking the vendor's client directly only freezes your assumptions about how it behaves — the mock keeps passing after the real dependency changes, and the break reaches production unseen.

Where the real contract matters, back the mocked unit tests with a small integration suite that exercises the live dependency on a schedule, so drift surfaces early.

The oracle must be independent#

A test proves nothing when it computes the expected value the same way the code computes the actual one. If a function derives anchors from headings, and the test reimplements that same derivation to check it, both are free to be wrong together and the suite stays green forever. The expected value has to come from somewhere the implementation cannot influence: the library the real consumer runs, a fixture recorded from the real system, the published specification, or a second implementation written independently.

This is don't mock what you don't own one level up. There the mock freezes your assumption about a dependency; here the assertion freezes your assumption about correctness.

When the oracle is an external tool or service, confirm it is answering the question you think you asked. A wrong mode flag, an expired token, or a renamed field can turn a rich response into an empty one — and every assertion built on it into a vacuous truth. So assert on the oracle too: an empty response is a failure of the check, not a pass (see nothing checked is not a pass). Where two independent oracles are available cheaply, agreeing them once is worth more than trusting either.

Nothing checked is not a pass#

Every assertion over an empty set is true. All links resolve holds trivially when no links were found. Every anchor exists holds when the oracle returned nothing to compare against. A check that iterates over nothing and reports success is not a check — it is a claim with no content behind it, and it is indistinguishable from a working one at a glance.

So make the count part of the result: a check fails when it checked nothing. Count the units examined and treat zero as a failure, with a message that says the check found nothing rather than that everything is fine.

This costs three lines next to the exit, and it covers a whole class at once — the wrong root directory, a filter that matches more than intended, a glob that missed, an empty checkout, a query that returned no rows, an oracle that answered with an empty response. None of those need to be anticipated individually. Each one drives the count to zero, and zero is a failure.

Prove the test can fail#

A check that has never failed has not been checked. Before trusting a new one, make it red on purpose: break the behavior it guards, confirm it fails, read the message it produces, then restore. If breaking the behavior leaves the check green, the check is decoration.

Counting is not a substitute for this. A healthy count only says the check examined something — not that it examined the right property. A conversion that reports every word of the source survived has a large denominator and still says nothing about whether the links in that text resolve. An empty check is caught by counting; a check measuring the wrong thing is caught only by breaking the thing it claims to guard and watching what happens.

Properties of a good test#

  • Deterministic. Same input, same result, every time. A test that passes intermittently is worse than no test — it trains people to ignore failures. No reliance on wall-clock time, network, ordering, or shared mutable state.
  • Isolated. Each test sets up and tears down its own world. Tests do not depend on each other or on run order.
  • Fast. The inner loop is where engineering time is spent; slow tests get skipped.
  • Readable. A test is also documentation. Arrange–act–assert, one behavior per test, a name that states the expectation.
  • One reason to fail. When a test breaks, its name and body should make the cause obvious.

Coverage with judgment#

Coverage is a signal, not a goal. High coverage of trivial code while the hard branches go untested is a false comfort. Aim coverage at the code that carries risk — logic, edge cases, error handling — and don't chase a percentage for its own sake.

Watch for a number that measures something adjacent to what you actually care about. A conversion reporting that every word of the source survived tells you the text is verbatim; it says nothing about whether the links inside that text resolve. The measurement is honest and the conclusion drawn from it is not — which is why the reassuring ones deserve the most scrutiny.

When a bug escapes#

A bug that reached production is a missing test. The fix is incomplete until a test reproduces the failure and then passes — so the same regression can never return silently. Fixing the source without closing the test gap leaves the next regression just as invisible.

Tests run in CI#

Every pull request runs the suite before a human review begins. Validation that depends on a reviewer remembering to check is validation that eventually fails. Automate it once; it protects every PR after. See Shift Left.