The coverage ratchet

Français · How-to · Published, reviewed by a person · Applies from: 4.2.3

The coverage floors of vite.config.ts only rise. Two checks hold them: Vitest fails when the tests reach less than a floor, and the ratchet fails when a floor lags three points or more behind what the tests reach. This page says what to do when either is red.

What you get

A coverage job green again: every floor met, and every floor within three points of what the tests measure.

Context and versions

The engine repository. The floors date from 4.1.0 "Clarity" (the level measured then); the ratchet, tools/coverage-ratchet.ts, from 4.1.3; its GitHub annotations from 4.1.9.

What is measured: V8 coverage of src/**/*.ts (not *.d.ts), bridge/src/**/*.ts and studio-server/src/**/*.ts, by the unit suite and the asset tests (npm run test:coverage). The floors as of 4.4.0:

The exact values are in vite.config.ts, test.coverage.thresholds, with the reason for each floor below 100 written beside it.

Limit · lines-not-proof Coverage counts lines and branches run, not results checked. The browser-only parts of src/ (the player's UI and the Studio's) are covered by the e2e, not by this measure; whether the tests tell a right result from a wrong one is Mutation testing's question.

Prerequisites

A prepared repository with Python and its image modules (Developer setup): the asset tests run under coverage too.

Where to run the commands

In the engine repository, on your machine. CI runs the same in its coverage job.

Steps

1. Measure.

   # in the engine repository
   npm run test:coverage
   

   It runs the unit suite under coverage and writes .cache/coverage/ (a text summary, coverage-summary.json, an HTML report in .cache/coverage/index.html). Vitest fails here when a floor is not met, naming the file and the metric.

2. A floor not met (the tests reach less than it): your change removed tests or added code no test runs. Add the tests; open .cache/coverage/index.html to see the lines and branches not run. A floor is never lowered to let a change through.

3. Read the ratchet.

   # in the engine repository, after test:coverage
   npx tsx tools/coverage-ratchet.ts            # warns
   npx tsx tools/coverage-ratchet.ts --strict   # fails, as on main
   

   Each line ⚠ <scope> <metric>: floor <n>, measured <m> is a floor three points or more under what the tests reach. The end says ✔ every coverage floor is within three points of what the tests reach, or the count to raise.

4. Raise the floors it names in vite.config.ts (test.coverage.thresholds), to a value at or a little under what was measured, and commit them with the change that earned them. vite.config.ts is a file the maintainer reviews (CODEOWNERS).

Limit · ratchet-scope The ratchet reads only the floors that exist: a file without a floor of its own is held by the global floors alone. A per-file floor applies when the summary holds that file.

Expected result, and how to check it

npm run test:coverage passes and npx tsx tools/coverage-ratchet.ts --strict exits 0. In CI, the coverage job is green with no "coverage ratchet" annotation.

Common errors

Files or data changed

vite.config.ts (the floors); the measure writes only .cache/coverage/.

Recovery or going back

A floor raised too high fails the next run: lower it back to what the tests reach (with the reason in the pull request), since a floor above the measure is a red build, not a promise.

References

• tools/coverage-ratchet.ts, vite.config.ts.
• Which tests run when, CI explained, Mutation testing.

Edit the source of this page · Report an error