Testing and verification

How correctness is established here, and why a green suite is treated as necessary rather than sufficient.

The correctness chain

Correctness is a chain of two links, and each one only works because of the other:

CPU against PyTorch, for semantics. Does relu mean what everyone else means by relu?
Vulkan against the CPU, for kernel bugs. The CPU backend shares vkML's exact semantics, so a mismatch here is unambiguously a kernel defect rather than a difference of convention.

The second link is the valuable one, and it requires the first: comparing a GPU kernel directly against torch would confound a kernel bug with a semantic difference, and the investigation would start in the wrong place.

It also forces a rule that is enforced by a test rather than by convention: CPU support must be a superset of Vulkan support. That broke once, when widening the Vulkan supports() gates for float16 made the GPU accept operators the CPU still rejected — every one of those had a GPU result no oracle could check, and the suite stayed green throughout.

The two suites

ctest --preset release                 # C++: 115 cases, 2677 assertions
python -m pytest tests/python -q       # Python + PyTorch: 1413 tests

The Python suite is where operator agreement is checked, because that is where PyTorch is. The C++ suite covers what has no Python surface — graph ownership, the dispatch grid, shape arithmetic, the environment helpers.

Gates beyond the suites

Gate	What it prevents
`check_layering.py`	A lower layer including a higher one. Has caught a real violation — autograd reaching into api.
`check_push_constants.py`	A push-constant block exceeding the 128 bytes Vulkan guarantees. Guards the defect class that produced 19 failing tests on a driver reporting exactly 128 and nothing at all on a development GPU reporting 256.
`check_cpu_only_build.py`	A test that assumes a GPU. Three CI jobs build CPU-only, so this fails locally rather than in three jobs at once.
`coverage_matrix.py`	An operator quietly losing coverage. Compares against a recorded baseline of accepted gaps; a new gap fails, a closed one only warns.
`mutation_check.py`	Tests that cannot fail.
`check_docs_examples.py`	A documented example whose output was never run.
`check_docs_references.py`	A cited file, line or constant that no longer exists.
`check_docs_links.py`	A broken internal link or anchor.
`check_versions.py`	A version stated in prose disagreeing with the file that decides it.

A green suite proves the tests ran

It does not prove they can fail. That distinction is not academic here: a green suite missed a documented list of real bugs, which is why the project's testing strategy exists as a document.

Two habits follow, and both are treated as standard rather than as extras:

Red-verify every new gate. Break the thing it guards and watch it fail. A gate that has never been seen to fail is a script.
Check for vacuity. A probe that reports "0 differences" may be reporting that it found nothing, or that it ran nothing. One in this project printed nothing because pytest captures stderr; another compared against instrumentation that had been removed. Both looked like passes.

Testing against limits you do not own

VKML_MIN_SPEC=1 python -m pytest tests/python -q

This makes any device report the Vulkan 1.3 Required Limits. It only ever reports limits smaller than the hardware has, so it can make vkML more conservative and never less.

It exists because most of this project's portability bugs were the same shape: a limit asserted against what the development GPU reports rather than what Vulkan guarantees. Push-constant budgets, workgroup counts and subgroup ranges all failed that way, and every one was invisible locally and fatal elsewhere.

ⓘ Note

Run it before claiming a limit is satisfied. It is the cheapest way to find the next instance of the most common bug this project has.

A second driver

The suite also runs against lavapipe, a software Vulkan implementation with a different SPIR-V compiler, a subgroup size of 8 against RADV's 64, and no real memory hierarchy. A different driver is where portability actually breaks — not a different device.

It earns its place: it is what surfaced a case where sign returns -0.0 for +0.0 and NaN on one Mesa version and +0.0 on another, from a shader whose source reads return 0.0.

Measuring anything

The project keeps a separate document listing the instruments that lie, and it is not optional reading before claiming a speedup. The short version:

Minimum, never mean. Noise is one-sided.
Minimum across process runs, not iterations within one — pipeline caches and allocator state persist.
Warm the pipelines, and say whether validation was on.
Run a same-binary control. Twice in this project a measured improvement turned out to be inside the noise floor of an identical binary, and once a padding-only change "improved" by −6.5%, which no padding can cause.