Performance

What vkML currently costs, where the time actually goes, and which of those numbers are trustworthy.

The machine

Validation layers are on by default. Every figure below includes that cost, which is the honest default to measure — but it is worth knowing before comparing against anything else.

The headline number

Same CNN architecture, same batch, same optimiser, CIFAR-100:

A 36-compute-unit discrete GPU is 4% slower than PyTorch on a CPU. Since torch on a GPU would be many times faster than torch on a CPU, the real distance to parity is not the 1.04× this shows — it is that whole further multiple.

Where the time goes: starvation, not slow kernels

Batch scaling separates fixed cost from arithmetic. If per-sample time falls as the batch grows, the device was idle waiting for work:

Per-sample cost falls 3.9×. The strongest single line is batch 128: it does twice the work of batch 64 in less wall time, which only happens when fixed per-step cost dominates arithmetic.

At the batch size the examples use, most of a training step is overhead. Batch scaling infers that; it cannot say how much, or which part. Measuring it directly is what the next section does.

Measured directly: where a CIFAR step's time goes

Batch scaling and device substitution both locate the problem without closing it. vkML publishes each dispatch as a measured interval and each kernel choice as a decision, both carrying the same dispatch identity, so a consumer can join the two and account for a whole step:

copypython examples/cifar100/train.py --attribute 20

20 steps at batch 64 after 20 warm-up steps, best of 5 rounds, on the RX 5600M. One round of identical work varies by 20% on this machine — GPU time and host time do not scale together, so a single round distorts the split and not only the total.

matmul is 30.9% and the next line is 6.0%. When this measurement started it was not the largest line at all. Five rounds of measure–fix–remeasure have taken every other kernel below a tenth of a step. The two sum rows are one kernel reported under the two structures it chooses between — the profiler's cost joined to the planner's decision, which is what the dispatch identity exists for.

Two things only direct attribution could say. The 20 steps made 160 submissions — 8 each, of which 4 carry compute; the others are two uploads and the two behind .item(). And GPU idle time inside submissions is 0.2%: the barriers between dispatches are not the cost.

What it paid for, five times over

The first table this produced showed the optimiser spending 24 of a step's 39 submissions on eight parameters. Every parameter's update is independent, so the optimisers were rewritten to build all of them first and realise them together: 1.5–1.9× on the optimiser phase across all seven configurations, parameters bit-identical.

Re-attributing after that change — rather than assuming it had finished the job — showed backward() doing the same thing one layer down, and worse: 11 submissions per backward pass, five of them carrying a single dispatch. Two causes. The loop that deposits each parameter's gradient realised them one at a time. And two backward rules, for max-pooling and slicing, called realize() unconditionally where every other rule realises only in eager mode — so in the lazy mode both examples train under, they cut the graph three times per pass in a three-block CNN. 11 → 1, gradients bit-identical.

Re-attributing again pointed at what was left: one parameter assignment per parameter, each its own submission at a measured 40–80 µs. That one had been written down as blocked on two larger changes — and re-checking found the first had already been dissolved by the optimiser rewrite, and the second was never on the path. assign_ did not need to become part of the graph; it needed to stop being one submission per call, and only the backend knows what a submission is. The copy primitive now takes a list, and an optimiser step costs a constant three submissions regardless of how many parameters the model has.

At that point the scheduling was done and the profile pointed at kernels — and all three it named turned out to be addressing-bound rather than memory-bound, a diagnosis only a comparison against an equal-traffic kernel could make. The reduction launched one workgroup per output where it wanted one lane; im2col, col2im and max_pool2d spent more time computing which four bytes to move than moving them.

13.57 ms → 8.04 ms, a 1.69× end-to-end speedup, and every result is bit-identical. Nothing here changed what vkML computes — only when, and in what order the addresses are worked out.

Note the host share rising in the last three columns while the step gets faster. The same host cost against a smaller step is a larger fraction of it. A percentage is a ratio and this one has two moving ends; the milliseconds are the thing to read.

The same report over any code of your own:

copyimport vkml
from vkml.attribution import capture

with capture() as cap:
    for _ in range(20):
        train_one_step()

print(cap.report().table())

capture turns on profiling, submission retention and decision recording for its duration and turns them off again on exit. It is a consumer: it joins what the profiler and the planner each publish, and neither of them knows it exists.

Three independent observations agree

• MNIST at batch 64 trained at 4.41 s/epoch on the 36-CU discrete card and 4.35 s/epoch on the 6-CU integrated one. Six times the compute, no difference — the workload never reaches the arithmetic. Both are 2.18 / 1.99 s/epoch after the scheduling work above, and the tie breaks above batch 256.
• A submission costs about 105 µs against 9 µs for a dispatch.
• The optimiser is 62.7% of an MLP step, across 12 submissions.

CIFAR-100's CNN spends 96.3% of its step in the forward/backward/optimiser region rather than in batch loading and transfer. That is a statement about the data path, not about the GPU: attributing inside that region puts 34.3% of the CNN's step outside every submission window too. Submission overhead is not confined to small models.

Two GPUs, identical results

The accuracies are identical to the last digit on two different GPUs. That is the determinism contract holding across hardware, not a coincidence.

The timings show the same split as the batch scaling: on the compute-bound CNN the 6-CU part is 3.5× slower, as its compute-unit count predicts; on the MLP at batch 64 it ties the discrete card.

That tie was read as a framework problem for a long time, and it is a property of the batch size. One epoch of the MLP, after the scheduling work above:

Above batch 256 the two separate cleanly and in the right direction, and the gap grows with the batch. A 784→128→10 MLP at batch 64 is 0.61 ms of arithmetic — small enough that a fixed per-step host cost dominates it no matter how small that cost gets. Every size above halved from this section's work; the tie at 64 did not move, and could not.

Against PyTorch, on accuracy

The CPU backend is not for training

Measured: 4 CIFAR steps took 12.66 s of compute, about 3.17 s/step. The full set is 782 steps per epoch, so one epoch is roughly 41 minutes and ten are about 6.9 hours. It did not complete a single epoch on even 2,000 examples within a 550-second budget.

That is the cost of the backend being a deliberately naive correctness oracle. Use it to check answers, not to get them.

What is not measured

There is no per-kernel attribution today. vulkan_last_profile returns submission-level ('submit', ms) pairs, so none of the evidence above identifies which kernel dominates a step. Everything here is indirect — batch scaling, device substitution, submission counting. That is enough to locate the problem and not enough to close it.

Timestamps are supported by the device; what is missing is recording them around each dispatch and aggregating by kernel name. Until that exists, treat any claim about a specific kernel's share of a step as unproven.

If you benchmark this yourself

• State the batch size. A 4× difference in per-sample cost sits between batch 64 and 512, so a figure without one is not comparable.
• Say whether validation layers were on. They are on by default.
• Report the minimum across process runs, never the mean. Noise here is one-sided — something slowing a run down is a cause, something speeding it up is not.
• Run a same-binary control. A previous measurement in this project showed −19% for a change whose control showed a ±18.9% noise floor, and another showed +3.7% for a padding-only change whose control moved −6.5% — which no padding can cause.