Tensors, storage and views

What a tensor owns, what it merely refers to, and where the boundary between the two is drawn.

Three types, one value

A value in vkML is described by three objects with sharply separated jobs. Keeping them separate is what makes a view cost nothing and a layering rule enforceable.

Type	Owns	Declared in
`Tensor`	A handle to a graph node. Nothing else.	`include/vkml/api/tensor.h`
`Shape`	Extents and strides. No data.	`include/vkml/core/shape.h`
`Storage`	A refcounted block of device memory.	`include/vkml/core/storage.h`

Tensor is cheap to copy — two of them sharing a node are two names for one value, as in PyTorch. A Storage is always held through shared_ptr, because several tensors may view one block and a view must keep the underlying memory alive. ggml solves the same problem by tracking a view_src pointer on each tensor.

Storage does not know how to allocate

A Storage holds a deleter supplied by whoever created it. That is not a generalisation for its own sake — it is what lets core (layer 1) own the type while backend/vulkan (layer 4) supplies Vulkan-specific freeing. Without the inversion, core would have to know about Vulkan, which the layering check rejects.

The Allocator interface is deliberately minimal: allocate, identify, report a device. It models "ask this thing for memory" rather than a general allocator framework, so alignment policies, memory kinds and async free lists can be added to a concrete allocator without touching the interface.

ⓘ Note

That seam is not hypothetical. The development GPU exposes only 256 MiB of host-visible device-local memory against 5.75 GiB of device-local total, so every upload goes through a separate staging allocator on the same device. Memory has to be requestable independently of who computes on it, or that case cannot be expressed.

CPU allocations are aligned to 64 bytes — one cache line on every CPU this runs on, and the alignment AVX-512 wants. The CPU backend is a correctness oracle and will not be hand-vectorised, but aligning costs nothing and removes a variable if the compiler ever vectorises part of it.

Layout: row-major, strides in bytes

shape()[0] is the outermost axis, matching NumPy, PyTorch and DLPack — so zero-copy interop needs no axis reversal. ggml reverses the order, which its own documentation records as a recurring source of confusion for people arriving from PyTorch. The cost of agreeing with NumPy is that anyone reading ggml kernels alongside vkML kernels must mentally reverse the indices, and that cost is paid once by the maintainers rather than continuously by every user of the Python API.

Strides are in bytes, following ggml and NumPy — ndarray.strides is also in bytes. Bytes rather than elements makes broadcasting (stride 0) and future mixed-dtype views expressible without special-casing.

Rank is capped at 4

>>> vkml.tensor(np.zeros((2, 2, 2, 2, 2), dtype=np.float32))
ShapeError: rank 5 exceeds kMaxDims=4

The reasoning is concrete rather than aesthetic. Every model in scope is rank ≤ 4 — CNNs are [N, C, H, W], transformers [B, H, S, D], RNNs [T, B, F] — and the push-constant budget decides the rest: three tensors × (dims + strides) costs 96 bytes at rank 4 but 192 at rank 8, which would force shape metadata into a uniform buffer and add an indirection to every kernel.

Raising it later is a contained change — the constant, plus the push-constant layout — but it taxes every kernel, so the header records that it should be a deliberate decision rather than a drift.

Which operations alias, and which copy

This is the distinction that decides whether an operation costs memory, and it is worth knowing precisely rather than by intuition.

Operation	Result	How
`reshape`	view	New extents over the same storage.
`permute`, `transpose`	view	Reorders the stride vector; data untouched. Usually non-contiguous after.
`squeeze`, `unsqueeze`	view	Adds or drops an axis of extent 1.
`slice`	view	Adjusts offset and extents, and the stride when `step > 1`.
`broadcast_to`	view	Stride 0 on expanded axes — the same element is re-read.
`contiguous`	copy	Materialises. Returns `*this` unchanged when already contiguous.
`to(dtype)`	copy	Converts, allocating.
`detach`	copy today	Forces realization; making it lazy is tracked architectural work.

Stride-0 broadcasting is why a broadcast costs no memory anywhere in vkML. It is also why conv2d can reshape a bias to (C_out, 1, 1) and add it across batch and space without materialising anything.

The one mutation

assign_ overwrites a tensor's storage in place, and it is the single deliberate escape from an otherwise functional graph. It exists for exactly one reason: optimisers must update parameters that modules already hold references to. Rebinding a new Tensor would leave every Module pointing at the old one, so PyTorch mutates in place and so does this.

⚠ Warning

Any already-computed node that read the tensor keeps its old result, while any node computed afterwards sees the new values. That is harmless in the intended use — the training graph is rebuilt every step — but assign_ must not be used mid-graph. It requires matching shape, dtype and device, and a contiguous destination.

The graph node is not visible

Node is a forward declaration in the public header and nothing more. Callers and the binding layer see an opaque handle, so the internal representation can change without breaking either. That is a recorded guardrail, which is why Tensor's special members are declared in the header and defined in tensor.cpp, where Node is complete.