Getting the workstation's GPU to do the training

Some of the ideas in this repository need a neural network trained. This package answers a purely practical question: can that training run on the graphics processor built into the research machine, and is it worth it?

The answer is yes on both counts, with two software defects to work around — and the work also uncovered two separate bugs on the machine itself that quietly corrupt numbers the rest of the repository was relying on.

What it does

The package is one benchmark and correctness harness plus an environment script, and it runs four independent stages:

1. Probe. Reports what is actually installed — interpreter, framework, GPU runtime, device properties — so that a later result can be attributed to a configuration.
2. Verify. Checks that the GPU computes correct results, scored against a double-precision reference on the CPU rather than against single-precision CPU output. That choice turned out to matter enormously.
3. Matrix throughput. A sweep over matrix sizes and number formats.
4. Training throughput. Forward and backward passes of a network shaped like the ones this repository would actually train: a 7-by-7 board encoded as twelve planes, a stack of residual blocks, and two output heads.

What it found about the machine

The GPU works, and it is right. Against a double-precision reference, a 4096-square single-precision matrix multiply came out with a relative error of 3.451e-06; gradients through the board-shaped network matched to 3.278e-07; a 200-step soak left every parameter finite and drove the loss from 2.948 to 0.003 on a fixed batch, so the network demonstrably learns rather than merely running.

Memory is one pool, not two. This is an integrated GPU with no dedicated video memory. The 512 MiB the system tool reports is a firmware carve-out and is misleading; the real figure is a 94.94 GiB pool shared with system RAM. The largest configuration measured used 5.81% of it, so memory is nowhere near being the constraint.

It is worth using for large batches, and roughly a wash for small ones.

The advantage is batch size, not raw speed: GPU throughput rises about 2.5 times from batch 256 to batch 8,192 while CPU throughput falls about 40% under cache pressure. At the small end the margin is thin enough that the CPU remains a legitimate fallback, and avoids the operational burden entirely.

The CPU and GPU share one power budget. On this kind of chip they are the same package. Measured mean shader clock fell from 2,084 MHz at a host load of 10.7 to 1,059 MHz at a load of 41 — nearly half. Running a heavy CPU job beside GPU training makes both miss their numbers, and it means the speedups above are if anything understated, because the GPU was clock-starved by the same contention that was slowing its CPU comparator.

The two defects it works around

The shipped GPU runtime crashes on the first operation. Every GPU call segfaulted out of the box. Compiling a native vector-add against the system-installed runtime worked perfectly, which separated "the framework's bundled copy is broken" from "the hardware or driver is broken". The fix is to preload the system copy of one specific leaf library — safe precisely because it pulls in nothing else.

Batch normalisation cannot be compiled for this GPU. The shipped kernel library selects a solver written for an older GPU generation and emits assembly this one rejects. It only affects training mode, which makes it easy to miss — the original correctness suite passed because it ran in evaluation mode. The obvious fix is a trap: preloading the system copy of that library does make it work, and then gives the process two copies of the GPU runtime and two device contexts, which survives simple tests and crashes nondeterministically a few training steps in. The workaround used instead is to avoid that layer entirely and use group normalisation, which is also the better choice on the merits for this kind of training.

The two host bugs found along the way

These are the most broadly important part of the package, because they are not about the GPU at all.

Multithreaded single-precision matrix multiply on this CPU returns wrong answers. Once the numerical library uses four or more threads, roughly 0.1% to 1% of output elements come back wrong — by around 10, five orders of magnitude beyond legitimate rounding — and differently on every run for byte-identical inputs. That makes it a data race, not a precision effect. Double precision never reproduced it; one and two threads never reproduced it; and a different maths library performing the same multiplication on the same CPU, in the same process, interleaved with the failing calls, was correct 40 times out of 40 while the failing one was wrong 40 out of 40. The mitigation is to pin that library to one thread, which costs essentially nothing here.

Convolutions on the CPU do not repeat. Two identical forward passes of the same convolution over the same input, in evaluation mode, with every thread count pinned to one, differ in about 6.25% of output elements. At whole-model scale the effect is larger, because normalisation mixes channels: repeated passes of a three-million-parameter network over an identical batch differ in 100% of outputs. This silently invalidated a parity check — a comparison that reported a maximum difference of 1.1e-5 reported failure on every head twenty minutes later, and neither run was wrong about the code being tested. The reference had moved.

What this taught us, and what is still open

The practical verdict is narrow and useful: only the neural training step belongs on the GPU. Game simulation stays on the CPU — nothing here suggests otherwise — and any training loop that intends to use the GPU should be written to push large batches in bf16, because that is where the hardware's advantage lives.

The methodological verdict is broader, and it is the reason this page exists at all rather than living only in a build script. Two independent, silent numerical defects were found on the research machine within one session, both of which had already produced a wrong conclusion before they were understood. Neither raises an error, a warning, or a not-a-number. The only thing that caught them was checking a reference against something more trustworthy than itself.

What is still open: every throughput figure needs re-measuring on an idle host under a resource lease before it can be quoted as a performance claim, and the matrix-multiply race deserves the two cheap upstream checks the record names — whether a differently-built copy of the same library at the same version reproduces it, and whether a newer version has already fixed it.