Building hal0, part 3: where the runners come from

Part 1 covered the slot model: every inference workload runs as its own podman container under its own systemd unit. That leaves a question I skipped at the time, and it turns out to be most of the actual work.

Which container?

A slot is a name, a port, a model and a state. It is not a runtime. Something has to supply the image that holds llama.cpp compiled against the right accelerator, or FastFlowLM built for the NPU, or ComfyUI with a ROCm stack under it. On a box with an AMD iGPU that nobody’s tooling targets by default, “just pull the official image” is not an available move. There isn’t one.

So the images are built. Here is where they come from, why they live in three repositories instead of one, and how you point a slot at a different one.

1. Reference, don’t absorb

The obvious approach is a single repo holding every Dockerfile. I started there and moved off it for two reasons.

The first is CI weight. Building a ROCm llama.cpp image is slow and enormous. Keeping that in the application repository meant every unrelated pull request sat behind GPU image builds, and the application’s own test suite — the fast, useful one — got buried under a workflow that had nothing to do with it. Splitting the images out isolates that.

The second reason is more interesting. Two of the runner sources were already hal0-owned forks with working CI of their own. Absorbing them would have meant copying live source into a second location and then competing with their own upstream-sync workflows forever. Vendoring code that is already maintained somewhere is how you end up with two versions and no idea which one shipped.

The rule I settled on is reference, don’t absorb. hal0-runner-images is an aggregator: it declares which runner images exist, and it holds Dockerfiles only for the ones nobody else maintains.

2. The three repositories

Hal0ai/hal0-runner-images is the registry. images.json is the source of truth for what exists, what it is pinned to, and how it is built. Its CI resolves published container digests and opens a pull request against the application to bump them. It builds only what it owns.

Hal0ai/amd-strix-halo-toolboxes is a fork of kyuz0’s Strix Halo toolboxes, and it produces the vulkan and rocm llama.cpp images. It carries its own upstream-sync workflow so it keeps tracking the project it forked from rather than drifting into a dead end. It also produces the builder base that the FPX runner compiles inside.

Hal0ai/Hal0_ROCmFPX is a fork of llama.cpp carrying FPX quantisation support, and it is the primary LLM runner actually serving on the boxes. It builds from a ROCm 7.2.1 base targeting gfx1151 — the Strix Halo iGPU — through the toolbox builder image above.

That last chain is worth reading twice: the FPX runner is built inside an image produced by the toolboxes repo, which is itself a fork tracking an upstream project. Three layers, each one owned, each one pinned. Nothing in the serving path comes from a registry I don’t control.

3. Pins are digests, not tags

A tag is a mutable pointer. :v1 today and :v1 next month can be different bytes, which makes “it worked yesterday” unfalsifiable. Every image in the manifest carries both its human-readable tag and the content digest it resolved to at pin time.

The bump is deliberately a pull request rather than an automatic write. A new runner image changes what executes on every box that updates, and that deserves the same review as a code change — because it is one.

4. Where to find them

Everything published lives under ghcr.io/hal0ai/. The registry itself is images.json at the root of hal0-runner-images, and it is public.

The dashboard’s Runner Images page reads that same file directly over HTTPS and then probes the container registry anonymously for each image’s current tag, digest and size — so the page shows you what is actually published, not what a local cache believes. If the file ever fails to parse, the page degrades to registry-only rows rather than going blank. That degradation path exists because I would rather see a partial answer than a spinner.

From there you can pull an image to the box before you need it, which matters more than it sounds: pulling a multi-gigabyte ROCm image at the moment a slot is trying to start is how a load “times out” for reasons that have nothing to do with the model.

5. Switching the runner a slot uses

Two mechanisms, deliberately different in character.

The normal one is the slot’s binary — which runner family it wants. The image is then resolved from the runner registry in code, or from a hardware-gated default if the slot hasn’t expressed a preference. This is the path you should be on. It means the launched image and the capability gates that decide whether features like multi-token prediction are available are derived from the same source, so they cannot drift apart.

The escape hatch is image_pin: a single typed field on the slot that, when set, is honored verbatim and never re-resolved.

image_pin exists for three situations and I would not use it for anything else: running a debug build, A/B testing two runner builds against the same model, and rolling back to a last-known-good image while something upstream is broken.

Note what is deliberately absent: profiles cannot carry an image. Profiles are device-agnostic tuning templates — context size, sampler settings, flags — and the moment one of them can also pin a container, “which image is this slot running” stops having a single answer. The image is a physical fact about the slot, and it lives on the slot.

6. The part I would rather not write down

Two of the owned Dockerfiles — Kokoro and Moonshine — were reconstructed from the published images’ own layer history, because the original build files were never committed anywhere. They were rebuilt and verified to produce working containers the same day, but “reconstructed from the artifact” is a materially weaker claim than “this is the source it was built from”, and the repository says so in plain text next to both entries.

The ComfyUI image is a similar shape: it is a drafted hal0-owned replacement for a third-party image, and its component versions are still floating branch references rather than commit hashes. That is fine for a draft and not fine for a release, which is why it is annotated rather than quietly shipped.

I am writing both of those down because a supply chain you only describe when it is clean is not a supply chain you can trust. The value of pinning every digest and owning every fork evaporates the moment there is an unlabelled gap in it. Better to name the gap and close it on purpose.

What this actually buys

The honest summary: nothing in the inference path is pulled from a registry or a project I don’t control, every image resolves to a specific digest someone reviewed, and switching a slot to a different runner is one field in one file rather than a rebuild.

None of that is glamorous, and none of it makes a model answer faster. It is the difference between “it runs on my box” and “it runs on yours, and I can tell you exactly what it is running.”