gptoss-spark
gpt-oss-120b · one DGX Spark · GB10, sm_121

Same GPU, same model.
Six patches faster.

Not a smaller model and not a different quantization — the same gpt-oss-120b on the same Spark, going from 52.6 to 68.9 tok/s. Six patches, a set of CUTLASS MXFP4 kernels built for this exact chip, and one counter-intuitive serving decision. Every number measured, every failed approach written down.

$ docker run --gpus all --network host --ipc host \
    --shm-size 32g -v /srv/models/gpt-oss-120b:/model:ro \
    -v gptoss-jit:/root/.cache/flashinfer \
    -e PROFILE=plain ghcr.io/luka-loehr/gptoss-spark:0.2.0
One paste, doneSource ↗
Decode
68.9
tok/s, was 52.6
At 30 users
299.5
tok/s aggregate, was 244.1
TTFT at 30
1.2 s
was 16.4 s
Patches
6
five on vLLM, one on kernels
GPU busy
24.9 s of 25 s
998 decode passes, nsys

The GPU runs kernels ~100% of the time — there is no host-side stall left to reclaim. An earlier claim of a “12 ms/pass engine overhead” came from a Python profiler that cannot tell waiting for the GPU from making the GPU wait, and was refuted three times over.

One DGX Spark, GB10, 121 GB unified memory. Identical benchmark for every row: 4 requests × 512 tokens, single stream, temperature 0, decode median. The two profile rows were measured from the published image, not a lab configuration. Fifteen approaches that did not work are in NEGATIVE-RESULTS.md.

One paste

Let your agent set it up

You were going to paste it into Claude anyway. I saved you the typing. It checks you actually have a Spark and the checkpoint, picks the right profile for your load, and waits out the JIT compile instead of declaring victory early.

Paste into your agent
Set me up with gptoss-spark — gpt-oss-120b served on one NVIDIA DGX Spark.
Run this first, then follow it exactly, top to bottom:

    curl -fsSL https://gptoss.lukaloehr.com/setup.txt

Ask me its first question before doing anything else. Never print my tokens.
Get started

Two steps

Ah, so you haven’t fallen for the vibe yet. Respect. I still read commands before I run them too.

01

Bring your own checkpoint

Needs a DGX Spark (GB10, sm_121), Docker with the NVIDIA runtime, the gpt-oss-120b MXFP4 checkpoint and ~90 GB free. The checkpoint is not part of the image. Mount the JIT cache volume shown below: the first start compiles the SM121 kernels and takes about ten minutes, every later start is warm.

$ docker run --gpus all --network host --ipc host \
    --shm-size 32g \
    -v /srv/models/gpt-oss-120b:/model:ro \
    -v /srv/tiktoken:/tiktoken:ro \
    -v gptoss-jit:/root/.cache/flashinfer \
    -e PROFILE=plain ghcr.io/luka-loehr/gptoss-spark:0.2.0
02

Pick the profile for your load

PROFILE=plain serves many users — 32 slots, 299.5 tok/s aggregate at 30. PROFILE=spec is the single-user record at 68.9 tok/s and additionally needs the Eagle3 head mounted at /eagle. They are different jobs; do not assume spec is strictly better.

$ bench/bench.py --base-url http://127.0.0.1:8100/v1 \
    --model gptoss --prompts bench/prompts.jsonl \
    --requests 4 --concurrency 1 --max-tokens 512
Measured

Against every other path

SGLang :spark (previous production)
52.6 tok/s
0.70 s
96.3 GiB
llama.cpp b6fdd0ac, MXFP4 GGUF
50.4 tok/s
0.46 s
—
stock vLLM nightly, any backend
33–35 tok/s
0.25 s
—
this repo, PROFILE=plain
64.3 tok/s
0.21 s
83 GiB
this repo, PROFILE=spec (K=1)
68.9 tok/s
0.24 s
83 GiB

Under load — previous vs this

10 users, aggregate143.5 tok/s → 173.0 tok/s
10 users, until all done35.7 s → 29.6 s
30 users, aggregate244.1 tok/s → 299.5 tok/s
30 users, TTFT median16.4 s → 1.2 s
30 users, until all done62.9 s → 51.3 s

The 30-user TTFT gap is structural, not noise. The previous stack ran --max-running-requests 15, so half the users queued while the other half decoded. This one admits all 30 at once and still finishes 11.6 s earlier.

What produced it, ranked

MXFP4 for the dense layers+28 tok/s — the single largest win
SM121 CUTLASS MXFP4 MoE kernels+~27 tok/s over Marlin MoE
Backend selection unblockedthree lines; enables the above
Speculation at K=1+2.8 tok/s — and K=2 is slower
MoE support kernel, loop inverted+2.6 plain, +3.1 spec
Draft head cut to 32,768 rows+0.5 tok/s

All measured in isolation. gpt-oss ships MXFP4 expert weights, but qkv_proj, o_proj and the 201k-row lm_head stay bf16 in every stock path — quantizing those is where most of it came from.