# gptoss-spark

> Serving gpt-oss-120b on a single NVIDIA DGX Spark (GB10, sm_121) at 68.9
> tok/s single-stream and 299.5 tok/s aggregate at 30 concurrent users — up
> from 52.6 and 244.1 on the previously deployed SGLang stack. Not a model
> change and not a different quantization: six patches (five on upstream vLLM,
> one on the MoE kernels), a set of CUTLASS MXFP4 kernels built for this exact
> chip, and one non-obvious serving decision (speculation depth K=1).

Made by Luka Löhr — https://lukaloehr.com

    docker run --gpus all --network host --ipc host --shm-size 32g \
      -v /srv/models/gpt-oss-120b:/model:ro \
      -v /srv/tiktoken:/tiktoken:ro \
      -v gptoss-jit:/root/.cache/flashinfer \
      -e PROFILE=plain ghcr.io/luka-loehr/gptoss-spark:0.2.0

Requires a DGX Spark, Docker with the NVIDIA runtime, the gpt-oss-120b MXFP4
checkpoint and ~90 GB free. The checkpoint is NOT part of the image.

## Setup

- [Agent setup runbook](https://gptoss.lukaloehr.com/setup.txt): written for a coding agent. Checks the hardware and the checkpoint first, picks the profile for the actual load, and waits out the JIT compile. Start here.
- [Profiles](https://gptoss.lukaloehr.com/profiles.txt): plain vs spec, what each is for, and why spec is not strictly better.

## Reference

- [Results](https://gptoss.lukaloehr.com/results.txt): every serving path measured on the same benchmark, plus concurrency.
- [What produced the speedup](https://gptoss.lukaloehr.com/speedup.txt): the six contributions ranked, each measured in isolation.
- [Everything in one file](https://gptoss.lukaloehr.com/llms-full.txt): all of the above concatenated.

## Source

- [GitHub repository](https://github.com/luka-loehr/gptoss-spark): patches, benchmark harness, measurement campaign. Apache-2.0.
- [NEGATIVE-RESULTS.md](https://github.com/luka-loehr/gptoss-spark/blob/main/docs/NEGATIVE-RESULTS.md): 15 approaches that did not work, and why.