vLLM Docker Optimized for DGX Spark (single or multi-node)

This repository contains the Docker configuration and startup scripts to run vLLM on DGX Spark, from a single node to multi-node clusters using Ray or vLLM's native PyTorch distributed mode. It supports InfiniBand/RDMA (NCCL), custom environment configuration, and high-performance model loading through fastsafetensors and InstantTensor. Cluster setup supports direct connections between dual Sparks, QSFP/RoCE switch configurations, and 3-node mesh configurations.

While it was primarily developed to support multi-node inference, it works just as well on single-node setups.

Table of Contents

• DISCLAIMER
• QUICK START
• CHANGELOG
• 1. Building the Docker Image
• 2. Launching the Cluster (Recommended)
• 3. Running the Container (Manual)
• 4. Configuration Details
• 5. Mods and Patches
• 6. Launch Scripts
• 7. Using cluster mode for inference
• 8. Model Loading
• 9. Benchmarking
• 10. Downloading Models

DISCLAIMER

This repository is not affiliated with NVIDIA or their subsidiaries. This is a community effort aimed to help DGX Spark users to set up and run the most recent versions of vLLM on Spark cluster or single nodes.

By default, build-and-copy.sh pulls the tested nightly runner image from DockerHub: eugr/spark-vllm:latest. Nightly images are built and tested on multiple models in both cluster and solo configuration before latest is advanced. We will expand the selection of models we test in the pipeline, but since vLLM is a rapidly developing platform, some things may break.

If you want to build from precompiled vLLM and Flashinfer wheels instead, specify --use-wheels. To build the latest vLLM from the main branch, use --rebuild-vllm; to target a specific vLLM release or commit, set --vllm-ref.

Similarly, --rebuild-flashinfer, --flashinfer-ref, and --apply-flashinfer-pr control the FlashInfer build and force the local build path.

QUICK START

Build

Check out locally. If using DGX Spark cluster, do it on the head node.

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker

Prepare the container image.

If you have only one DGX Spark:

./build-and-copy.sh

On DGX Spark cluster:

Make sure you connect your Sparks together and enable passwordless SSH as described in our Networking Guide. You can also check out NVIDIA's Connect Two Sparks Playbook, but using our guide is the best way to get started. The guide includes instructions for 3-node Spark mesh clusters.

Then run the following command to pull, tag, and distribute the image across the cluster.

./build-and-copy.sh -c

The default image preparation speed depends mostly on your Internet connection and whether eugr/spark-vllm:latest is already present locally.

For slower internet connections it can be faster to build from the precompiled wheels by using --use-wheels parameter. An initial build speed depends on your Internet connection speed and whether the base image is already present on your machine. After base image pull, the build should take only 2-3 minutes.

If --use-wheels, --rebuild-vllm, --rebuild-flashinfer, or another build customization is used, the script keeps the local wheel-based build path; full source rebuilds can take 20-40 minutes, but subsequent builds are faster.

Run

On a single node:

launch-cluster.sh supports solo mode, which is now a recommended way to run the container on a single Spark:

./launch-cluster.sh --solo exec \
  vllm serve \
    QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ \
    --port 8000 --host 0.0.0.0 \
    --gpu-memory-utilization 0.8 \
    --load-format fastsafetensors

To publish the server port instead of using host networking, pass the Docker port mapping before exec:

./launch-cluster.sh --solo -p 8000:8000 exec \
  vllm serve \
    QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ \
    --port 8000 --host 0.0.0.0

On a cluster

It's recommended to download the model on one node and distribute across the cluster using ConnectX interconnect prior to launching. This is to avoid re-downloading the model from the Internet on every node in the cluster.

This repository provides a convenience script, hf-download.sh. The following command will download the model and distribute it across the cluster using autodiscovery.

./hf-download.sh QuantTrio/MiniMax-M2-AWQ -c --copy-parallel

To launch the model:

./launch-cluster.sh exec vllm serve \
  QuantTrio/MiniMax-M2-AWQ \
  --port 8000 --host 0.0.0.0 \
  --gpu-memory-utilization 0.8 \
  -tp 2 \
  --distributed-executor-backend ray \
  --max-model-len 128000 \
  --load-format fastsafetensors \
  --enable-auto-tool-choice --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2

The launcher will use the number of nodes required by the parallelism flags. In a 2-node cluster, this command uses both nodes; in a larger configured cluster, extra nodes are not utilized.

NOTE: do not use --load-format fastsafetensors if you are loading models that would take >0.85 of available RAM (without KV cache) as it may result in out of memory situation.

Also: You can use other vLLM containers with the launch script as long as they have bash available. The launcher clears image entrypoints by default, to prevent containers such as vllm-openai to start vLLM before all necessary initialization is complete. However, it's recommended to build the container using this repository for best compatibility and most up-to-date features.

IMPORTANT

You may want to prune your build cache every once in a while, especially if you've been using these container builds since the beginning.

You can check the build cache size by running:

docker system df

To prune the cache for the first time or if you notice unusually big cache size, use:

docker builder prune

Don't do it every time you rebuild, because it will slow down compilation times.

For periodic maintenance, I recommend using a filter: docker builder prune --filter until=72h

CHANGELOG

2026-07-02

Prebuilt runner image by default

build-and-copy.sh now pulls prebuilt eugr/spark-vllm:latest by default and tags it locally as vllm-node or the tag requested with -t. The latest tag points at the latest tested nightly image. The prebuilt image is updated at the same time as prebuilt wheels by the CI pipeline, so they all stay in sync.

Use --use-wheels to keep the previous wheel-based runner build path. Build customization flags such as --exp-mxfp4, non-default --gpu-arch, --vllm-ref, --flashinfer-ref, rebuild/download flags, and PR application flags also keep the local build path. --tf5 remains a tag-compatibility alias and pulls the prebuilt image as vllm-node-tf5.

Copy now checks the image ID locally and on each remote host before saving the image. Hosts that already have the same image ID are skipped, and docker save is skipped entirely when every target is already current. --no-build still skips image preparation and only copies an already-local tag when needed.

2026-07-01

No-Ray is now the default multi-node backend

launch-cluster.sh and run-recipe.sh now default to no-Ray multi-node launches. Use --ray to opt into Ray; Ray mode ensures vLLM commands include --distributed-executor-backend ray when they omit it. --no-ray remains accepted for compatibility in multi-node launches.

Build and dependency updates

We now use NCCL main branch and include new experimental vLLM Rust frontend in the builds. DeepGEMM now tracks the nv_dev branch.

Transformers 5 flag deprecation

--tf5, --pre-tf, and --pre-transformers are now deprecated compatibility aliases. They no longer override dependency resolution; they only preserve the legacy default image tag. Recipes that previously used vllm-node-tf5 now use the standard vllm-node image.

Recipe updates

Added the gemma4-26b-a4b-nvfp4 recipe, reverted the Qwen3.6-35B-A3B-NVFP4 recipes to --kv-cache-dtype fp8, and cleaned up stale TF5 build args from affected recipes.

2026-06-22

Deepseek V4 Flash support

Support and a recipe for Deepseek V4 Flash has been added based on a newly merged vLLM PR. Please note that this PR requires the DeepGEMM nv_dev branch, which is not present in default vLLM builds, but included in this community build.

To run DSV4F you will need a Spark cluster (2 or more nodes).

To run:

git pull
./build-and-copy.sh -c
./hf-download.sh deepseek-ai/DeepSeek-V4-Flash -c
./run-recipe.sh deepseek-v4-flash --no-ray

DeepGEMM support added

vLLM now uses NVIDIA branch for DeepGEMM that includes support for sm12x GPU family, including DGX Spark. Please note that it is currently not compatible with NVRTC compiler, so DG_JIT_USE_NVRTC has been turned off for new builds.

MiniMax AWQ Weight-Shape Loader Workaround

Added a Dockerfile-level workaround for a vLLM nightly regression where compressed-tensors MoE weight_shape metadata is treated as a scalar during load, causing affected MoE quantized models to fail with shape '[]' is invalid for input of size 2.

2026-06-18

KV Cache Preallocation Cleanup Mod & Updated Qwen3.5-397B recipe for dual Sparks

Added KV cache preallocation cleanup for source-built vLLM wheels, which clears cached CUDA allocator memory before vLLM sizes and allocates KV cache blocks. The dual-node Qwen3.5-397B INT4 AutoRound recipe also applies mods/kv-cache-prealloc-cleanup after mods/gpu-mem-util-gb for its model-specific memory policy tweaks.

The same recipe now keeps the 108 GiB startup reservation, sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, and uses a manual 2.25 GiB KV-cache allocation to bypass vLLM's conservative profiler-derived KV budget. The source build handles profiling-only graph-pool cleanup before KV-cache sizing, while the recipe mod makes the env var skip CUDA graph memory profiling entirely and allows the fixed-GiB reservation argument to coexist with --kv-cache-memory-bytes for this model to load. This may result in a bit of swap to be used to offload unused resources, so make sure swap is enabled.

Added recipes for nvidia/Qwen3.6-35B-A3B-NVFP4

Added recipes for NVIDIA's mixed-precision quant for Qwen3.6-35B model. It has a higher precision and an improved performance compared to the earlier NVFP4 quants.

Use:

• qwen3.6-35b-a3b-nvfp4 to run with MTP on
• qwen3.6-35b-a3b-nvfp4-no-mtp to run without MTP.

2026-06-10

DiffusionGemma Recipes and Mod

Added day0 support for Google DeepMind's DiffusionGemma model via mods/diffusiongemma. Check out NVIDIA blog for details!

Added four solo-only DiffusionGemma recipes:

• diffusion-gemma-bf16-thinking for google/diffusiongemma-26B-A4B-it with thinking enabled.
• diffusion-gemma-bf16 for google/diffusiongemma-26B-A4B-it with thinking disabled.
• diffusion-gemma-nvfp4-thinking for nvidia/diffusiongemma-26B-A4B-it-NVFP4 with thinking enabled.
• diffusion-gemma-nvfp4 for nvidia/diffusiongemma-26B-A4B-it-NVFP4 with thinking disabled.

The non-thinking variants still keep --reasoning-parser gemma4, since these models can emit Gemma4 channel markers even when thinking is disabled.

Example:

./hf-download.sh google/diffusiongemma-26B-A4B-it
./run-recipe.sh diffusion-gemma-bf16-thinking --solo

run-recipe.sh Launch Flag Passthrough

run-recipe.sh now passes additional launch-cluster.sh flags through when running recipes: --apply-mod, -p / --publish, and --keep-entrypoint.

Port publishing is still solo-only, matching launch-cluster.sh behavior.

2026-06-09

Recipe Memory Defaults

Raised the default gpu_memory_utilization from 0.7 to 0.8 across the main single-node and two-node recipes to match the current vLLM memory allocation behavior.

2026-06-07

Docker Base Image Compatibility

The default CUDA base image was changed to nvidia/cuda:13.0.2-devel-ubuntu24.04 for broader host compatibility.

The Dockerfile also now passes --allow-change-held-packages when installing the custom NCCL Debian packages, avoiding apt failures when replacing held CUDA/NCCL packages during image builds.

2026-06-06

MiniMax Multi-Node Regression Workaround

Added a targeted Dockerfile patch that disables the MiniMax QK RMSNorm CUDA IPC fused path introduced by vLLM PR #43410. The fused path can fail when tensor parallelism spans DGX Spark nodes; the workaround preserves MiniMax multi-node TP while avoiding a full upstream revert.

2026-06-03

Solo Port Publishing

launch-cluster.sh now supports Docker-style -p / --publish port mappings in solo mode. When port publishing is used, the launcher switches from host networking to Docker bridge networking for that solo container.

Example:

./launch-cluster.sh --solo -p 8000:8000 exec vllm serve ...

2026-05-29

Wheel Freshness Detection

Improved build-and-copy.sh wheel freshness checks so newer locally built wheels are not overwritten just because their filenames differ from the latest release assets. The script now compares local wheel mtimes against remote release asset timestamps before deciding to download. Also switched from using GitHub API to regular HTTP checks to avoid throttling.

gpu-mem-util-gb Patch Refresh

Refreshed mods/gpu-mem-util-gb so it applies against newer vLLM CacheConfig code after upstream line/context drift.

2026-05-28

StepFun Step 3.7 Flash Support

Added support for StepFun Step 3.7 Flash multimodal model.

Requires at least 2 Sparks in a cluster. Both FP8 and NVFP4 checkpoints are supported. FP8 requires more memory, so using NVFP4 is recommended.

Update the repo and build a fresh container first:

git pull
./build-and-copy.sh --cleanup -c

To run NVFP4 version:

Download the model:

./hf-download.sh stepfun-ai/Step-3.7-Flash-NVFP4 -c

Run:

./run-recipe.sh step-3.7-flash-nvfp4 --no-ray

To run FP8 version:

Download the model:

./hf-download.sh stepfun-ai/Step-3.7-Flash-FP8 -c

Run:

./run-recipe.sh step-3.7-flash-fp8 --no-ray

Please note that --no-ray is required for F