jamesob's guide to running SOTA LLMs locally

Note: nothing in this README aside from the tables was written by AI.

Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence?

How about $40k?

If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally.

In this repo you'll find

• the hardware I use to run SOTA locally,
  • why I bought what and little-known secrets for configuring it,
• how I run speech-to-text (STT) locally,
• ready-to-run configuration for running models I think are good within Docker containers.

Contents

My setup

I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards, the parts for which I got off eBay. This allowed me to keep base system cost reasonable while still getting a lot of VRAM.

Another somewhat unusual thing I did was to use PCIe4 switches (from c-payne.com). This allows the GPUs to communicate to one another "directly" at wire speeds during the allreduce step in tensor parallelism, rather than having to send all data through the PCI root complex. The upshot of this is reduced latency between the cards with less of a need for expensive PCIe5 hardware.

Consequently, I'm spending money on VRAM (where it counts) rather than on a PCIe5/DDR5 base system, which is terrifically expensive as of July 2026.

By using an older host-gen4 setup rather than gen5, we're saving about $10,000 in host costs while still getting very good performance in VRAM, which is what we want. Generally, spilling over into system RAM makes LLM performance unusably slower for agentic workloads.

My particular BOM is detailed below.

How much are you willing to spend?

~$2k

A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model.

You can also run SOTA speech-to-text (STT) with cohere-transcribe (I was previously using whisper-large-v3), which I find very useful. That's the model - you could then access it with my cross-platform stt harness.

I've found local STT surprisingly useful - and I feel comfortable using it, unlike a hosted equivalent. You can find a ready-to-run config in ./runners/stt that only assumes the presence of ~11GB of VRAM on an Nvidia GPU.

~$20k

(TODO)

• 2x RTX 6000 Pro
• 4x DGX Sparks networked together
• Apple hardware

I don't have any experience with this regime so I'm not of much help here.

~$40k

At this level, you get the next step up in model intelligence. Something pretty close to Claude Opus.

You'd buy 4x RTX 6000 Pros for a total of 384GB of VRAM.

Current best models for 4x RTX6kPRO

Other approaches

Note: these are my recommendations, but there are other completely valid ways to spend your money. For example, there's probably also some regime where rather than getting 4 rtx6kpros, you allocate most of your money to building out a linked 4x DGX Spark cluster for a total of 512GB VRAM and use that as the slow, big brain to drive Qwen3.7-27b to do the rote tasks quickly.

Hardware

Here's the hardware I wound up purchasing for the 4x RTX 6000 pro machine.

Base system

A modest, last-gen EPYC system purchased in parts almost entirely from eBay.

GPUs

c-payne PCIe Gen4 Switch Sub-BOM (c-payne.com)

GPU mount

I had to custom fabricate a wood enclosure for the PCI switch and GPUs, which took about a day.

I found the PCI switch's builtin fan very loud and seemingly useless, so I simply unplugged that from the board.

Hoarding model weights

I save all model weights locally on a ZFS filesystem that's replicated across the two 8TB drives, which is mounted at ~/storage.

For any model I want to run, I first download the model using

hf download <model-name> --local-dir ~/storage/<model-name>

Running models

Once the model weights are cached locally, I have a specific directory for each model that contains a docker-compose.yml file that cordones off the running of each model to its own Docker container.

You can find these configurations in ./runners/.

Each container mounts in ~/storage/models in read-only mode to obtain the weights that I've cached locally.

I then use opencode hosted on a VM on another machine to access the models once they're serving on http://clank.j.co:5000.

I use a network-internal DNS server to point clank.j.co to the LLM machine, but you could simply do http://<llm-machine-ip>:5000 too.

The harness itself

I created a VM and clanked up an application that basically just creates a tmux session for each directory within the VM's ~/src tree, which then runs an opencode instance that backs up to the inference machine's HTTP API (http://clank.j.co:5000).

One key to making the opensource models good is tooling them properly; a summary of my skills/ is:

• camofox, kagi.com API key, and searXNG for web browsing and search,
• Telegram bot for communication and alerting,
• a local private Gitea instance for collaborating on source code.

The clanker will either work with me interactively in a session, or can be farmed off to work on Gitea issues and file PRs there.

All this happens in a sandboxed VM where the only communication back to the host system happens via a shared filesystem mount, so the thing can go ham and install whatever it wants.

Getting the PCI switches to work properly

There was a lot of fiddling with the BIOS in order to make sure the motherboard wasn't downregulating the PCI switch speeds.

BIOS Configuration (ROMED8-2T)

Reducing gain on the redriver

Per c-payne's advice, I did reduce the gain to "lvl 3" using his tool, which was probably the most finicky part of the process.

The gain level is going to be a function of how long your SAS connector cables are.

Picking the right SAS cables

I screwed up and ordered too few of the cables from c-payne directly, so I bought what I thought was the same SAS cable off of Amazon. There was actually a slight difference that was causing issues, and I had to reorder cables - so double-check that you're getting the right stuff!

Kernel / GRUB Parameters

# /etc/default/grub
GRUB_CMDLINE_LINUX="iommu=off amd_iommu=off nomodeset"
sudo update-grub

# nvidia_uvm P2P fix
echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf
sudo update-initramfs -u

Without iommu=off, NCCL hangs on multi-GPU P2P.

ACS Disable (critical for switch P2P)

With ACS enabled (default), P2P traffic gets bounced through the CPU root port instead of staying inside the switch fabric, negating the switch entirely. pcie_acs_override requires a patched kernel, so we disable via setpci at runtime.

# /usr/local/bin/disable-acs.sh
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: must be run as root"
  exit 1
fi

for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
  setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
  if [ $? -ne 0 ]; then
    continue
  fi
  echo "Disabling ACS on $(lspci -s ${BDF})"
  setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done

Run on every boot via systemd oneshot:

# /etc/systemd/system/disable-acs.service
[Unit]
Description=Disable PCIe ACS for GPU P2P
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disable-acs.sh

[Install]
WantedBy=multi-user.target

Verify: lspci -vvv | grep ACSCtl should show all minus signs, and nvidia-smi topo -m should show PIX between all four GPUs (not PHB/NODE).

Use ./tools/measure-gpu-speed.sh to measure this easily.

GPU Power Limiting

In order to avoid installing a 220V circuit, I (probably unwisely) run this rig on a single 110V circuit, but I power regulate the cards.

Persistence mode + power cap applied at boot via systemd (install-gpu-power-limit.sh):

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 350    # 350W per GPU (default 600W)

350W/GPU = 1,400W GPU load, sized for the PSU budget. During the interim single-1700W-PSU phase (before the 240V circuit), cards ran at ~260W (4×260 = 1,040W GPUs + ~280W system ≈ 1,320W total).

Verify: nvidia-smi --query-gpu=index,power.limit,power.draw --format=csv

Result

Upstream: Gen4 x16 (~30 GB/s to CPU). P2P through switch: 27.5 GB/s unidirectional / 50.4 GB/s bidirectional, 0.37–0.45 µs latency, i.e. Gen4 line rate. Note: lspci may still show downstream GPU links as "2.5GT/s (downgraded)" at idle if ASPM is active anywhere; this is cosmetic. Links retrain to Gen4 under load.

Resources

• A frequently updated repo on getting the most out of 4, 6, or 8 RTX 6000 Pro cards: https://github.com/local-inference-lab/rtx6kpro
• Indie PCI switches that I use: https://c-payne.com
• RTX6kPRO discord server; lotta guys benching and testing new models: https://discord.gg/QMNvFkuDN