Run Meta's Muse Glimmer on One 24GB GPU: Step-by-Step

Meta's Apache-2.0 Muse Glimmer 30B runs a local agent on a single 24GB GPU. Quantization picks, Ollama and llama.cpp paths, the 3.1x DFlash speedup, and honest benchmark expectations.

🧠 What You Are Deploying

On August 10, 2026, Meta Superintelligence Labs open-sourced Muse Glimmer 30B under Apache 2.0 — a model built for one purpose: a local, always-on agent. It is a dense causal Transformer (about 29.6B parameters) plus a roughly 1.8B ViT-G/14 vision tower, distilled from Meta's larger closed model Muse Spark. It supports 131,072-token context, 100+ languages, native tool calling, and image input.

The point of the exercise, as Zuckerberg framed it in his essay "The Future Is For Everyone": powerful AI that runs on your own hardware, as a counterweight to institutional concentration. Practically: a 4-bit quantized build fits on a single 24GB consumer GPU (RTX 4090/5090, Mac M4/M5 Max).

🧰 Step 1: Pick the Right Build

Meta ships two K-Quant builds, and choosing correctly saves you a weekend of pain:

• K-Quant-17GB (~17GB, needs 24GB VRAM): average accuracy loss about 1.0% across 15 benchmarks — the minimum viable agent build.
• K-Quant-Dynamic (~20GB, needs 32GB VRAM): only ~0.2% average loss — pick this for long-document RAG and precision-sensitive work.
• BF16 reference (~60GB, one H100 80GB): for evaluation and fine-tuning only, not for serving.

Rule of thumb: on a 24GB card, take K-Quant-17GB without hesitating. On 16GB, do not force the official builds — agent reliability degrades in ways that are much harder to notice than worse prose. Community 12–14GB quants are the wait-and-see option.

🚀 Step 2, Path A: Ollama (Fastest Start)

Ollama 0.32.7 shipped with the model definition built in, and v0.32.8 (August 11) extended support across NVIDIA, AMD, and more platforms:

• ollama run muse-glimmer:30b-q4_k_m — pulls and starts the 17GB quant on a 24GB card.
• ollama show muse-glimmer:30b-q4_k_m — verify the model card loaded correctly.

Note that Ollama does not expose DFlash (below) directly — it applies its own speculative-decoding approximation. For full control, go to Path B.

⚙️ Step 2, Path B: llama.cpp (Maximum Control)

For teams wiring the model into an agent framework, llama.cpp gives you sampling control and the DFlash switch:

• Text-only agent inference: llama-cli --model Muse-Glimmer-30B-UD-Q4_K_XL.gguf --temp 1.0 --top-p 0.95 --top-k 64 --ctx-size 32768
• Image input (document screenshots, UI recognition): add the vision projector with --mmproj mmproj-BF16.gguf.

⚠️ The sampling trap: Meta's official defaults are temp 1.0 / top-p 0.95 / top-k 64 — much hotter than the 0.2 you habitually use for code. The model was trained at 1.0; lowering it actually makes agent behavior less reliable, not more.

⚡ Step 3: Turn On DFlash for a 3.1x Speedup

DFlash is the headline trick: a block-diffusion speculative decoding drafter that proposes up to 16 tokens per forward pass while the main model verifies them in parallel. Serve the model as an OpenAI-compatible endpoint and enable it:

• llama serve -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash --spec-draft-n-max 15

On an RTX 5090, this lifts decode throughput from 74.9 to 233.4 tokens/s — 3.1x. It is optional, but for interactive agent sessions the difference is night and day. (For reference: an RTX 4090 at Q4 runs roughly 20–30 tokens/s per community reports.)

🤖 Step 4: Wire a Minimal Agent Loop

The model speaks the OpenAI-compatible protocol natively, including tool_calls. Point an OpenAI SDK client at http://localhost:8080/v1, define a function like "search_docs", and send a chat completion with tools and tool_choice="auto" — the model returns structured calls your loop executes. That is the entire skeleton of a private agent.

📊 Step 5: Know What You Got

Set expectations with the agent benchmarks: Muse Glimmer scores 75.5 on MCP Atlas (vs 62.5 for Qwen3.6 27B) and 74.6 on DeepSearch QA (vs 71.1), leading its tier on tool use (τ3-Banking: 23.5). But it is not omniscient: on GDPval-AA v2 it rates 953 Elo, below the 1,000 human baseline; its hallucination rate on AA-Omniscience is 82% (vs 49%); Terminal-Bench 2.1 lands at 52% (vs 61%). Translation: strong at local tool calling and workflow execution, but pair it with retrieval and human review for knowledge-heavy tasks. When you outgrow the stock model, full SFT via TorchTitan or LoRA via NVIDIA NeMo works without format conversion.

That is the whole path: one GPU, one command to start, one switch for speed, and an agent that never leaves your machine.