1-Bit Kimi K3 GGUF: How Unsloth Squeezed a 2.8T Model Down to 594 GB

Moonshot AI's Kimi K3 is one of the largest open-weight models ever announced: 2.8 trillion total parameters, 104 billion active parameters per token, native vision, a 1-million-token context window, and a sparse Mixture-of-Experts (MoE) architecture. Full-precision inference needs 1.56 TB of storage. That is a research-lab size model, not something you casually run on a workstation.

Unsloth changed that. Its 1-bit Kimi K3 GGUF quantization drops the model to 594 GB, a 62% reduction, while keeping roughly 78.9% top-1 accuracy. This guide explains what that means, how the dynamic quant works, what hardware you actually need, the available quant options, and how to run it with Unsloth Studio or llama.cpp.

Quick answer: The 1-bit GGUF makes Kimi K3 local inference possible on a large server or workstation with ~610 GB of total memory, but it is not a consumer GPU model. The dynamic quant is the reason the size drop is usable where older 1-bit methods usually collapse.

What 1-bit Kimi K3 GGUF actually is

GGUF is the file format used by llama.cpp for quantized large language models. A 1-bit quantization means the weights are stored with extremely low precision, which dramatically reduces memory and storage. Traditional 1-bit quants have a reputation for degrading quality because they throw away too much information.

Unsloth's version is different. It uses dynamic quantization plus careful calibration against the 1.56 TB lossless UD-Q8_K_XL reference. The result is a 594 GB model that Unsloth reports as usable for real tasks, not just a curiosity.

The UD-Q8_K_XL perplexity of 1.4581 is the reference Unsloth used for calibration. The 1-bit UD-IQ1_S reaches 2.5789 perplexity. That is a real loss, but Unsloth emphasizes that competing community 1-bit quants at similar sizes degrade far more: one 619 GB IQ1_M quant jumps to 54.56 perplexity, which is roughly 21× worse than Unsloth's 594 GB version. The same pattern holds for 2-bit: Unsloth's 711 GB UD-IQ2_XXS is at 2.13 PPL, while a 725 GB IQ2_XXS alternative hits 96 PPL, about 45× worse.

That is the core point: 1-bit is not automatically good; the quantization method matters enormously.

Hardware requirements

Unsloth lists total memory requirements as the sum of RAM plus VRAM, or unified memory on Apple Silicon. The important numbers are:

This means a DGX Station, a large multi-GPU server, or a Mac Studio connected to a 128 GB device is not enough for the 1-bit model by itself. The 594 GB model still needs around 610 GB of total memory when you include overhead. A Mac Studio alone tops out below that, so the reference to a Mac Studio in Unsloth's materials likely refers to smaller configurations or the broader Unsloth ecosystem, not the full Kimi K3 1-bit run.

Unsloth has demonstrated ~36 tokens per second on four NVIDIA B200 GPUs with the 1-bit model. That gives a practical sense of throughput for people with the right hardware. If your total memory is below the quant size, the model will still load but will be much slower due to disk offloading.

Why Kimi K3 is so large in the first place

Kimi K3 is a sparse MoE model. The 2.8 trillion figure is the total parameter count across all experts, but only a subset is active per token. Moonshot says 16 of 896 experts are activated per forward pass, giving 104 billion active parameters per token. The weights are stored in MXFP4 for the MoE layers and BF16 for the rest, which is why the Q8 quant is considered lossless against the released version.

The full-precision version uses 1.56 TB of storage. That is already compressed compared to a naive dense 2.8T model, but it is still enormous. The 1-bit GGUF is the next compression step for local inference.

How to run 1-bit Kimi K3 GGUF

Option 1: Unsloth Studio

Unsloth Studio is the simplest path. It is a web UI that downloads GGUF models, handles GPU/CPU offloading automatically, and can run on macOS, Windows and Linux.

1curl -fsSL https://unsloth.ai/install.sh | sh
2
3unsloth studio

Then open the local URL (usually http://127.0.0.1:8888), search the model hub for Kimi K3, select the UD-IQ1_S quant, and start inference. The studio will automatically detect available GPUs and offload layers as needed.

For a secure public tunnel over HTTPS, Unsloth supports Cloudflare tunneling:

1unsloth studio --secure

Option 2: llama.cpp with the Unsloth fork

Kimi K3 is not yet supported by mainline llama.cpp. Unsloth maintains a fork that adds vision support and the necessary model-specific fixes. You need to build it from the correct branch.

1git clone https://github.com/unslothai/llama.cpp
2cd llama.cpp
3git fetch origin pull/48/head:kimi-k3-fullsize-vision
4git checkout kimi-k3-fullsize-vision
5
6cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
7cmake --build build --config Release -j \
8  --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

Download the model from Hugging Face:

1pip install huggingface_hub
2hf download unsloth/Kimi-K3-GGUF \
3  --local-dir unsloth/Kimi-K3-GGUF \
4  --include "*mmproj-BF16*" \
5  --include "*UD-IQ1_S*"

Run in conversation mode:

1./llama.cpp/llama-cli \
2  --model unsloth/Kimi-K3-GGUF/UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00014.gguf \
3  --mmproj unsloth/Kimi-K3-GGUF/mmproj-BF16.gguf \
4  --temp 1.0 \
5  --top-p 0.95

The mmproj file is the vision projector. Without it, text-only inference will still work, but image understanding will not.

Inference settings and chat template

Kimi K3 is thinking-only. Unsloth says preserve_thinking is always enabled, and the default thinking mode is max. There is no "instant" mode. You can adjust thinking effort through the reasoning_effort request field with values "low", "high" or "max".

Unsloth also notes that the llama.cpp default n_tokens * 40 buffer was too small for Kimi K3 at large batch sizes and had to be raised to n_tokens * 160. The chat template was converted to Jinja format for llama.cpp compatibility. These are small but important details if you are trying to build a custom server rather than use Unsloth Studio.

Performance and benchmark expectations

Unsloth's documentation includes a comparison table for Kimi K3 against Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 and GLM-5.2. These are Moonshot-reported or Unsloth-reported figures, not independently audited, but they give a sense of the model's capabilities.

These numbers are not necessarily the same as what you will get from the 1-bit GGUF. Quantization reduces accuracy, especially for complex reasoning and code. The 78.9% top-1 accuracy figure means the 1-bit model agrees with the lossless model on roughly 78.9% of top-token predictions, not that it scores 78.9% on downstream benchmarks. Real-world performance will depend on the task.

What is the best quant to use?

Unsloth recommends UD-IQ1_S as the best balance between size and quality. The argument is that if you have enough memory to run the 1-bit model at all, you get a usable Kimi K3 at roughly one-third the storage of the lossless version.

If you have more memory and care more about quality, the hierarchy is roughly:

• UD-Q8_K_XL (1,560 GB): lossless reference; use this if you have the hardware.
• UD-Q4_K_XL (1,510 GB): near-lossless; only slightly smaller than Q8.
• UD-Q2_K_XL (861 GB): strong accuracy (~90.4%) and still 45% smaller than Q8.
• UD-IQ2_XXS (711 GB): good 2-bit dynamic quant; 84.1% top-1 accuracy.
• UD-IQ1_M (649 GB): larger 1-bit; 81.2% top-1 accuracy.
• UD-IQ1_S (594 GB): smallest dynamic 1-bit; 78.9% top-1 accuracy.

The rule of thumb is simple: total RAM plus VRAM should be at least the quant size. If you have 610 GB, use UD-IQ1_S. If you have 880 GB, use UD-Q2_K_XL. If you have 1.6 TB, use UD-Q8_K_XL and forget about quantization loss.

Limitations and caveats

1. Not a consumer model. Even the 1-bit version needs hundreds of gigabytes of memory. A single consumer GPU or a standard desktop will not run it.
2. Quantization loss is real. 78.9% top-1 accuracy means roughly one in five top-token predictions differs from the lossless model. On complex coding or reasoning tasks, that gap can compound.
3. Vision requires the mmproj file. Text-only inference is simpler, but the full multimodal model needs the vision projector downloaded and loaded correctly.
4. Fork dependency. You must use the Unsloth llama.cpp fork or Unsloth Studio. Mainline llama.cpp does not yet support Kimi K3 fully.
5. Thinking-only. The model always generates internal reasoning traces. If you need low-latency, non-reasoning output, this is not the right model.
6. Vendor-reported metrics. Sizes, perplexity values, accuracy figures, and benchmark comparisons come from Unsloth and Moonshot documentation. Independent verification is still needed.
7. Context length is theoretical until tested. A 1-million-token window is only useful if inference remains fast enough to search and generate within it. Long-context performance on the quantized model is not yet widely documented.

Who is this for?

The 1-bit Kimi K3 GGUF is most interesting for three groups:

• AI researchers and labs who want to run a frontier-scale open model locally for reproducibility, safety testing, or architecture experiments.
• Enterprise teams with large on-premise GPU clusters who want to keep data inside their network and avoid API pricing for high-volume workloads.
• Inference providers who are building specialized hosting services and can amortize the hardware cost across many users.

It is not for most individual developers or small teams. For those users, a smaller model like Nanbeige 4.2-3B, Qwen 3.5, or a 70B-class quantized model is a more practical local option.

Final verdict

Unsloth's 1-bit Kimi K3 GGUF is a genuine technical achievement. It takes a 1.56 TB frontier MoE model and compresses it into 594 GB while keeping enough accuracy to be usable. That makes Kimi K3 the first open model at this scale that can plausibly run on a single high-end server rather than a data-center cluster.

But the practical bar is still high. You need roughly 610 GB of total memory, a patched llama.cpp build, and realistic expectations about quantization loss. The model is thinking-only, multimodal, and long-context, which adds complexity to any deployment.

The right way to think about it is not that Kimi K3 is now "small." It is that the range of hardware that can run it has moved from supercluster to large server. For researchers, privacy-sensitive enterprises, and specialized hosting providers, that is a meaningful shift. For everyone else, the 1-bit GGUF is an impressive proof of what dynamic quantization can do—and a signal that the open-weight frontier is moving from leaderboards to actual deployable infrastructure.

Sources and further reading

• Unsloth: Kimi K3 local inference guide
• Unsloth Kimi-K3-GGUF on Hugging Face
• AtomicChat Kimi-K3-GGUF on Hugging Face
• Unsloth llama.cpp fork PR for Kimi K3 vision
• Upstream llama.cpp Kimi K3 PR
• Moonshot AI Kimi K3 announcement
• Kimi K3 on Hugging Face

Editorial note: Quantization metrics, sizes, and benchmark scores are from Unsloth and Moonshot AI documentation. They are not independently reproduced here. Check the latest Hugging Face repo and llama.cpp PR status before deploying.