llama.app Guide: Run Hugging Face Models Locally with llama.cpp, Ollama and Unsloth

Local AI has a new front door. llama.app packages the llama.cpp ecosystem behind a simpler website, installer and unified llama command, while the underlying llama.cpp engine still gives advanced users direct control over quantization, GPU offload, context and an OpenAI-compatible server.

The important correction is that no runtime can execute literally every repository on Hugging Face. A repository may contain a text model, vision model, diffusion checkpoint, adapter, dataset, unsupported architecture, gated weights or only training artifacts. The practical promise is narrower and more useful: you can run any model whose architecture and file format are supported by your chosen runtime, provided the weights and working context fit your memory.

Use the free Local AI Model Hardware Compatibility Calculator alongside this guide. It searches this site's AI model directory and estimates weight, runtime and KV-cache memory for NVIDIA GPUs, Apple M1 through M6 systems and CPU-only computers.

The shortest decision tree

What llama.app changes in 2026

The llama.cpp maintainers announced llama.app on May 29, 2026 as the official website and simpler installation path for the project. The new distribution adds a cross-platform installer and a unified llama binary around familiar tools. The official llama.cpp README lists llama.app as an installation option, while the launch announcement explains the move from many separate binaries toward one command.

For a compatible model already published as GGUF on Hugging Face, the modern flow can be as small as:

1llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

To expose the same model as a local API:

1llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The server implements an OpenAI-compatible HTTP interface, so many existing chat UIs, coding agents and scripts can point at the local endpoint. According to the llama.cpp server documentation, -hf can resolve a Hugging Face repository and choose a suitable GGUF file; when available, its default quantization preference is Q4_K_M.

This convenience does not remove model due diligence. Read the model card, confirm the license, check whether access is gated, verify the expected chat template and confirm that llama.cpp supports the architecture. The llama.cpp model documentation says the normal runtime path expects GGUF; supported Hugging Face checkpoints can be converted using the repository's conversion scripts.

Step 1: choose the model by workload, not hype

Start with the task:

• Coding: prefer an instruct or code-tuned model with a license that permits your use.
• Document analysis: prioritize context length, retrieval quality and enough memory for the intended context—not just parameter count.
• Vision: confirm that the runtime supports the model's multimodal projector and input path. llama.cpp's multimodal documentation covers image, audio and video support through its multimodal library, but support remains architecture-specific.
• Agents and tool calling: inspect the chat template and tool-call format. A model can be excellent at prose and still be unreliable at structured tool arguments.
• Fine-tuning: begin from the original supported checkpoint and preserve the tokenizer, base model and chat template through export.

On Hugging Face, inspect four things before downloading:

1. The model card and license.
2. Architecture and pipeline tags.
3. Parameter count and file sizes.
4. Available formats: GGUF, Safetensors, MLX, adapters or another runtime-specific build.

Hugging Face's GGUF guide explains that GGUF combines tensors with standardized metadata and provides a dedicated Hub filter for GGUF repositories. The Hub can also display GGUF tensor and metadata information before download.

Step 2: estimate whether it fits

The first-order weight estimate is simple:

1weight memory in GB ≈ parameters in billions × quantization bits ÷ 8

A 7B model at 4-bit is roughly 3.5GB of raw weights. Real usage is higher because the runtime needs metadata, temporary buffers and a KV cache for the working context. A 70B model at 4-bit begins around 35GB before those additions. Long context can add several more gigabytes, and the exact KV-cache cost depends on the architecture, layers, attention heads, cache precision and batch size.

That is why the hardware compatibility calculator asks for both quantization and working context. Its result is a conservative capacity estimate—not a tokens-per-second benchmark. If a model barely fits, reduce context first, close memory-heavy applications, or select a smaller quantization. For production, measure the exact build on the exact machine.

Practical memory bands

These are starting points, not guarantees. Mixture-of-experts models complicate the table: compute per token may use only a subset of experts, while memory may still need the full checkpoint. Use total parameters for fit unless the runtime explicitly supports expert offloading.

Option A: llama.app and llama.cpp

Choose this route when you want GGUF portability, strong CPU support, Metal on Apple Silicon, CUDA on NVIDIA and a lightweight local API.

After installing through llama.app or the platform instructions in the llama.cpp repository, run a compatible GGUF repository directly:

1llama cli -hf owner/model-GGUF

Choose a quantization explicitly when the repository offers several:

1llama cli -hf owner/model-GGUF:Q4_K_M

Start a local server:

1llama serve -hf owner/model-GGUF:Q4_K_M --ctx-size 8192

For NVIDIA builds from source, enable CUDA; for Apple builds, llama.cpp uses Metal support. CPU-only execution remains useful for smaller quantized models, but memory bandwidth usually becomes the limiting factor.

Convert a supported Safetensors checkpoint to GGUF

If a GGUF version does not exist, download the original checkpoint and use llama.cpp's conversion tooling:

1hf download owner/model --local-dir ./model
2python convert_hf_to_gguf.py ./model --outfile model-f16.gguf --outtype f16
3llama-quantize model-f16.gguf model-q4-k-m.gguf Q4_K_M

Exact conversion arguments and supported architectures change, so use the scripts and documentation from the same llama.cpp revision. Do not assume conversion success means semantic correctness: compare test prompts against the original model and verify the chat template.

Option B: Ollama for managed local models

Ollama is a good default when you want a persistent local model service and a concise CLI. For a model in Ollama's library:

1ollama run model-name

For a downloaded GGUF file, create a Modelfile:

1FROM ./model-q4-k-m.gguf
2PARAMETER num_ctx 8192

Then import and run it:

1ollama create my-local-model -f Modelfile
2ollama run my-local-model

The official Ollama import guide documents GGUF imports, supported Safetensors architectures and adapters. Its Modelfile reference defines FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE and other controls.

Do not copy a random prompt template from another model. If the repository supplies a template or tokenizer configuration, preserve it. A wrong template can make a valid checkpoint look broken.

Option C: Unsloth for fine-tuning and GGUF export

Unsloth belongs earlier in the lifecycle: fine-tune or merge a supported model, validate it, then export a compact artifact for llama.cpp or Ollama.

For a loaded Unsloth model and tokenizer:

1model.save_pretrained_gguf(
2    "my-model-gguf",
3    tokenizer,
4    quantization_method="q4_k_m",
5)

Or publish the export to your Hugging Face account:

1model.push_to_hub_gguf(
2    "your-name/my-model-GGUF",
3    tokenizer,
4    quantization_method="q4_k_m",
5)

The Unsloth GGUF export documentation recommends Q4_K_M and Q5_K_M among its practical quantization options. It also identifies incorrect chat templates, EOS tokens and start-of-sequence behavior as common reasons an exported model produces gibberish or repetition in another runtime.

An adapter must be applied to the same base model used during training. Ollama's import documentation gives the same warning: a mismatched base can produce erratic results.

Option D: MLX LM for Apple M1 through M6

Apple Silicon uses unified memory shared by the CPU and GPU. Capacity is therefore determined mainly by the machine's configured unified memory after macOS and other applications take their share—not by the chip generation alone.

MLX LM is Apple's open-source Python package for generation and fine-tuning on Apple Silicon. It integrates directly with Hugging Face and can load an MLX-compatible model with one command:

1pip install mlx-lm
2mlx_lm.generate \
3  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
4  --prompt "Explain this repository architecture"

It can also download a supported Hugging Face model directly or convert and quantize one:

1mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q

MLX LM documents prompt caching, rotating KV caches and prefill controls for long prompts. Its large-model notes warn that models close to total available RAM can become slow even when they technically fit.

As of September 2026, Apple's official specifications span a wide range: M-series laptops may have 16GB or 32GB, M5 Pro/Max configurations reach higher, and M5 Ultra reaches 512GB unified memory. The base M6 supports up to 32GB. This is why the calculator separates chip family from actual memory capacity. An M6 with 16GB cannot fit a model merely because it is newer than an M1 Ultra with far more memory. See Apple's M6 and M5 Ultra announcement and current Mac Studio specifications for official configurations.

Option E: Transformers and vLLM on NVIDIA

Use Transformers when you need the repository's native implementation, full-precision behavior, adapters or training ecosystem. Use vLLM when the model architecture is supported and API throughput matters. Both generally work from Safetensors rather than requiring GGUF.

Download without cloning every artifact:

1hf download owner/model --include "*.safetensors" "*.json" "tokenizer*" --local-dir ./model

The Hugging Face download guide supports dry runs, revision pinning and include/exclude patterns. Run a dry run before pulling a giant repository:

1hf download owner/model --dry-run

For reproducibility and supply-chain safety, pin a commit revision rather than silently consuming a changed main branch. Review any repository that asks for trust_remote_code=True; that flag authorizes model-provided Python code to execute locally.

NVIDIA memory is dedicated VRAM, so a 24GB RTX 4090 and 32GB RTX 5090 have different fit ceilings even if both are fast cards. NVIDIA's current official pages specify 24GB for RTX 4090, 32GB for RTX 5090, 96GB for RTX PRO 6000 Blackwell and 141GB for H200. Multi-GPU serving adds another layer: the runtime must support tensor, pipeline or expert parallelism; VRAM does not automatically become one transparent pool.

Browser hardware detection: what is possible

The calculator performs detection locally in the browser. It can usually read:

• the reported platform;
• the number of logical processors available to the browser;
• an approximate RAM bucket in browsers that implement Device Memory;
• a WebGL renderer string when the browser and privacy settings expose it.

It cannot reliably identify the exact CPU, exact NVIDIA VRAM, or whether a Mac is M1, M2, M3, M4, M5 or M6. Browsers deliberately reduce fingerprinting data. MDN notes that navigator.deviceMemory is approximate, rounded and clamped, while navigator.hardwareConcurrency reports logical processors available to the user agent—not a guaranteed physical-core specification. WebGL renderer details may also be blocked.

Therefore detection is a convenience, not the final authority. Confirm VRAM with nvidia-smi, unified memory in macOS System Information, or installed RAM in the operating system, then choose the matching profile manually.

A production-safe installation checklist

1. Confirm the model license and intended use. Open weights do not always mean unrestricted commercial use.
2. Inspect the repository and format. Prefer Safetensors over legacy pickle-based weights where possible; use a trusted GGUF conversion.
3. Pin a revision. Record the model commit, runtime version, quantization and prompt template.
4. Dry-run the download. Confirm disk requirements before transferring tens or hundreds of gigabytes.
5. Start with Q4_K_M and 4K–8K context. Raise quality or context after measuring memory headroom.
6. Validate output against the original. Use deterministic prompts and inspect tool calls, EOS behavior and repetition.
7. Bind local servers safely. Keep the API on localhost unless authentication, firewall and network exposure are intentionally configured.
8. Measure real workload performance. Record prompt-processing speed, generation speed, peak memory and time-to-first-token.

Troubleshooting

The model downloads but will not load

The architecture may be unsupported, the wrong file may have been selected, the file may be incomplete, or memory may be insufficient. Confirm the SHA/revision and test a smaller context or quantization.

Output is gibberish or never stops

Check the chat template, tokenizer, EOS token and beginning-of-sequence behavior. This is especially important after fine-tuning, merging or converting.

The model fits but is painfully slow

Capacity and speed are different questions. CPU-only inference may fit in RAM but be limited by memory bandwidth. GPU offload may leave too many layers on the CPU. On Apple Silicon, a model near the unified-memory ceiling can trigger pressure and swapping. Reduce context, use a smaller quantization or choose a smaller model.

A vision model answers text but cannot see images

The text weights may have loaded without the required multimodal projector or runtime support. Follow the runtime's model-specific multimodal instructions.

Should I use 2-bit quantization to force a larger model to fit?

Only after testing quality. Very low-bit quantization may fit but lose enough accuracy that a smaller model at Q4 or Q5 performs better for your task.

Frequently asked questions

Can llama.app run every Hugging Face model?

No. It can run compatible architectures in formats supported by llama.cpp, most commonly GGUF. Diffusion models, unsupported architectures, datasets and incomplete repositories need different tools.

Is Ollama better than llama.cpp?

Ollama is easier for managed local models and a persistent service. llama.cpp exposes more low-level controls and is the underlying choice when tuning inference behavior matters. Both can run GGUF files.

Is Unsloth an inference runtime?

It can run models, but its strongest role in this workflow is efficient fine-tuning, merging and export. Deploy the resulting GGUF with llama.cpp or Ollama, or keep Safetensors for Transformers/vLLM.

Which is best for an Apple Mac?

Start with MLX LM for an Apple-native Python workflow or llama.cpp/Ollama for GGUF portability. Actual unified-memory capacity matters more for model fit than whether the chip is M1 or M6.

How do I know which NVIDIA GPU can run a model?

Use the hardware compatibility calculator, then verify with the exact quantized file size and a real load test. Leave VRAM headroom for the KV cache and runtime buffers.

Does the browser detector upload my hardware data?

No. Detection and estimation run in the browser. The model-name search calls the site's directory endpoint, but the detected hardware values are not sent with that request.

The practical recommendation

For most users, search for an official or reputable GGUF build, estimate it against real RAM/VRAM, start at Q4_K_M with an 8K context, and run it through llama.app, llama.cpp or Ollama. Apple developers who want deeper integration should test MLX LM. NVIDIA teams that need native checkpoints, fine-tuning or high-throughput serving should keep Transformers, Unsloth and vLLM in the stack.

The model name is only the beginning. Format, template, quantization, context, memory and runtime determine whether local AI actually works.