SGLang Tutorial: Fast OpenAI-Compatible LLM Serving

SGLang Tutorial: Run a Fast OpenAI-Compatible LLM Server

SGLang is an open-source inference server for LLMs and multimodal models. It turns a supported model into an OpenAI-compatible API while adding continuous batching, prefix caching, structured outputs, tool calling and distributed serving options.

This tutorial starts with one GPU and one model, then shows how to call the server from curl and the OpenAI Python client. It is aimed at developers who want to self-host a model for an app or agent workflow—not train a model from scratch.

Courtesy: SGLang Project. Source: https://github.com/sgl-project/sglang. Accessed: July 29, 2026.

What you need before starting

• A Linux machine with a supported accelerator. NVIDIA CUDA is the most common path; SGLang also documents AMD, TPU, CPU and other backend options.
• Python and a driver/runtime combination compatible with the SGLang package you install.
• Enough GPU memory for your chosen model, its KV cache and concurrent requests.
• A Hugging Face model you are licensed and authorised to run.

Start small. A 7B–8B instruct model is a better first server than a giant mixture-of-experts checkpoint. It makes installation, memory sizing and API debugging much easier.

1. Install SGLang

Use a clean Python environment. The exact wheel and CUDA choice changes over time, so check the official installation guide for your platform before installing.

1python -m venv .venv
2source .venv/bin/activate
3python -m pip install --upgrade pip
4pip install sglang

SGLang’s documentation says its default package targets CUDA 13. For CUDA 12, use the documented CUDA-specific package or container tag rather than mixing arbitrary CUDA libraries. If you see CUDA_HOME errors, set it to the installed CUDA root only after confirming the driver/toolkit versions.

2. Start an inference server

The following command launches a Qwen instruct model on the local machine and exposes an HTTP server on port 30000:

1python -m sglang.launch_server \
2  --model-path Qwen/Qwen2.5-7B-Instruct \
3  --host 127.0.0.1 \
4  --port 30000

Keeping the first server bound to 127.0.0.1 is intentional. An inference endpoint can expose prompts, generated text and sometimes tool capabilities. Put a reverse proxy, TLS, authentication, rate limits and network policy in front of it before making it reachable outside the host.

If the process cannot load the model, check GPU memory first. Reduce the model size or concurrency before trying random performance flags. If it starts but is unexpectedly slow, verify that the required attention kernels are installed and that it did not fall back to a CPU or generic PyTorch path.

3. Send a chat request with curl

SGLang supports OpenAI-compatible endpoints. With the server running locally:

1curl http://127.0.0.1:30000/v1/chat/completions \
2  -H 'Content-Type: application/json' \
3  -d '{
4    "model": "Qwen/Qwen2.5-7B-Instruct",
5    "messages": [
6      {"role": "system", "content": "You are a concise technical assistant."},
7      {"role": "user", "content": "Explain prefix caching in two sentences."}
8    ],
9    "temperature": 0.2,
10    "max_tokens": 160
11  }'

The response shape is familiar to applications already built for OpenAI’s chat-completions API. That portability is the point: your product can keep an OpenAI-style client while routing selected workloads to infrastructure you operate.

4. Call it from Python

Use the official OpenAI client with a local base_url:

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://127.0.0.1:30000/v1",
5    api_key="not-used-for-local-dev",
6)
7
8response = client.chat.completions.create(
9    model="Qwen/Qwen2.5-7B-Instruct",
10    messages=[
11        {"role": "user", "content": "Return valid JSON with a title and three tags for SGLang."}
12    ],
13    temperature=0.2,
14)
15
16print(response.choices[0].message.content)

For production, do not hard-code the endpoint or an API key. Put them in a secret manager or environment configuration, rotate credentials and audit which service is allowed to call the model.

5. Add structured output before adding tools

The safest first integration is often structured extraction, not unrestricted tool use. Ask the model for a schema-constrained result, validate it in your application, and reject output that fails validation.

For example, a support-triage service could accept a model result only after it passes a JSON schema containing a fixed category enum, a confidence range and a short rationale. The model proposes; ordinary application code decides whether the result is usable.

SGLang documents OpenAI-compatible structured-output support for JSON, regular expressions and EBNF grammars. Use the structured outputs documentation for the current request format instead of relying on a copied flag from an old blog post.

6. Use function calling with real guardrails

SGLang can serve models that emit tool calls. That is useful for an agent that searches documentation, checks a test result or retrieves a read-only record. It is not permission to hand a model a production shell.

A safe pattern is:

1. Define a small allowlist of functions with strict JSON schemas.
2. Validate every function name and argument server-side.
3. Run risky operations in a sandbox with timeouts, egress controls and no ambient credentials.
4. Require a human approval step for deployments, payments, data deletion, credential access and external messages.
5. Log the prompt, tool request, validation result and tool output with sensitive values redacted.

This pattern pairs well with MCP: MCP standardises the tool boundary, while SGLang can provide the local model-serving endpoint behind an agent client.

Where SGLang earns its complexity

These are not automatic speed buttons. Benchmark a representative workload: prompt length distribution, output length, concurrency, cache hit rate, model, quantisation and hardware. A benchmark using a single short prompt can hide the costs your production workload actually has.

Docker is a good next step

Once the local server works, containerise it so CUDA versions, Python packages and launch arguments are reproducible. SGLang publishes container images; use the official docs to select the matching CUDA tag. Mount model-cache storage deliberately, set a memory and GPU policy, and make the server health check part of your deployment.

Do not expose the container’s port directly to the public internet. Terminate TLS at a proxy, authenticate callers and cap request size, tokens, concurrency and timeouts. Log latency and GPU-memory pressure, but avoid storing raw prompts by default if they may contain customer or proprietary data.

Troubleshooting checklist

• Out of memory: use a smaller model, lower concurrency or context limit, then verify KV-cache settings.
• Slow first request: expected model download/initialisation; distinguish it from steady-state latency.
• Slow all requests: check GPU visibility, CUDA compatibility and kernel fallbacks.
• Bad JSON/tool calls: lower temperature, use structured output, validate schemas and test the exact model template.
• Different answers after a model swap: test prompts, tokenisation, tool-call format and safety policy; OpenAI compatibility is an API shape, not behavioural identity.

Related reading

• MiniCPM5-1B local tool-use guide
• Qwythos-27B-v1 open agent model analysis
• Speculative decoding for production serving

Sources

1. SGLang GitHub repository
2. SGLang installation guide
3. SGLang OpenAI-compatible APIs
4. SGLang structured outputs