Verwendung
- Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
- Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
- Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/runpod/runpod-plugins-official --skill runpod-vllmVor dem Start
- Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
- Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.
vLLM on Runpod
You take a model id and leave the user with a working OpenAI-compatible vLLM server: the URL, the auth, a tested client snippet, what it costs per hour, and what is still billing. A create call that returned an id is not a working server. The deployment is done only when `scripts/test_endpoint.py` passes against the real URL.
Creating a Pod or an endpoint bills. State the hourly price from the catalog read before each create. The generic endpoint procedure (smoke job, hold limits, hand-back fields) is runpod-serverless-deploy's and the generic Pod procedure is runpod-pod-deploy's. This skill adds what is specific to vLLM. A worked run of the Pod and load-balancing paths is golden path 27; the full index is in golden-paths/README.md.
Tool binding
If a tool named here is missing from the session's tool list, say so and use the nearest read. Never invent a result. Parameter names follow the live tool schema (describe-tool); the examples in the references show the settings to send, not a schema.
| Capability | Tool |
|---|---|
| Hub worker-vllm release: listed tag, image, CUDA pin, GPU pools, env schema | list-hub-repos (repoOwner:"runpod-workers", searchTerm:"worker-vllm", includeConfig:true) |
| GPU price and stock per product; stock at an exact GPU count and host CUDA version | list-gpu-types (include:["AVAILABILITY"], product:["POD"] or ["SERVERLESS"]); get-capacity (gpuCount, cudaVersions) |
| Queue endpoint from the Hub release | deploy-hub-repo |
| Load-balancing endpoint on a vLLM image | create-endpoint (type:"LOAD_BALANCER") |
Pod running vllm serve | create-pod (args carries the vllm serve arguments, gpu.minCudaVersion the floor) |
| CUDA floor or GPU change on an endpoint | set-endpoint-gpus or update-endpoint (gpu) |
| Boot progress and failures | stream-pod-logs; list-endpoint-workers + stream-worker-logs |
| Read back what you set | get-pod, get-endpoint |
| Teardown | delete-pod, delete-endpoint (and the template a Hub deploy created, delete-template) |
Step 1: read the facts
Read these before you choose anything. Do the reads in parallel.
- The model. From the Hugging Face model card and
config.json: parameter count, native dtype, context length, whether it is a reasoning model, whether it supports tool calls, whether an official quantized variant exists (FP8, AWQ, GPTQ). vLLM's GGUF support is limited and version-dependent: for a*.ggufrepo, prefer the same model's safetensors repo on vLLM, or a GGUF engine from runpod-serverless-deploy's engine table. - The vLLM version the model needs. The model card or the vLLM recipe for the model (
https://github.com/vllm-project/recipes) names a minimum vLLM version and the serving flags. If neither says, treat the first boot as the test. - The Hub worker.
list-hub-reposforrunpod-workers/worker-vllmgives the listed release, its image and itsallowedCudaVersions. The release notes on GitHub name the vLLM version it bundles. Read it live every time: the listed release changes, and it can lag the newest GitHub release.
Step 2: choose the product
Does the Hub worker's vLLM support the model?
├── yes → Is it request/response traffic that can scale to zero?
│ ├── yes → queue endpoint (reference/queue-endpoint.md)
│ └── no, the user wants a server they keep → Pod (reference/pod.md)
└── no (new architecture, needs a newer vLLM or a nightly image)
├── model boots in minutes on ≤ 2 GPUs → load-balancing endpoint (reference/lb-endpoint.md)
└── very large or slow to boot → Pod first; move to an endpoint after it passes| Product | URL | Auth | Choose it when |
|---|---|---|---|
| Queue endpoint (Hub worker-vllm) | https://api.runpod.ai/v2/<endpoint-id>/openai/v1 | Runpod API key | the Hub vLLM supports the model; scale-to-zero; no image to manage |
Load-balancing endpoint (vllm/vllm-openai:<tag>) | https://<endpoint-id>.api.runpod.ai/v1 | Runpod API key at the gateway | the model needs a vLLM version the Hub worker does not have; direct HTTP, no queue |
Pod (vllm/vllm-openai:<tag>) | https://<pod-id>-8000.proxy.runpod.net/v1 | vLLM's own API key (VLLM_API_KEY) | a long-running server, debugging a boot, or a model too large or slow for the endpoint health window |
| Pod the user already runs (pip install, loopback) | http://127.0.0.1:<port>/v1, from a shell on the Pod | vLLM's own API key (VLLM_API_KEY) | the user names an existing Pod (for example a PyTorch template) and wants vLLM on it, private, with no new Pod |
For a new or very large model, prove the config on a Pod first: a Pod has no health watchdog, so a long first boot (download plus compile) is fine and the logs stay readable. Then move the same image and arguments to an endpoint.
Step 3: size the GPUs
Show the arithmetic in the answer.
- Weights in GB ≈ parameters (billions) × bytes per parameter: bf16/fp16 2, FP8 1, 4-bit 0.5.
Qwen/Qwen3-8Bin bf16 is about 16 GB. - Add room for the KV cache and activations: at least 20–30 % of the weights for a short context, more for a long one. Lower
--max-model-lenwhen the model's native context does not fit (the boot fails with a KV-cache error that names the largest length that fits). - One card is best. When one card cannot hold the model, set the GPU count to the next of 1, 2, 4 or 8 and set tensor parallelism to that count (
--tensor-parallel-size, orTENSOR_PARALLEL_SIZEon the Hub worker). Say that the cost scales with the count. - Read stock for the product you create (
product:["POD"]for a Pod,["SERVERLESS"]for an endpoint), at the GPU count you will use, and at the host CUDA versions that meet the image floor (get-capacitywithgpuCountandcudaVersions). Stock for one product, count or CUDA version says nothing about another.
Step 4: pin the host CUDA floor
vllm/vllm-openai:v0.32.0 and the Hub worker-vllm image built on it are CUDA 13 builds (NVIDIA_REQUIRE_CUDA=cuda>=13.0); read that variable in the image config for any other tag. Set gpu.minCudaVersion to the image floor on every create. The general rule and why: gpu-selection.md step 3.
Prefer the open floor (minCudaVersion) to an exact list (allowedCudaVersions): the CUDA version is a property of each machine's driver and differs between machines of one GPU type, so an exact list matches only the hosts that report exactly those versions. A Hub release can carry an exact list, and deploy-hub-repo copies it to the endpoint. Read the endpoint back after the deploy. If the workers stay at zero or throttled while the stock read shows capacity at a newer CUDA version, widen it with set-endpoint-gpus (allowedCudaVersions:"", minCudaVersion:"13.0").
Step 5: set the vLLM flags
The flags and what each one costs are in reference/serving-flags.md. The short list:
--max-model-len: cap it to what the job needs and the KV cache holds.--gpu-memory-utilization 0.90: leave the default unless the boot log says otherwise.--tensor-parallel-size N= GPU count.- Tool calls:
--enable-auto-tool-choice --tool-call-parser <parser>. Reasoning models:--reasoning-parser <parser>. Take the parser names from the model card or the vLLM recipe. - A quantized checkpoint (FP8, AWQ, GPTQ) needs no flag: vLLM reads the method from the checkpoint config.
- Gated weights:
HF_TOKENin the env. Never print it.
Step 6: deploy and watch the boot
Follow exactly one reference: queue-endpoint.md, lb-endpoint.md, pod.md or, for a Pod the user already runs, existing-pod.md (its verify step replaces Step 7: the server is not reachable from outside the Pod).
Watch the boot in short reads, never one long blind wait. Each read checks three things: the resource still exists, its status, and the log tail. Judge progress by forward movement (download, shard load, compile, CUDA graph capture, Application startup complete), not by the word ERROR: a healthy vLLM boot prints warnings. The failure signatures and their fixes are in reference/boot-log-triage.md. On a known crash loop, stop the spend first (delete the Pod, or set the endpoint's workers.max to 0), then diagnose.
The first log lines prove which settings took effect: vLLM prints non-default args: {...} at startup. Check it against what you set before you wait for the rest of the boot.
Step 7: verify with a real client
pip install openai
python3 scripts/test_endpoint.py --base-url <url>/v1 --api-key-env RUNPOD_API_KEY --model <served-model-id> \
--wait-for-ready 1800 [--thinking-switch] [--skip-tools] [--vision]The key comes from the named environment variable, so it stays out of the process list: RUNPOD_API_KEY for an endpoint, VLLM_API_KEY for a Pod.
The script waits for GET /v1/models, then tests chat, streaming (time to first token), tool calls and, with flags, the Qwen-style thinking switch and an image input. It exits non-zero if any enabled test fails. A model that answers with garbage or repeats itself has the wrong parser or chat template: fix the server before you report anything.
The first request after a boot can be several times slower than the next one (kernels compile on first use). Quote both numbers if you quote latency.
Step 8: hand back and clean up
The final message is standalone:
- product, model id, image or Hub release, GPU type × count, data center, hourly price;
- the sizing arithmetic and the flags you set, with the reason for each non-default flag;
- the base URL, which key authenticates it, and a working OpenAI SDK snippet that reads the key from an environment variable;
- the test result: each check, pass or fail, with the measured first-request and warm latency;
- what is billing now, and how to stop it.
Default to keeping what the user asked for. Tear down when the user framed the deployment as a test or asks for cleanup: delete the Pod or endpoint (and the template a Hub deploy created), then confirm with list-pods / list-endpoints that nothing you created is left.
Hard rules
- Never claim the server works without a passing
test_endpoint.pyrun, or the quoted failure. - Never put a vLLM
--api-keyon a load-balancing endpoint: the gateway checks the Runpod API key, and a second key makes every request fail. A Pod is the opposite: its proxy URL is public, so it always gets a randomVLLM_API_KEY. - Never change the config of a warm endpoint during a test without saying it: an image or env change cuts a new release and replaces the workers with cold ones.
- Deploy only what the user asked for, and change or delete only what this conversation created.
Contract
The cross-journey answer contract is in the runpod-mcp skill. If it is not loaded in this conversation, load it before the next tool call.