runpod/runpod-plugins-official

runpod-vllm

Serve a Hugging Face model with vLLM on Runpod (Hub worker-vllm queue endpoint, load-balancing endpoint on a vllm/vllm-openai image, or a Pod), sized, CUDA-pinned and verified with a real OpenAI client.

Zobacz źródło

Jak używać

  1. Skopiuj polecenie instalacji i sprawdź źródła przed instalacją we własnym środowisku agenta.
  2. W oryginale sprawdź wymagane narzędzia, dane dostępowe i obsługiwane środowiska.
  3. Opisz dane wejściowe i oczekiwany rezultat; porównaj wynik ze źródłami.
npx skills add https://github.com/runpod/runpod-plugins-official --skill runpod-vllm

Przed rozpoczęciem

  • Instalacja nie obejmuje subskrypcji modelu, limitu API ani uprawnień do plików.
  • Gwiazdki i instalacje pokazują aktywność, a nie niezależny test jakości lub bezpieczeństwa.
Oryginalny dokument Skill

Treść z repozytorium z zachowaniem nagłówków, przykładów, kodu, tabel, linków i obrazów.

vLLM on Runpod

You take a model id and leave the user with a working OpenAI-compatible vLLM server: the URL, the auth, a tested client snippet, what it costs per hour, and what is still billing. A create call that returned an id is not a working server. The deployment is done only when `scripts/test_endpoint.py` passes against the real URL.

Creating a Pod or an endpoint bills. State the hourly price from the catalog read before each create. The generic endpoint procedure (smoke job, hold limits, hand-back fields) is runpod-serverless-deploy's and the generic Pod procedure is runpod-pod-deploy's. This skill adds what is specific to vLLM. A worked run of the Pod and load-balancing paths is golden path 27; the full index is in golden-paths/README.md.

Tool binding

If a tool named here is missing from the session's tool list, say so and use the nearest read. Never invent a result. Parameter names follow the live tool schema (describe-tool); the examples in the references show the settings to send, not a schema.

CapabilityTool
Hub worker-vllm release: listed tag, image, CUDA pin, GPU pools, env schemalist-hub-repos (repoOwner:"runpod-workers", searchTerm:"worker-vllm", includeConfig:true)
GPU price and stock per product; stock at an exact GPU count and host CUDA versionlist-gpu-types (include:["AVAILABILITY"], product:["POD"] or ["SERVERLESS"]); get-capacity (gpuCount, cudaVersions)
Queue endpoint from the Hub releasedeploy-hub-repo
Load-balancing endpoint on a vLLM imagecreate-endpoint (type:"LOAD_BALANCER")
Pod running vllm servecreate-pod (args carries the vllm serve arguments, gpu.minCudaVersion the floor)
CUDA floor or GPU change on an endpointset-endpoint-gpus or update-endpoint (gpu)
Boot progress and failuresstream-pod-logs; list-endpoint-workers + stream-worker-logs
Read back what you setget-pod, get-endpoint
Teardowndelete-pod, delete-endpoint (and the template a Hub deploy created, delete-template)

Step 1: read the facts

Read these before you choose anything. Do the reads in parallel.

  • The model. From the Hugging Face model card and config.json: parameter count, native dtype, context length, whether it is a reasoning model, whether it supports tool calls, whether an official quantized variant exists (FP8, AWQ, GPTQ). vLLM's GGUF support is limited and version-dependent: for a *.gguf repo, prefer the same model's safetensors repo on vLLM, or a GGUF engine from runpod-serverless-deploy's engine table.
  • The vLLM version the model needs. The model card or the vLLM recipe for the model (https://github.com/vllm-project/recipes) names a minimum vLLM version and the serving flags. If neither says, treat the first boot as the test.
  • The Hub worker. list-hub-repos for runpod-workers/worker-vllm gives the listed release, its image and its allowedCudaVersions. The release notes on GitHub name the vLLM version it bundles. Read it live every time: the listed release changes, and it can lag the newest GitHub release.

Step 2: choose the product

Does the Hub worker's vLLM support the model?
├── yes → Is it request/response traffic that can scale to zero?
│         ├── yes → queue endpoint (reference/queue-endpoint.md)
│         └── no, the user wants a server they keep → Pod (reference/pod.md)
└── no (new architecture, needs a newer vLLM or a nightly image)
          ├── model boots in minutes on ≤ 2 GPUs → load-balancing endpoint (reference/lb-endpoint.md)
          └── very large or slow to boot → Pod first; move to an endpoint after it passes
ProductURLAuthChoose it when
Queue endpoint (Hub worker-vllm)https://api.runpod.ai/v2/<endpoint-id>/openai/v1Runpod API keythe Hub vLLM supports the model; scale-to-zero; no image to manage
Load-balancing endpoint (vllm/vllm-openai:<tag>)https://<endpoint-id>.api.runpod.ai/v1Runpod API key at the gatewaythe model needs a vLLM version the Hub worker does not have; direct HTTP, no queue
Pod (vllm/vllm-openai:<tag>)https://<pod-id>-8000.proxy.runpod.net/v1vLLM's own API key (VLLM_API_KEY)a long-running server, debugging a boot, or a model too large or slow for the endpoint health window
Pod the user already runs (pip install, loopback)http://127.0.0.1:<port>/v1, from a shell on the PodvLLM's own API key (VLLM_API_KEY)the user names an existing Pod (for example a PyTorch template) and wants vLLM on it, private, with no new Pod

For a new or very large model, prove the config on a Pod first: a Pod has no health watchdog, so a long first boot (download plus compile) is fine and the logs stay readable. Then move the same image and arguments to an endpoint.

Step 3: size the GPUs

Show the arithmetic in the answer.

  • Weights in GB ≈ parameters (billions) × bytes per parameter: bf16/fp16 2, FP8 1, 4-bit 0.5. Qwen/Qwen3-8B in bf16 is about 16 GB.
  • Add room for the KV cache and activations: at least 20–30 % of the weights for a short context, more for a long one. Lower --max-model-len when the model's native context does not fit (the boot fails with a KV-cache error that names the largest length that fits).
  • One card is best. When one card cannot hold the model, set the GPU count to the next of 1, 2, 4 or 8 and set tensor parallelism to that count (--tensor-parallel-size, or TENSOR_PARALLEL_SIZE on the Hub worker). Say that the cost scales with the count.
  • Read stock for the product you create (product:["POD"] for a Pod, ["SERVERLESS"] for an endpoint), at the GPU count you will use, and at the host CUDA versions that meet the image floor (get-capacity with gpuCount and cudaVersions). Stock for one product, count or CUDA version says nothing about another.

Step 4: pin the host CUDA floor

vllm/vllm-openai:v0.32.0 and the Hub worker-vllm image built on it are CUDA 13 builds (NVIDIA_REQUIRE_CUDA=cuda>=13.0); read that variable in the image config for any other tag. Set gpu.minCudaVersion to the image floor on every create. The general rule and why: gpu-selection.md step 3.

Prefer the open floor (minCudaVersion) to an exact list (allowedCudaVersions): the CUDA version is a property of each machine's driver and differs between machines of one GPU type, so an exact list matches only the hosts that report exactly those versions. A Hub release can carry an exact list, and deploy-hub-repo copies it to the endpoint. Read the endpoint back after the deploy. If the workers stay at zero or throttled while the stock read shows capacity at a newer CUDA version, widen it with set-endpoint-gpus (allowedCudaVersions:"", minCudaVersion:"13.0").

Step 5: set the vLLM flags

The flags and what each one costs are in reference/serving-flags.md. The short list:

  • --max-model-len: cap it to what the job needs and the KV cache holds.
  • --gpu-memory-utilization 0.90: leave the default unless the boot log says otherwise.
  • --tensor-parallel-size N = GPU count.
  • Tool calls: --enable-auto-tool-choice --tool-call-parser <parser>. Reasoning models: --reasoning-parser <parser>. Take the parser names from the model card or the vLLM recipe.
  • A quantized checkpoint (FP8, AWQ, GPTQ) needs no flag: vLLM reads the method from the checkpoint config.
  • Gated weights: HF_TOKEN in the env. Never print it.

Step 6: deploy and watch the boot

Follow exactly one reference: queue-endpoint.md, lb-endpoint.md, pod.md or, for a Pod the user already runs, existing-pod.md (its verify step replaces Step 7: the server is not reachable from outside the Pod).

Watch the boot in short reads, never one long blind wait. Each read checks three things: the resource still exists, its status, and the log tail. Judge progress by forward movement (download, shard load, compile, CUDA graph capture, Application startup complete), not by the word ERROR: a healthy vLLM boot prints warnings. The failure signatures and their fixes are in reference/boot-log-triage.md. On a known crash loop, stop the spend first (delete the Pod, or set the endpoint's workers.max to 0), then diagnose.

The first log lines prove which settings took effect: vLLM prints non-default args: {...} at startup. Check it against what you set before you wait for the rest of the boot.

Step 7: verify with a real client

bash
pip install openai
python3 scripts/test_endpoint.py --base-url <url>/v1 --api-key-env RUNPOD_API_KEY --model <served-model-id> \
  --wait-for-ready 1800 [--thinking-switch] [--skip-tools] [--vision]

The key comes from the named environment variable, so it stays out of the process list: RUNPOD_API_KEY for an endpoint, VLLM_API_KEY for a Pod.

The script waits for GET /v1/models, then tests chat, streaming (time to first token), tool calls and, with flags, the Qwen-style thinking switch and an image input. It exits non-zero if any enabled test fails. A model that answers with garbage or repeats itself has the wrong parser or chat template: fix the server before you report anything.

The first request after a boot can be several times slower than the next one (kernels compile on first use). Quote both numbers if you quote latency.

Step 8: hand back and clean up

The final message is standalone:

  • product, model id, image or Hub release, GPU type × count, data center, hourly price;
  • the sizing arithmetic and the flags you set, with the reason for each non-default flag;
  • the base URL, which key authenticates it, and a working OpenAI SDK snippet that reads the key from an environment variable;
  • the test result: each check, pass or fail, with the measured first-request and warm latency;
  • what is billing now, and how to stop it.

Default to keeping what the user asked for. Tear down when the user framed the deployment as a test or asks for cleanup: delete the Pod or endpoint (and the template a Hub deploy created), then confirm with list-pods / list-endpoints that nothing you created is left.

Hard rules

  • Never claim the server works without a passing test_endpoint.py run, or the quoted failure.
  • Never put a vLLM --api-key on a load-balancing endpoint: the gateway checks the Runpod API key, and a second key makes every request fail. A Pod is the opposite: its proxy URL is public, so it always gets a random VLLM_API_KEY.
  • Never change the config of a warm endpoint during a test without saying it: an image or env change cuts a new release and replaces the workers with cold ones.
  • Deploy only what the user asked for, and change or delete only what this conversation created.

Contract

The cross-journey answer contract is in the runpod-mcp skill. If it is not loaded in this conversation, load it before the next tool call.

z tego samego repozytorium

Więcej Skills

Wszystkie Skills
runpod
Społeczność

runpod

Start here when the Runpod lane is unclear: picks the skill for a task (runpod-mcp and the journey playbooks it routes to, such as runpod-serverless-deploy, runpod-pod-deploy, runpod-pod-doctor and runpod-cost-audit; runpodctl, runpod-flash, runpod-companion-clis, runpod-usage, runpod-templates, runpod-build-template, runpod-migrate, runpod-cloud-migrate) and indexes two dozen end-to-end worked examples (golden paths). Use it before any multi-step or provisioning task, even when the lane is clea

Instalacje / 1h
3
GitHub Stars
50
Aktualizacja
19 lis
runpod
Społeczność

runpod-build-template

Create or improve a complete Runpod template project from an application request, local code, or GitHub repository, with a documented Dockerfile and app-specific files for Pods or queue/load-balanced Serverless. Use runpod-templates for finding or operating existing official templates, runpod-mcp/runpodctl to register an already-built image as a template, runpod-flash for its code-first workflow, and runpod-companion-clis for standalone image build/push commands.

Instalacje / 1h
3
GitHub Stars
50
Aktualizacja
19 lis
runpod
Społeczność

runpod-cloud-migrate

Move a workload from another cloud or GPU platform to Runpod: Vast.ai, Lambda, CoreWeave, Paperspace, Modal, Replicate, Baseten, Together AI, Fireworks AI, fal, Hugging Face Inference Endpoints, AWS, Google Cloud or Azure. Inventories what the project uses, maps it to Runpod, plans, then migrates one workload at a time with verification. To move Runpod code between Runpod API versions, use runpod-migrate instead.

Instalacje / 1h
3
GitHub Stars
50
Aktualizacja
19 lis
runpod
Społeczność

runpod-companion-clis

Companion CLIs for Runpod workflows — HuggingFace, GitHub, Docker, and AWS. Use runpod-build-template for complete template project authoring, and the ComfyUI model-repair guide in runpod-templates for imported workflows lacking model download metadata.

Instalacje / 1h
3
GitHub Stars
50
Aktualizacja
19 lis