runpod/runpod-plugins-official

runpod-vllm

Serve a Hugging Face model with vLLM on Runpod (Hub worker-vllm queue endpoint, load-balancing endpoint on a vllm/vllm-openai image, or a Pod), sized, CUDA-pinned and verified with a real OpenAI client.

查看源码

使用方法

  1. 复制安装命令,在自己的智能体环境安装前先检查源码包。
  2. 从原始文档核对所需工具、凭据和支持的宿主。
  3. 说明输入材料与期望交付物,并对照来源检查输出。
npx skills add https://github.com/runpod/runpod-plugins-official --skill runpod-vllm

开始前需了解

  • 安装不包含模型订阅、API 额度或访问文件的权限。
  • 仓库星数和安装量反映活跃情况,不等于独立质量或安全测试。
仓库原始内容

按源仓库内容呈现,保留标题、案例、代码、表格、链接以及原文引用的演示图片。

vLLM on Runpod

You take a model id and leave the user with a working OpenAI-compatible vLLM server: the URL, the auth, a tested client snippet, what it costs per hour, and what is still billing. A create call that returned an id is not a working server. The deployment is done only when `scripts/test_endpoint.py` passes against the real URL.

Creating a Pod or an endpoint bills. State the hourly price from the catalog read before each create. The generic endpoint procedure (smoke job, hold limits, hand-back fields) is runpod-serverless-deploy's and the generic Pod procedure is runpod-pod-deploy's. This skill adds what is specific to vLLM. A worked run of the Pod and load-balancing paths is golden path 27; the full index is in golden-paths/README.md.

Tool binding

If a tool named here is missing from the session's tool list, say so and use the nearest read. Never invent a result. Parameter names follow the live tool schema (describe-tool); the examples in the references show the settings to send, not a schema.

CapabilityTool
Hub worker-vllm release: listed tag, image, CUDA pin, GPU pools, env schemalist-hub-repos (repoOwner:"runpod-workers", searchTerm:"worker-vllm", includeConfig:true)
GPU price and stock per product; stock at an exact GPU count and host CUDA versionlist-gpu-types (include:["AVAILABILITY"], product:["POD"] or ["SERVERLESS"]); get-capacity (gpuCount, cudaVersions)
Queue endpoint from the Hub releasedeploy-hub-repo
Load-balancing endpoint on a vLLM imagecreate-endpoint (type:"LOAD_BALANCER")
Pod running vllm servecreate-pod (args carries the vllm serve arguments, gpu.minCudaVersion the floor)
CUDA floor or GPU change on an endpointset-endpoint-gpus or update-endpoint (gpu)
Boot progress and failuresstream-pod-logs; list-endpoint-workers + stream-worker-logs
Read back what you setget-pod, get-endpoint
Teardowndelete-pod, delete-endpoint (and the template a Hub deploy created, delete-template)

Step 1: read the facts

Read these before you choose anything. Do the reads in parallel.

  • The model. From the Hugging Face model card and config.json: parameter count, native dtype, context length, whether it is a reasoning model, whether it supports tool calls, whether an official quantized variant exists (FP8, AWQ, GPTQ). vLLM's GGUF support is limited and version-dependent: for a *.gguf repo, prefer the same model's safetensors repo on vLLM, or a GGUF engine from runpod-serverless-deploy's engine table.
  • The vLLM version the model needs. The model card or the vLLM recipe for the model (https://github.com/vllm-project/recipes) names a minimum vLLM version and the serving flags. If neither says, treat the first boot as the test.
  • The Hub worker. list-hub-repos for runpod-workers/worker-vllm gives the listed release, its image and its allowedCudaVersions. The release notes on GitHub name the vLLM version it bundles. Read it live every time: the listed release changes, and it can lag the newest GitHub release.

Step 2: choose the product

Does the Hub worker's vLLM support the model?
├── yes → Is it request/response traffic that can scale to zero?
│         ├── yes → queue endpoint (reference/queue-endpoint.md)
│         └── no, the user wants a server they keep → Pod (reference/pod.md)
└── no (new architecture, needs a newer vLLM or a nightly image)
          ├── model boots in minutes on ≤ 2 GPUs → load-balancing endpoint (reference/lb-endpoint.md)
          └── very large or slow to boot → Pod first; move to an endpoint after it passes
ProductURLAuthChoose it when
Queue endpoint (Hub worker-vllm)https://api.runpod.ai/v2/<endpoint-id>/openai/v1Runpod API keythe Hub vLLM supports the model; scale-to-zero; no image to manage
Load-balancing endpoint (vllm/vllm-openai:<tag>)https://<endpoint-id>.api.runpod.ai/v1Runpod API key at the gatewaythe model needs a vLLM version the Hub worker does not have; direct HTTP, no queue
Pod (vllm/vllm-openai:<tag>)https://<pod-id>-8000.proxy.runpod.net/v1vLLM's own API key (VLLM_API_KEY)a long-running server, debugging a boot, or a model too large or slow for the endpoint health window
Pod the user already runs (pip install, loopback)http://127.0.0.1:<port>/v1, from a shell on the PodvLLM's own API key (VLLM_API_KEY)the user names an existing Pod (for example a PyTorch template) and wants vLLM on it, private, with no new Pod

For a new or very large model, prove the config on a Pod first: a Pod has no health watchdog, so a long first boot (download plus compile) is fine and the logs stay readable. Then move the same image and arguments to an endpoint.

Step 3: size the GPUs

Show the arithmetic in the answer.

  • Weights in GB ≈ parameters (billions) × bytes per parameter: bf16/fp16 2, FP8 1, 4-bit 0.5. Qwen/Qwen3-8B in bf16 is about 16 GB.
  • Add room for the KV cache and activations: at least 20–30 % of the weights for a short context, more for a long one. Lower --max-model-len when the model's native context does not fit (the boot fails with a KV-cache error that names the largest length that fits).
  • One card is best. When one card cannot hold the model, set the GPU count to the next of 1, 2, 4 or 8 and set tensor parallelism to that count (--tensor-parallel-size, or TENSOR_PARALLEL_SIZE on the Hub worker). Say that the cost scales with the count.
  • Read stock for the product you create (product:["POD"] for a Pod, ["SERVERLESS"] for an endpoint), at the GPU count you will use, and at the host CUDA versions that meet the image floor (get-capacity with gpuCount and cudaVersions). Stock for one product, count or CUDA version says nothing about another.

Step 4: pin the host CUDA floor

vllm/vllm-openai:v0.32.0 and the Hub worker-vllm image built on it are CUDA 13 builds (NVIDIA_REQUIRE_CUDA=cuda>=13.0); read that variable in the image config for any other tag. Set gpu.minCudaVersion to the image floor on every create. The general rule and why: gpu-selection.md step 3.

Prefer the open floor (minCudaVersion) to an exact list (allowedCudaVersions): the CUDA version is a property of each machine's driver and differs between machines of one GPU type, so an exact list matches only the hosts that report exactly those versions. A Hub release can carry an exact list, and deploy-hub-repo copies it to the endpoint. Read the endpoint back after the deploy. If the workers stay at zero or throttled while the stock read shows capacity at a newer CUDA version, widen it with set-endpoint-gpus (allowedCudaVersions:"", minCudaVersion:"13.0").

Step 5: set the vLLM flags

The flags and what each one costs are in reference/serving-flags.md. The short list:

  • --max-model-len: cap it to what the job needs and the KV cache holds.
  • --gpu-memory-utilization 0.90: leave the default unless the boot log says otherwise.
  • --tensor-parallel-size N = GPU count.
  • Tool calls: --enable-auto-tool-choice --tool-call-parser <parser>. Reasoning models: --reasoning-parser <parser>. Take the parser names from the model card or the vLLM recipe.
  • A quantized checkpoint (FP8, AWQ, GPTQ) needs no flag: vLLM reads the method from the checkpoint config.
  • Gated weights: HF_TOKEN in the env. Never print it.

Step 6: deploy and watch the boot

Follow exactly one reference: queue-endpoint.md, lb-endpoint.md, pod.md or, for a Pod the user already runs, existing-pod.md (its verify step replaces Step 7: the server is not reachable from outside the Pod).

Watch the boot in short reads, never one long blind wait. Each read checks three things: the resource still exists, its status, and the log tail. Judge progress by forward movement (download, shard load, compile, CUDA graph capture, Application startup complete), not by the word ERROR: a healthy vLLM boot prints warnings. The failure signatures and their fixes are in reference/boot-log-triage.md. On a known crash loop, stop the spend first (delete the Pod, or set the endpoint's workers.max to 0), then diagnose.

The first log lines prove which settings took effect: vLLM prints non-default args: {...} at startup. Check it against what you set before you wait for the rest of the boot.

Step 7: verify with a real client

bash
pip install openai
python3 scripts/test_endpoint.py --base-url <url>/v1 --api-key-env RUNPOD_API_KEY --model <served-model-id> \
  --wait-for-ready 1800 [--thinking-switch] [--skip-tools] [--vision]

The key comes from the named environment variable, so it stays out of the process list: RUNPOD_API_KEY for an endpoint, VLLM_API_KEY for a Pod.

The script waits for GET /v1/models, then tests chat, streaming (time to first token), tool calls and, with flags, the Qwen-style thinking switch and an image input. It exits non-zero if any enabled test fails. A model that answers with garbage or repeats itself has the wrong parser or chat template: fix the server before you report anything.

The first request after a boot can be several times slower than the next one (kernels compile on first use). Quote both numbers if you quote latency.

Step 8: hand back and clean up

The final message is standalone:

  • product, model id, image or Hub release, GPU type × count, data center, hourly price;
  • the sizing arithmetic and the flags you set, with the reason for each non-default flag;
  • the base URL, which key authenticates it, and a working OpenAI SDK snippet that reads the key from an environment variable;
  • the test result: each check, pass or fail, with the measured first-request and warm latency;
  • what is billing now, and how to stop it.

Default to keeping what the user asked for. Tear down when the user framed the deployment as a test or asks for cleanup: delete the Pod or endpoint (and the template a Hub deploy created), then confirm with list-pods / list-endpoints that nothing you created is left.

Hard rules

  • Never claim the server works without a passing test_endpoint.py run, or the quoted failure.
  • Never put a vLLM --api-key on a load-balancing endpoint: the gateway checks the Runpod API key, and a second key makes every request fail. A Pod is the opposite: its proxy URL is public, so it always gets a random VLLM_API_KEY.
  • Never change the config of a warm endpoint during a test without saying it: an image or env change cuts a new release and replaces the workers with cold ones.
  • Deploy only what the user asked for, and change or delete only what this conversation created.

Contract

The cross-journey answer contract is in the runpod-mcp skill. If it is not loaded in this conversation, load it before the next tool call.

来自同一仓库

更多 Skills

全部 Skills
runpod
社区

runpod

Start here when the Runpod lane is unclear: picks the skill for a task (runpod-mcp and the journey playbooks it routes to, such as runpod-serverless-deploy, runpod-pod-deploy, runpod-pod-doctor and runpod-cost-audit; runpodctl, runpod-flash, runpod-companion-clis, runpod-usage, runpod-templates, runpod-build-template, runpod-migrate, runpod-cloud-migrate) and indexes two dozen end-to-end worked examples (golden paths). Use it before any multi-step or provisioning task, even when the lane is clea

1小时安装量
3
GitHub Stars
50
最近更新
11月19日
runpod
社区

runpod-build-template

Create or improve a complete Runpod template project from an application request, local code, or GitHub repository, with a documented Dockerfile and app-specific files for Pods or queue/load-balanced Serverless. Use runpod-templates for finding or operating existing official templates, runpod-mcp/runpodctl to register an already-built image as a template, runpod-flash for its code-first workflow, and runpod-companion-clis for standalone image build/push commands.

1小时安装量
3
GitHub Stars
50
最近更新
11月19日
runpod
社区

runpod-cloud-migrate

Move a workload from another cloud or GPU platform to Runpod: Vast.ai, Lambda, CoreWeave, Paperspace, Modal, Replicate, Baseten, Together AI, Fireworks AI, fal, Hugging Face Inference Endpoints, AWS, Google Cloud or Azure. Inventories what the project uses, maps it to Runpod, plans, then migrates one workload at a time with verification. To move Runpod code between Runpod API versions, use runpod-migrate instead.

1小时安装量
3
GitHub Stars
50
最近更新
11月19日
runpod
社区

runpod-companion-clis

Companion CLIs for Runpod workflows — HuggingFace, GitHub, Docker, and AWS. Use runpod-build-template for complete template project authoring, and the ComfyUI model-repair guide in runpod-templates for imported workflows lacking model download metadata.

1小时安装量
3
GitHub Stars
50
最近更新
11月19日