See it in action
GPU scheduling: capture target-scoped evidence
Complete author read-only command block. Replace all resource placeholders and use the intended Azure/kubectl context; no node mutation or capacity diagnosis is claimed.
Example input
```bash
az account show --query '{subscription:id,tenant:tenantId}' -o json
az aks show --subscription <subscription> -g <resource-group> -n <cluster> \
--query '{id:id,fqdn:fqdn,location:location}' -o json
kubectl config current-context
kubectl describe pod -n <namespace> <pod>
kubectl get node <gpu-node> \
-o jsonpath='{.metadata.name}{"\n"}{.spec.taints}{"\n"}{.status.capacity}{"\n"}{.status.allocatable}{"\n"}'
az aks nodepool show --subscription <subscription> -g <resource-group> \
--cluster-name <cluster> -n <pool> \
--query '{id:id,vmSize:vmSize,gpuProfile:gpuProfile,enableAutoScaling:enableAutoScaling,provisioningState:provisioningState}' -o json
```License
Complete selected author section retained verbatim; surrounding chapter omitted. Source placeholders and limitations preserved. No cloud action, source installer, account sign-in, model, command or API execution. This is an author recipe/illustration, not a reproduced site result.
MIT License
Copyright 2025 (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWAREGPU OOM: correlate pod, logs, node and profile
Complete author evidence commands. Container-memory and CUDA-device hypotheses must remain distinct; this recipe returns no real telemetry here.
Example input
```bash
kubectl get pod -n <namespace> <pod> -o json
kubectl logs -n <namespace> <pod> -c <container> --previous --timestamps
kubectl get node <gpu-node> -o json
az aks nodepool show --subscription <subscription> -g <resource-group> \
--cluster-name <cluster> -n <pool> --query gpuProfile -o json
```License
Complete selected author section retained verbatim; surrounding chapter omitted. Source placeholders and limitations preserved. No cloud action, source installer, account sign-in, model, command or API execution. This is an author recipe/illustration, not a reproduced site result.
MIT License
Copyright 2025 (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWAREHow to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/microsoft/azure-skills --skill aks-gpu-inferenceBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
AKS GPU Inference Day-2
Quick Reference
| Property | Value |
|---|---|
| Best for | AKS GPU/KAITO failures |
| Evidence | Bound target, events/status, gpuProfile, DCGM |
When to Use This Skill
Use for scheduling, VRAM/OOM, KAITO readiness, and workload scaling. Route exclusions as described above.
MCP Tools
Use a fitting host-advertised Azure read or the references' read-only queries.
Host Capability Gate
Before executing any referenced pipeline, verify that the host authorizes the required shell and kubectl/az commands against the bound target. File-backed collection also requires approved artifact storage and access to any bundled files it uses. A governed Azure CLI tool alone does not establish these capabilities. Equivalent host reads may replace commands only where their advertised schemas provide the required evidence.
If execution is unavailable or prohibited, say so and analyze supplied or redacted events, status, logs, and metrics, or give the operator a scoped collection plan. State which reads did not run and leave conclusions requiring missing evidence unconfirmed. Never route kubectl through Azure MCP or bypass host policy. The authorization requirements below still apply.
Workflow
- Bind subscription, cluster, kube context, pool, and affected resource.
- Capture exact events/status and observed
gpuProfile; state which reads ran. - Route to scheduling,
observability, KAITO, or scaling.
- Separate container/host OOM from device-allocation failures. For
OOMKilled,
correlate container memory limits/usage and node conditions. For Managed+Install device-memory evidence, use the exporter on port 19400 and incident-window FB_USED/FB_FREE; sizing tables do not establish a cause.
- For KAITO not-ready, warn that Workspace deletion leaves its GPU pools;
cleanup is separate and needs explicit authorization.
- Report evidence, confidence, missing evidence, and owner handoff.
Require explicit authorization before scaling, cordon/drain, deletion, add-on enablement, or monitoring mutation.
Error Handling
| Condition | Response |
|---|---|
| Missing/contradictory evidence | Mark unconfirmed; request the owner/read |
| Unknown profile | Preserve evidence; do not prescribe stack repair |
| Mutation required | Propose separately and wait for authorization |
