debug-openshell-cluster

Debug OpenShell Gateway Deployment

Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, the experimental VM driver, or an operator-managed out-of-tree compute driver.

Use openshell first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver: docker, podman, kubectl/helm, or VM driver logs.

Overview

The target deployment flow is:

1. Operator starts or deploys the gateway with system packages, systemd, or Helm. The CLI does not start, stop, or destroy gateway services.
2. Operator configures the compute driver.
3. Operator provides the CLI and supervisor authentication material required by the deployment mode: edge or OIDC user auth, optional CLI mTLS, and gateway-minted sandbox JWTs.
4. The CLI registers a reachable gateway endpoint with openshell gateway add.
5. The gateway creates sandboxes through the selected compute driver.

The openshell-gateway composition crate explicitly installs its compiled Docker, Podman, Kubernetes, and VM registrations at startup; openshell-server does not link compute-driver crates. Custom gateway binaries may include a subset of those registrations. With no configured driver, the gateway probes only installed registrations in priority order (Kubernetes, Podman, then Docker); VM has no probe and remains opt-in. Confirm the binary's registered drivers when auto-detection reports that no suitable driver is available. If configuration selects a driver that was not compiled in, the gateway treats the name as an external driver and reports a missing socket_path unless an endpoint is configured.

On Windows, custom binaries can include MXC independently. Registrations for Docker, Podman, Kubernetes, and VM are rejection stubs when included; they do not enable those runtimes on Windows.

See the compute driver reference for selective-build options and external-driver configuration.

For local evaluation only, TLS may be disabled and the gateway can be reached through http://127.0.0.1:<port>.

Prerequisites

• The openshell CLI must be available for endpoint checks.
• Know the active gateway name and endpoint, or be able to inspect local gateway metadata.
• Know the compute platform: Docker, Podman, Kubernetes, VM, or an out-of-tree driver.
• For Kubernetes: kubectl must target the cluster that hosts OpenShell and Helm version 3 or later must be available.
• For Docker or Podman: the runtime socket must be reachable from the gateway host.

Use openshell --help and nested --help output as the authority for the installed CLI version. Use the published installation guide, compute-driver reference, gateway configuration reference, and Kubernetes setup guide as the authority for deployment and configuration behavior.

Workflow

Run diagnostics in order and stop once the root cause is clear.

Step 1: Check CLI Reachability

openshell gateway list --output json
openshell gateway info
openshell status

For a one-off endpoint check that bypasses stored gateway selection and metadata:

openshell --gateway-endpoint <url> status

Common findings:

• No active gateway: register one with openshell gateway add <endpoint>.
• gateway has no user authentication configured: configure OIDC, mTLS user or operator authentication, or gateway JWTs using the published gateway configuration reference. For trusted local development, explicitly set [openshell.gateway.auth] allow_unauthenticated_users = true; the gateway otherwise refuses to bind any ports.
• Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active.
• TLS/certificate errors: the endpoint scheme or trust chain is wrong, a CLI mTLS bundle does not match the gateway CA, a supervisor is missing the gateway CA, or TLS termination does not match the gateway listener. Workloads and supervisors should not contain a user TLS client certificate or private key.
• Fresh snap installations select gateway-mode=user, but the user gateway is globally disabled by default. install.sh enables snap.openshell.user-gateway.service only for its target user; direct snap installations must select one user and run systemctl --user enable --now snap.openshell.user-gateway.service from that user's session. Do not enable it for multiple concurrently logged-in users: every instance binds to 127.0.0.1:17670, so later instances fail with an address-in-use error. Older snap revisions could leave the unit globally enabled; remove that state with sudo systemctl disable --global snap.openshell.user-gateway.service, then disable and stop the service from every other affected user's session. The global command does not stop running user services. Setting gateway-mode=user disables the system gateway without changing any user's service. To change the selected user, disable and stop the current user's unit before enabling the new user's unit. Upgraded installations retain their system-owned state through openshell.system-gateway. Existing HTTPS registrations continue to work after the service rename. Follow the published snap installation steps to enroll a new user, replace an old HTTP registration, or change modes safely through the disable maintenance state. If gateway-mode=disable, no local snap gateway is expected to run.
• Unauthenticated from an edge or OIDC gateway: refresh stored credentials with openshell gateway login [name], then retry. Use gateway logout only when intentionally clearing local credentials.
• Operator credential retrieval requires direct gateway mTLS, [openshell.gateway.mtls_auth] operator_enabled = true, and a verified certificate with exact OU=operator. OIDC admin roles and forwarded certificate headers do not grant access. Do not combine the operator certificate with an Authorization header. Enabling this role grants gateway-wide administration; never repair a denial by issuing operator certificates to ordinary users. See the published gateway authentication and provider references.
• A direct development endpoint with a private or self-signed certificate can be isolated with --gateway-endpoint <url> --gateway-insecure; do not persist or recommend insecure verification for shared gateways.

Step 2: Identify the Compute Platform

Use gateway metadata, deployment values, or the user's setup notes to identify the driver.

Step 3: Check Gateway Startup Dependencies

Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready.

The gateway container uses a Distroless Debian runtime. For OS-library vulnerability findings, check the deployed image digest and package version; deploy a rebuilt gateway image with the patched base. Updating the gateway binary alone does not update the libraries supplied by its container image.

For stable releases, inspect the version tag or deployed digest. Promotion preserves the qualified prerelease's commit-SHA image tags; those tags do not identify the stable rebuild. See image contents verification.

For resource-admission failures, distinguish disabled caller driver config from missing resource approval. Helm defaults server.drivers.kubernetes.allowDriverConfig to false and resourceAdmission.enabled to true. Existing PVCs, RuntimeClasses, and PriorityClasses need matching administrator-owned labels; namespace membership and read-only access do not grant approval. GPU devices and operator-selected image-pull Secrets do not need admission labels. In managed mode, inspect the configured source image-pull Secret in the gateway namespace and the generation copies (openshell.ai/component=image-pull) in the workspace namespace. Legacy workloads without admission provenance need recreation. Do not automatically label control-plane resources or disable enforcement as a repair.

For Kubernetes mount-path conflicts, move caller PVC mounts away from /.openshell and /run to a data path such as /data or below /sandbox. These driver-managed paths remain reserved even for approved resources and sub_path mounts; changing approval labels does not resolve a path conflict.

For out-of-tree compute drivers, also check that their versioned admission-policy acknowledgement matches the gateway's policy. Configure standalone driver policy through its administrator-owned --admission-config-json option.

For out-of-tree compute drivers, confirm the selected driver name and socket agree across CLI flags or gateway.toml, and that the operator-owned driver is running before the gateway starts:

rg -n '^version|compute_driver|socket_path|guest_tls_' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200

Gateway configuration requires [openshell] version = 2, a singular compute_driver selector, and driver-owned settings under [openshell.drivers.<name>]. The gateway rejects legacy compute_drivers and --drivers selectors rather than silently migrating them. One valid, nonempty OPENSHELL_DRIVERS value remains a deprecated environment-only alias when the canonical selector is absent; the gateway selects that driver with a warning. Empty, invalid, comma-delimited, or conflicting values fail startup. The WebSocket tunnel for edge-proxy CLI access is off unless enable_websocket_tunnel = true (server.enableWebsocketTunnel in Helm). Homebrew and RPM package startup migrates only exact package-generated v1 defaults. If an upgraded package still reports an unsupported version, inspect the active prefix or ~/.config/openshell/gateway.toml; an edited v1 file must follow the published schema-v2 migration steps and must not be overwritten. The supervisor gateway CA is the exception to driver ownership: configure guest_tls_ca under [openshell.gateway], and the gateway injects it only into the selected local driver. Remove the retired guest_tls_cert and guest_tls_key fields. TLS-enabled Docker, Podman, and VM drivers fail startup when neither that CA nor the package-managed local CA is available. Kubernetes projects only the gateway CA from a Secret into supervisor Pods; supervisors authenticate gateway RPCs with sandbox bearer tokens.

Custom names use [openshell.drivers.<name>].socket_path. A launch-time --compute-driver-socket override may also use docker, podman, kubernetes, or vm; the endpoint then takes precedence over built-in construction. First-party standalone drivers require the socket parent directory to be owned by the driver's effective UID, force its mode to 0700, create the socket with mode 0600, and accept only peers with that same UID. Check the parent and socket separately with stat; a gateway running under a different UID cannot connect even when filesystem permissions or group membership would otherwise allow it. Operator-supplied drivers must provide equivalent access control appropriate to their implementation. Check gateway logs for connection errors, GetCapabilities failures, missing peer metadata, protocol-major mismatch, unmet required capabilities, or an unexpected advertised driver name. openshell gateway info reports successful startup negotiations. The advertised name is diagnostic metadata; negotiated features control optional behavior. The gateway does not create or supervise operator-supplied driver processes or sockets.

For the Kubernetes Secrets credential driver, every provider credential lives in the configured namespace, in every workspace mode. A PermissionDenied error naming another namespace means the provider's credential handle points outside the configured namespace; recreate the provider. An unknown field startup error for [openshell.credential_drivers.kubernetes-secrets] means the table sets a key the driver does not accept. Confirm the gateway can reach the credential namespace:

kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -A3 '^\[openshell\.credential_drivers\.kubernetes-secrets\]'
kubectl auth can-i get secrets -n <credential-namespace> --as system:serviceaccount:openshell:openshell

For a configured Vault credential driver, inspect its endpoint and trust bundle before debugging provider resolution. Non-loopback addresses must use HTTPS, and the driver never follows redirects. A private CA bundle augments platform roots but does not disable hostname verification. With Helm, server.credentialDrivers.vault.caConfigMapName names a ConfigMap whose ca.crt key is mounted at /etc/openshell-tls/vault-ca/ca.crt:

kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -A10 '^\[openshell\.credential_drivers\.vault\]'
kubectl -n openshell get pod -l app.kubernetes.io/name=openshell -o jsonpath='{range .items[0].spec.containers[0].volumeMounts[*]}{.name}{" "}{.mountPath}{"\n"}{end}' | grep vault-ca
kubectl -n openshell get configmap <vault-ca-configmap> -o jsonpath='{.data.ca\.crt}' | openssl x509 -noout -subject -issuer -dates
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200

An HTTP service DNS address fails configuration validation. UnknownIssuer or an invalid CA error means the ConfigMap is missing, the ca.crt key is wrong, or the bundle does not contain the Vault server's issuer. A hostname mismatch means the HTTPS address host is absent from the server certificate SANs; keep verification enabled and issue a certificate for the service DNS name.

For configured gateway interceptors, inspect [[openshell.gateway.interceptors]], their Unix or network endpoints, and gateway startup logs:

rg -n 'interceptors|provider_profile_sources|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|binding_policy|failure_policy|gateway_jwt' /etc/openshell/gateway.toml
stat /run/openshell/interceptors/<name>.sock
journalctl -u <interceptor-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200

The gateway calls each interceptor's Describe RPC and validates its manifest at startup. Check for missing peer metadata, protocol-major mismatch, unmet required capabilities, unreachable endpoints, invalid RPC/phase bindings, strict allowlist or exact mismatches, and post_commit bindings that resolve to fail_closed. If gateway JWT signing is enabled, authenticated network interceptors require HTTPS and a valid bearer token; check the private CA path, endpoint hostname, expected audience, issuer, kid, and interceptor logs for token rejection. allow_insecure_transport = true explicitly preserves unauthenticated plaintext behavior. If provider_profile_sources names an interceptor, that interceptor must advertise provider-profile capability and return a valid, duplicate-free catalog. A selected interceptor-only source is authoritative; include a user source explicitly when composition is intended. The builtin source type was removed: a config that still names it is rejected at startup.

If the deployment uses supervisor middleware, follow the supervisor middleware troubleshooting reference for startup, authentication, policy validation, and HTTP or WebSocket failures.

For network policy validation failures, first distinguish a gateway mutation rejection from a supervisor runtime rejection. Direct policy updates, incremental merges and approvals, provider attachments, and provider-profile fanout are validated against the complete effective policy before persistence when the gateway knows the affected sandbox scope. A FAILED_PRECONDITION ambiguity response means no invalid revision or partial fanout was stored. Supervisor validation remains defense in depth for startup, races, and policy sources outside those mutation paths.

Runtime rejection behavior is configured only in gateway.toml:

[openshell.gateway]
policy_validation_failure_mode = "fail_closed"

The default fail_closed mode deactivates the previous generation, closes pinned relays, and quarantines new egress until a valid generation loads. retain_last_valid explicitly keeps the previous valid policy active; without one it still fails closed. Restart the gateway after changing this field. Inspect sandbox OCSF configuration and finding events for the validation rationale, configured and effective modes, active generation, and the explicit previous_policy_active state.

The published supervisor image uses a shell-free distroless Debian 13 base. For custom builds using SUPERVISOR_BASE_IMAGE, check the selected base's GNU runtime libraries, CA certificates, and inherited user and working directory. Use container logs, engine inspection and the configured exec health probe for diagnostics; exec ... sh, package installation and in-container shell scripts are unavailable. Workload shells belong to the separate sandbox image. Preserve the driver-selected UID and writable runtime/log mounts when reproducing a supervisor startup failure.

A ConfigurationInvalid readiness condition means startup admission rejected the image/effective policy or provider configuration. The supervisor remains alive while the workload stays unstarted. Inspect openshell sandbox get and repair the desired configuration with a complete policy replacement or provider change; do not treat a healthy container as proof that the workload is ready. If the 300-second provisioning repair window expires, the gateway records ProvisioningTimedOut and stops workload and supervisor compute. Inspect provisioning in sandbox JSON and TUI NOTES to distinguish cleanup pending from complete. Repairing configuration after expiry does not restart compute: wait for cleanup, then explicitly use sandbox start. Repeated rejected reports do not refresh the deadline, and the CLI wait timeout does not control it. See policy validation and repair. The isolated supervisor requests image-policy discovery through the authenticated sandbox boundary before admission. The workload boundary can remain alive without launching the workload while configuration is repaired. An unavailable boundary fails discovery within its control-request deadline. Permanent gateway errors and exhausted transient retries terminate startup; inspect those errors as connectivity, authorization, or lifecycle failures.

Step 4: Check Docker-Backed Gateways

The sandbox container's log holds the sandbox runtime's warnings and the main process's stdout and stderr when it runs without a TTY. The supervisor container's log holds supervisor diagnostics and OCSF shorthand. With ocsf_json_enabled=true, it also contains compact OCSF-JSON records for log collectors. Read these through docker logs <supervisor-container>; docker cp does not expose the live /var/log tmpfs, and workload exec accesses a separate filesystem. See the published OCSF JSON export guide for the marker format and delivery limits.

docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-ghcr.io/nvidia/openshell/sandbox:latest}" --version
openshell status

For Docker GPU failures, check CDI support and NVIDIA CDI discovery separately:

docker info --format '{{json .CDISpecDirs}}'
docker info --format '{{json .DiscoveredDevices}}'
for dir in /etc/cdi /var/run/cdi; do
  if [ -d "$dir" ]; then
    find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print
  else
    echo "$dir missing"
  fi
done
systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50
journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100

When the NVIDIA Container Toolkit CDI refresh units are not enabled or no NVIDIA CDI spec has been generated, enable them and trigger a refresh:

sudo systemctl enable --now nvidia-cdi-refresh.path
sudo systemctl enable --now nvidia-cdi-refresh.service
sudo systemctl restart nvidia-cdi-refresh.service
docker info --format '{{json .DiscoveredDevices}}'

Common findings:

• Docker daemon unavailable: start Docker Desktop or Docker Engine.
• Gateway process stopped: inspect exit status and logs.
• Sandbox image missing or pull denied: verify image reference and registry credentials.
• Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI USER and matching /etc/passwd and /etc/group entries, or explicitly set both process identity fields in policy. Numeric workload identities 1 through 4294967294 are accepted; root, the invalid identity sentinel, and missing identities are rejected.
• Sandbox fails before readiness with an OCI workspace validation error: inspect the image's WorkingDir using the immutable image ID reported by the gateway. Empty, /, and explicit /sandbox use the managed /sandbox compatibility workspace. Any other workdir must be an absolute normalized directory with no symlink components; the final policy UID, primary GID, and supplementary groups must pass the kernel's effective traverse/write checks, including POSIX ACL and LSM decisions. OpenShell does not create, chown, or chmod a non-default image workdir.
• Docker and Podman also reject an image VOLUME or driver mount that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the VOLUME below the workspace or remove the declaration.
• Custom Docker and Podman workspace changes live in the workload container's writable layer, not a managed named volume. Stop/start of the same container retains them; removing or replacing that container does not. Export files you need before deleting the sandbox.
• A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
• Local Docker gateway setup cannot copy openshell-sandbox after exporting a supervisor image: the sandbox runtime and supervisor are separate artifacts. The runtime image must provide /openshell-sandbox; the supervisor image provides /openshell-supervisor.
• Docker driver cannot initialize because it cannot find openshell-sandbox: verify the sibling binary next to openshell-gateway, or that the configured sandbox_runtime_image contains /openshell-sandbox.
• Sandbox never registers: check gateway logs and the supervisor's gateway endpoint.
• SSH host-key startup errors: use matching gateway, compute-driver, and supervisor releases. The gateway retains each sandbox's SSH key in its configured credential store and sends it only through the supervisor bootstrap bundle. Check credential-driver availability and compare the public host_key_fingerprint from sandbox JSON output; never print the bootstrap bundle or private key. A missing stored key for a sandbox with a fingerprint is an error, not permission to replace its identity. See the sandbox SSH identity documentation.
• Calls to an external tool server fail while the sandbox is Ready: inspect Tool server connections in openshell sandbox get <name>. For configured MCP-over-HTTP endpoints, JSON output exposes each address together with last_result and last_reported_at in endpoint_statuses. Select the endpoint by host, path, and ports, then check the reported failure boundary. last_reported_at records gateway acceptance time and can advance when retained evidence is accepted after a reset. Results do not expire or prove current availability; HttpResponseReceived can still contain a tool error. For token grant failures on any HTTP protocol, including failures before endpoint selection, inspect last_token_grant_failure in JSON output. It contains the bounded destination and cause, survives unrelated successful traffic, and clears on matching grant success or a policy, provider, or supervisor reset. If several paths share a host and port, other failures before the path is known, such as TLS failures, remain only in logs. A CredentialUnavailable result from a failed dynamic token grant also carries last_failure with a closed-set code, stage, recovery_action, the provider and credential names, and the request_id that the workload received in its x-openshell-request-id response header. Search openshell logs <name> for that ID: the supervisor's Token grant failed line carries the code and stage for every endpoint, and the gateway's dynamic token grant failed line appears when the gateway hop failed. Verify the actual operation when current tool availability matters.
• On Docker Desktop, repeated Policy fetch failed after 5 attempts messages

can mean the supervisor's selected network mode cannot reach the gateway. Check the gateway's Selected Docker supervisor networking log. The default supervisor_network_mode = "host" uses host networking; auto also selects host. Host mode requires Desktop host networking enabled; set bridge explicitly to use the bridge network. Bridge mode uses host.docker.internal for the Desktop host and requires a bridge-reachable listener on native Linux. Explicit grpc_endpoint values must be reachable in the selected mode, including any remapped published port, with matching TLS certificate names. The local gateway:docker task discovers its endpoint unless OPENSHELL_GRPC_ENDPOINT supplies an override. The OTLP collector endpoint must also be reachable from the supervisor's selected network. Mode selection does not bypass Landlock/seccomp qualification. Bridge mode and host.docker.internal gateway endpoints discover Docker's host address once at driver startup. If discovery fails, check that the configured supervisor image can run --help and the Docker API permits container creation, startup, archive download, and removal. host_gateway_ip overrides discovery if needed. See the published Docker configuration for the current networking options.

• Sandbox runtime image exits before printing openshell-sandbox --version: verify the configured image contains a static executable at /openshell-sandbox.
• A sandbox with explicit protocol: tcp endpoints fails before workload readiness: inspect supervisor and backend logs for policy rejection, network-mediation initialization, protected-channel setup, or listener failures. A backend that cannot enforce the admitted policy must reject it before starting workload code.
• Supervisor runtime validation fails: verify supervisor_image contains an /openshell-supervisor executable from the same release as the sandbox runtime, and that the dynamic loader and shared libraries it links against are available inside that image. docker run --rm --network none --entrypoint /openshell-supervisor <supervisor_image> --version should print that release; a no such file or directory error for a binary that exists means the loader or a library is missing. The supervisor runs from its own image and does not need to be static; only /openshell-sandbox must be.
• New Docker and Podman sandboxes dial the supervisor over a separate private

Unix socket volume; the workload keeps network=none. Check both containers' mounts and socket-directory ownership if attachment times out. Existing containers retain their saved legacy transport through restart. New VM generations dial host CID 2 over vsock; libkrun forwards to a protected host Unix socket, while QEMU uses a per-guest host port. Keep supervisor and runtime binaries compatible when creating new generations.

• Sandbox startup reports an isolation control contract failure: inspect the supervisor and sandbox-runtime image versions and update them together. An older runtime or incompatible isolation capability set fails before workload startup. See the published Sandbox Runtimes documentation for trusted image override precedence.
• The sandbox fails its enforcement probe: inspect the sandbox log for the exact nested seccomp user-notification, task-memory, Landlock, loopback DNS, or socket-injection check that failed. A runtime may return ENOSYS for process_vm_readv and process_vm_writev while satisfying the production parent-to-workload-child task-memory probe through /proc/<pid>/mem; only failure of both backends is fatal. Do not add capabilities or switch to an unconfined seccomp profile; use a runtime whose default profile permits the unprivileged probe.
• A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify .DiscoveredDevices contains entries such as nvidia.com/gpu=all, verify /etc/cdi or /var/run/cdi contains a generated NVIDIA spec, and check that nvidia-cdi-refresh.service and nvidia-cdi-refresh.path from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so inactive (dead) can be normal after a successful run; use systemctl status and journalctl to distinguish success from a skipped or failed refresh. Restart nvidia-cdi-refresh.service to regenerate missing or stale CDI specs, then restart or reload Docker and re-check docker info.
• Docker or Podman rejects an explicit cdi_devices selector: use a CDI qualified name (<vendor>/<class>=<name>), not a host-device path. Syntax validation accepts other vendors; it does not establish that the runtime has the device or that its GPU workload is supported. Consult the published GPU selection documentation for request requirements.

Corporate upstream proxy

Docker corporate proxy settings are operator-owned fields under [openshell.drivers.docker]. Confirm the complete proxy table and inspect the companion supervisor command and logs:

grep -A20 '^\[openshell.drivers.docker\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
docker ps --filter label=openshell.ai/isolation-role=supervisor
docker inspect --format '{{json .Config.Cmd}} {{json .Mounts}}' <supervisor-container>
docker logs <supervisor-container> --tail=200 | grep -Ei 'upstream|connect|proxy|certificate'

proxy_ca_bundle names a gateway-host PEM file and requires https_proxy. The proxy URL may use http:// or https://; a plain HTTP proxy may still re-sign destination TLS. Missing, unreadable, empty, oversized, malformed, or certificate-free bundles fail closed. The Docker driver copies a validated bundle into its supervisor-only named volume and passes the fixed path /.openshell/supervisor/upstream-proxy-ca-bundle.pem. The gateway-host path must not appear in container arguments, mounts, workload environment, or template.driver_config.docker.

An HTTPS proxy certificate error usually means the bundle lacks the proxy listener issuer or its certificate does not match the proxy hostname. A TLS-intercepted destination error means the re-signing issuer is missing or the supervisor did not receive the bundle. Keep verification enabled and correct the operator bundle.

During a graceful gateway restart, Docker, Podman, and VM sandboxes with running intent should stop before the gateway exits and restart after it returns. Check for Stopped sandbox during gateway shutdown and Started sandbox during gateway startup in gateway logs. A sandbox explicitly stopped through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do not follow this local gateway lifecycle. Internal and external drivers follow the same rule: GetCapabilities.gateway_manages_lifecycle must be true for the gateway to run shutdown and startup sweeps.

The gateway also drains supervisor-session ownership cleanup before exiting. If shutdown reports Gateway supervisor session cleanup incomplete, inspect the associated persistence errors: a stopped supervisor's owner record may remain until its lease expires and temporarily block reconnection. Successful compute stop alone does not confirm that session cleanup finished.

For a sandbox with --restart-policy on-failure or always, inspect openshell sandbox get <name> for the policy, exit code, restart count, and next restart time. Starting can mean the gateway is waiting for backoff or for a replacement supervisor. Check gateway logs for stop or start errors if the deadline passes without a transition to Ready. Docker, Podman, Kubernetes, and VM native restart policies remain disabled; the gateway owns the replacement.

Step 5: Check Podman-Backed Gateways

podman info
podman ps --filter name=openshell
podman logs <container> --tail=200
openshell status

Common findings:

• Podman socket unavailable: start or expose the user socket.
• Rootless networking unavailable: inspect Podman network configuration.
• Sandbox image missing or pull denied: verify image reference and registry credentials.
• Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI USER and matching /etc/passwd and /etc/group entries, or explicitly set both process identity fields in policy. Numeric workload identities 1 through 4294967294 are accepted; root, the invalid identity sentinel, and missing identities are rejected.
• Supervisor cannot connect: check its gateway endpoint and gateway logs.
• Inspect both Podman containers for the sandbox: the sandbox isolation role

must have network mode none; the supervisor role owns the gateway session and egress. Both run non-root with all capabilities dropped. Check the private channel volume and shared user-namespace mapping if authentication fails.

• If a sandbox fails before readiness, inspect its unprivileged enforcement

probe and the companion supervisor's private health check. Do not add capabilities, attach a workload network, or disable the runtime seccomp profile. There is no sandbox nftables or nested-network setup to repair.

• Gateway fails to start with an additional_ca_bundle error: gateway startup

validates the configured single bundle. The file must exist on the gateway host, be a regular non-symlink file, and contain at least one PEM certificate.

• Inspected HTTPS to a private-CA destination fails (for example

curl: (52) Empty reply from server), or the supervisor exits at startup, with a global additional_ca_bundle configured: the supervisor container needs the staged CA, not just the workload. Check podman inspect <sandbox>-supervisor --format '{{json .Mounts}}' for a read-only bind of the bundle at /etc/openshell/tls/additional/ca-bundle.pem, and --format '{{json .Config.Cmd}}' for --additional-ca-bundle with that path. The sandbox role container must not have the mount.

• On Linux, verify that the host-networked Podman supervisor can reach the

gateway's primary loopback endpoint. On macOS, verify Podman Machine's host-loopback forwarding or configure an explicit grpc_endpoint.

• If host.openshell.internal does not resolve inside a workload, verify its

/etc/resolv.conf contains nameserver 127.0.0.53. The Podman driver mounts that file from a per-sandbox secret and supplies the alias destination to the supervisor. Check host_gateway_ip only when the platform default (127.0.0.1 on native Linux or 192.168.127.254 on macOS Podman Machine) does not reach the gateway host.

When userns is configured (e.g. userns = "auto" or userns = "keep-id"):

• Sandbox runtime delivery uses bind-mount fallback instead of image volumes because

overlay mounts do not support idmapped mounts. The supervisor binary is extracted from the sandbox runtime image and cached at $XDG_DATA_HOME/openshell/podman-supervisor/ (typically ~/.local/share/openshell/podman-supervisor/).

• Stale cache: if the sandbox runtime image is updated but the cached binary is not

refreshed, sandbox creation may fail with an ELF validation error or version mismatch. Remove the cache directory and retry.

• auto mode requires subuid/subgid ranges for the current user in

/etc/subuid and /etc/subgid. If missing, Podman returns a user-namespace mapping error at container creation.

• private mode requires explicit uidmap and gidmap arrays in the TOML

config. Without both, the gateway rejects the config at startup. Rootless Podman uses intermediate IDs (e.g. uidmap = ["0:0:1", "1:1:65535"]); rootful Podman uses absolute host IDs (e.g. uidmap = ["0:1000:1", "1:100000:65536"]).

• nomap (without hyphen) is accepted as input but canonicalized to no-map

for Podman's API.

• A workload remains in stopping until Podman resorts to SIGKILL: inspect

supervisor logs for failed to signal entrypoint process group. The supervisor must retain CAP_KILL so its root process can forward SIGTERM to a workload that runs as the sandbox user.

Step 6: Check Kubernetes Helm Gateways

helm -n openshell status openshell
helm -n openshell get values openshell
kubectl -n openshell get deployment,statefulset,pod,svc,pvc
GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)"
kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" -c openshell-gateway --tail=200
kubectl -n openshell rollout status "${GATEWAY_DEPLOYMENT}"

Use the log and rollout commands for the gateway resource kind that exists in the release. Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures.

When checking a Helm values migration, compare the rendered gateway.toml with the intended gatewayConfig tables. Explicit resource-admission settings and the Kubernetes Secrets credential namespace take precedence over deprecated aliases. Confirm the credential driver's namespace matches its Role and RoleBinding. An unset workload image_pull_policy uses Kubernetes defaults; the global Helm pull policy applies to runtime and supervisor images.

When multiple gateways share one service base domain, confirm each release sets server.serviceHostnameFormat: hashed and a server.name unique among those gateways — the installation name is the hash discriminator for exposed-service hostnames, so a duplicate name silently routes two gateways' services to the same hostnames and a rename invalidates every previously issued URL. Check the rendered pair with:

kubectl -n <gateway-namespace> get cm <release>-config -o jsonpath='{.data.gateway\.toml}' | rg -n '^name|service_hostname_format'

The chart mounts the gateway.toml ConfigMap key directly at /etc/openshell/gateway.toml as a read-only subPath file. This avoids the atomic-writer symlink exposed by a ConfigMap directory mount because the gateway rejects symlinked configuration. A checksum pod-template annotation rolls the workload when the ConfigMap changes. If config preflight reports a symlink or nonregular path, inspect the rendered mount and confirm the workload rolled to the current chart revision:

kubectl -n openshell get deployment,statefulset -o yaml | rg -n 'gateway-config|mountPath|subPath|checksum/gateway-config'
kubectl -n openshell rollout status <deployment-or-statefulset>/openshell
kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200

server.telemetryEnabled renders OPENSHELL_TELEMETRY_ENABLED on the gateway pod, and the gateway propagates the effective value to sandbox supervisors.

Additional destination CA ConfigMap

When supervisor.network.additionalCaConfigMapName is set, Helm must mount the source operator ConfigMap's ca.crt key into the gateway at /etc/openshell-tls/additional-ca-source/ca.crt. The rendered gateway.toml must contain the global (not Kubernetes-driver-local) setting:

[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell-tls/additional-ca-source/ca.crt"

Inspect the source, render, and live gateway mount before inspecting sandbox pods:

helm -n openshell get values openshell | grep -A3 additionalCaConfigMapName
kubectl -n openshell get configmap <source-configmap> -o jsonpath='{.data.ca\.crt}' >/dev/null
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -A2 '^\[openshell.supervisor.network\]'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.volumes[?(@.name=="additional-ca-source")]}{"\n"}'

At startup, the gateway normalizes the source once and stages a content-addressed, read-only artifact below /var/lib/openshell-state/openshell/network-supervisor/. The gateway workload sets XDG_STATE_HOME=/var/lib/openshell-state and mounts the writable gateway-state volume there. Check that wiring before debugging the driver:

kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.containers[0].env[?(@.name=="XDG_STATE_HOME")]}{"\n"}'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.containers[0].volumeMounts[?(@.name=="gateway-state")]}{"\n"}'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.volumes[?(@.name=="gateway-state")]}{"\n"}'

At sandbox creation or restart, the Kubernetes driver creates or verifies an immutable, content-addressed ConfigMap in the target sandbox namespace. Its name binds the effective gateway identity and normalized trust digest. The managed ConfigMap must have openshell.ai/managed-by=openshell, the matching openshell.ai/gateway-id label, and an openshell.ai/additional-ca-generation annotation. Different trust generations intentionally coexist so running sandboxes retain their startup trust. The supervisor Pod mounts the ConfigMap read-only at /etc/openshell/tls/additional/ca-bundle.pem and receives --additional-ca-bundle for that path; the immutable flag plus the driver's per-creation digest and exact-byte validation close deletion/recreation races after the API check.

Kubernetes RBAC cannot restrict create by resourceNames or restrict get by a name prefix. When this feature is enabled, the chart grants the gateway service account ConfigMap get/create in the shared sandbox namespace. In managed/operator modes, the existing ClusterRoleBinding makes those permissions cluster-wide. It does not grant ConfigMap list, watch, patch, update, or delete. Prefer a dedicated gateway service account and namespace-scoped bindings when the read boundary matters. Check the objects and authorization:

kubectl -n <sandbox-namespace> get configmap \
  -l openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id> \
  --show-labels
kubectl auth can-i get configmaps -n <sandbox-namespace> \
  --as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i create configmaps -n <sandbox-namespace> \
  --as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i patch configmaps -n <sandbox-namespace> \
  --as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i delete configmaps -n <sandbox-namespace> \
  --as system:serviceaccount:openshell:<gateway-service-account>

A missing, mutable, unowned, wrong-generation, or wrong-data managed ConfigMap fails closed. Do not relabel or edit it blindly. Investigate the object owner; remove an invalid object only after confirming no sandbox uses it, then retry so the driver can recreate the exact generation. Diagnostics deliberately omit certificate contents; inspect the source object separately rather than copying PEM into logs or tickets.

Changing source ConfigMap contents does not reload the gateway or a running supervisor: restart the gateway to reread/normalize it, then recreate or restart affected sandboxes. Removing the Helm setting means newly created or recreated sandbox pods have no additional-CA volume/mount, but existing running pods retain startup trust until restarted. The driver intentionally has no ConfigMap list/delete permission; unused managed generations can remain. After all sandboxes are migrated, use an operator identity to verify the generation annotation and pod references, then explicitly delete unused ConfigMaps in each applicable namespace. Do not add list/delete RBAC as a cleanup workaround.

When no external credential driver is enabled, the Helm chart uses the gateway's default encrypted database credential storage. The chart creates a retained Kubernetes Secret for the shared KEK, injects it into gateway pods, and stores encrypted credential envelopes in the OpenShell database. For workload.kind=deployment or multi-replica gateways, confirm server.externalDbSecret points at a shared database. A render/install error mentioning multiple credential drivers means the gatewayConfig.openshell.gateway.credential_drivers list selected more than one external credential backend.

For HA or PostgreSQL-backed installs, also check the external database Secret referenced by server.externalDbSecret and the PostgreSQL workload when it is deployed in-cluster:

kubectl -n <namespace> get secret <external-db-secret> -o yaml
kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<postgres-workload>
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200

Gateways that use an external PostgreSQL database guard cross-object mutations with PostgreSQL advisory locks at three levels: one global key, one key per workspace, and one key per sandbox, each taken shared or exclusive. Sandbox operations on different sandboxes do not wait on each other's locks; provider and workspace-profile changes block guarded sandbox mutations in their workspace; gateway-global policy and settings changes block every guarded mutation. Guarded sandbox mutations are sandbox create, provider attach and detach, sandbox-scoped policy and settings changes, and supervisor configuration, policy status, and endpoint status reports. Lifecycle paths (start, stop, delete, restart, supervisor connection and process-exit state, driver-watch updates, and reconcile) take only a process-local shared global and exclusive sandbox lock, with no time limit and no mutation advisory lock (start, restart, and delete also take the SSH identity key below), so only a global change on the same replica blocks them directly. Provider, workspace-profile, and other replicas' global changes delay them only while another operation on the same sandbox waits behind the change, holding that sandbox's key. Provisioning-deadline expiry also takes its workspace key locally, so provider and workspace-profile changes on the same replica can delay it; openshell_server_provisioning_deadline_skips_total{overdue="true"} counts the checks that skipped an overdue sandbox. Each replica takes its advisory locks on a dedicated lock pool of PostgreSQL connections, one per guard, 4 by default (gateway config database_lock_max_connections), so at most that many guarded operations per replica hold or wait for PostgreSQL locks at once. openshell_server_mutation_lock_connections_in_use shows how many are checked out, against openshell_server_mutation_lock_connections_capacity. Its data pool (database_max_connections, 10 by default) is separate. Helm deployments set both keys under gatewayConfig.openshell.gateway. The gateway logs both sizes at startup in sizing Postgres connection pools. A request cancelled during its lock wait frees its slot while its backend keeps waiting, with any keys it took, until the 10-second deadline, so pg_locks can briefly show more lock sessions from one pod. A guard that waits on a PostgreSQL lock keeps its connection while it waits, so contention on one key can fill the pool and make unrelated guarded operations on that replica queue for a connection, within the same 10-second limit.

A lock wait longer than 10 seconds returns UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and the message "... timed out waiting for a concurrent mutation; retry the request", logs mutation lock acquisition timed out with scope, waited_ms, and detail fields, and increments openshell_server_mutation_lock_timeouts_total. The detail field says where the wait stopped:

• waiting for a local mutation lock: another operation on the same replica

holds a conflicting key.

• waiting for a mutation lock connection: every lock-pool connection of that

replica was in use, by waiters or by holders slowed by a compute driver, credential backend, middleware, or provider profile source call, or a local wait left less than a second to open one. pg_locks shows no row for the timed-out operation. Look for the pod's granted or waiting advisory-lock rows by client_addr, and find the holder they wait on.

• waiting for a PostgreSQL advisory lock, or PostgreSQL's own

canceling statement due to lock timeout: a conflicting key is held on another connection, usually by another replica or by an older gateway during an upgrade. pg_locks shows the holder and the waiters.

A starting gateway resets stale endpoint status one sandbox at a time before it binds its listeners. It retries lock timeouts there for the first 30 seconds of that scan, then logs tool server endpoint-status startup reconciliation gave up on a sandbox with the sandbox_id and exits with tool server endpoint-status startup reconciliation failed. A pod that restarts with that error during a rollout or scale-out is waiting behind a long holder: find it as above. When the scan finishes, its waits and timeouts are already in the lock metrics at the pod's first scrape, so read the counter and _count values on the new pod: rate() and increase() take that first scrape as their baseline and do not show them. A scan that fails exits before the metrics endpoint starts, so its timeouts appear in the gateway log only.

A lock connection that PostgreSQL does not open, although at least a second of the 10-second limit remained, is not a lock timeout. The request fails with INTERNAL and "could not open a mutation lock connection", and the gateway logs mutation lock acquisition failed and increments openshell_server_mutation_lock_errors_total instead of the timeout counter. A lock statement that fails for another reason, such as a PostgreSQL restart, is counted the same way. PostgreSQL refused the connection, ran out of connection slots, was starting up, or did not answer. Check the PostgreSQL logs for "too many clients already", "remaining connection slots are reserved", or "the database system is starting up", and compare SELECT count(*) FROM pg_stat_activity with SHOW max_connections. Each replica opens up to its data plus lock pool sizes, 14 with the defaults.

A gateway log line timed out returning PostgreSQL mutation lock connection; discarded connection means returning a lock session stalled for 5 seconds and the gateway dropped it; that slot stays busy for up to those 5 seconds. The warning does not identify the PostgreSQL backend. Healthy guards also hold idle advisory-lock sessions while validation and writes use separate data connections. Neither an idle duration nor a matching client_addr proves that a holder is orphaned, even when it blocks a timed-out request.

Guarded work has no time limit of its own. A holder slowed by a compute driver, credential backend, middleware, or provider profile source call keeps its locks until that call returns, and conflicting acquisitions on every replica time out after 10 seconds. When it releases a lock held longer than 10 seconds, its gateway logs mutation lock held longer than the lock wait timeout with scope and held_ms, and openshell_server_mutation_lock_hold_seconds records every hold by scope. Neither reports a hold that is still in progress. Before treating a long-held advisory lock as orphaned, check the gateway pod at its client_addr for these warnings and for slow dependency calls. A pod that logs them is still running slow guarded work, so fix that dependency or stop the pod instead of terminating its lock sessions:

kubectl -n openshell logs <gateway-pod> --since=1h \
  | grep 'mutation lock held longer than the lock wait timeout'

Before using SELECT pg_terminate_backend(<pid>), conclusively map that backend to its owning gateway process and confirm that process has stopped or can no longer write. If the owner is still running, stop it first and verify its exit; terminating or failing readiness alone is insufficient. Killing its lock session while it can still write removes exclusion from an active mutation. Recheck the backend PID and backend_start before terminating the confirmed orphan: pod IPs and PIDs can be reused, and a database proxy can hide several gateways behind one client_addr. If ownership cannot be established, investigate connectivity instead of choosing a backend by IP or idle state. Setting PostgreSQL tcp_keepalives_idle, tcp_keepalives_interval, and tcp_keepalives_count (for example 60, 10, and 6) bounds how long such sessions survive.

If mutations stall or time out while reads and health checks work, inspect advisory-lock holders and waiters. Do not print the database URI or Secret contents into logs:

SELECT l.pid, l.mode, l.granted, l.waitstart, l.classid, l.objid,
       a.client_addr, a.backend_start, a.state,
       now() - a.state_change AS in_state_for
FROM pg_locks l
JOIN pg_stat_activity a USING (pid)
WHERE l.locktype = 'advisory'
ORDER BY l.granted, l.waitstart;

Gateways hold these locks at session level outside any transaction, so a holder (granted = t) usually shows state = idle. It still holds the lock, and in_state_for approximates how long. client_addr is the holding gateway pod's IP, or the pooler's IP when a connection pooler sits in between.

The global key appears as classid = 1330660686 and objid = 1397247052 (key 0x4F50454E53484C4C). Older gateways during a rolling upgrade take that key exclusively for every guarded mutation, so mutations can queue behind them until the rollout finishes.

SSH host identity provisioning and cleanup, in sandbox create, start, restart, and delete, take one more fleet-wide key, classid = 1330860872 and objid = 1213158228 (key 0x4F535348484F5354), on a data connection instead of the lock pool, with a 10-second PostgreSQL lock_timeout. Sandbox create holds its mutation guard while it waits for that key. A wait that runs out returns UNAVAILABLE with "lock sandbox SSH identity failed", and neither openshell_server_mutation_lock_timeouts_total nor openshell_server_mutation_lock_errors_total counts it.

For multi-replica gateway installs, supervisor and client session traffic may be served by a non-owner gateway replica and relayed to the current supervisor owner over the internal PeerRelay RPC. Check the headless peer Service, projected peer ServiceAccount token volume, and TokenReview RBAC:

kubectl -n openshell get svc openshell-peer -o wide
kubectl -n openshell get endpoints openshell-peer
kubectl -n openshell get pod -l app.kubernetes.io/instance=openshell \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.spec.volumes[?(@.name=="gateway-peer-token")]}{"\n"}{.spec.volumes[?(@.name=="peer-client-tls")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_ENDPOINT")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_TLS_SERVER_NAME")]}{"\n"}{end}'
kubectl auth can-i create tokenreviews.authentication.k8s.io \
  --as=system:serviceaccount:openshell:openshell
kubectl auth can-i get pods -n openshell \
  --as=system:serviceaccount:openshell:openshell
kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" --tail=200 | grep -E 'gateway peer|PeerRelay|supervisor owner|owner relay'

Expected gateway startup logs include gateway peer ServiceAccount TokenReview authentication enabled. If peer relay calls fail with Unauthenticated, verify the gateway-peer-token projected volume has audience openshell-gateway-peer and that the receiving gateway can create TokenReviews. If they fail with PermissionDenied, verify the gateway ServiceAccount name, release namespace, pod UID, and Helm selector labels match the live gateway pods. Deployment-backed gateway pods should also publish OPENSHELL_PEER_ENDPOINT from their pod IP. The OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE name follows the existing token-file convention used by OPENSHELL_SANDBOX_TOKEN_FILE and OPENSHELL_K8S_SA_TOKEN_FILE.

For TLS-enabled gateways, peer clients verify the stable gateway Service DNS name and load the chart CA plus client identity from OPENSHELL_PEER_TLS_CA_FILE, OPENSHELL_PEER_TLS_CERT_FILE, and OPENSHELL_PEER_TLS_KEY_FILE. If peer calls fail during TLS negotiation, verify the peer-client-tls volume exists, those files are readable, and the server certificate includes the name in OPENSHELL_PEER_TLS_SERVER_NAME.

To check per-replica capacity on a multi-replica gateway, read each gateway pod's metrics and the autoscaler:

for pod in $(kubectl -n openshell get pod \
    -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
    -o jsonpath='{range .items[?(@.spec.containers[0].name=="openshell-gateway")]}{.metadata.name}{" "}{end}'); do
  echo "${pod}"
  kubectl get --raw "/api/v1/namespaces/openshell/pods/${pod}:9090/proxy/metrics" \
    | grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total|mutation_lock_(timeouts_total|errors_total|connections))'
done
kubectl -n openshell get hpa
kubectl -n openshell describe hpa openshell

The JSONPath filter keeps only pods whose first container is the gateway (openshell-gateway). It skips certificate hook Job pods, which older charts labeled like gateway pods. The metrics port is service.metricsPort (default 9090).

The API server proxy connects to each pod from the control plane. A NetworkPolicy that accepts the metrics port only from a monitoring namespace blocks it unless the policy also allows the control plane. In that case, read one pod at a time through kubectl port-forward, which reaches the pod through the kubelet and is not blocked by NetworkPolicy:

kubectl -n openshell port-forward pod/<gateway-pod> 9090:9090 >/dev/null &
pf_pid=$!
sleep 2
curl -s http://localhost:9090/metrics \
  | grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total|mutation_lock_(timeouts_total|errors_total|connections))'
kill "${pf_pid}"

Check required Helm deployment secrets:

kubectl -n openshell get secret \
  openshell-server-tls \
  openshell-server-client-ca \
  openshell-client-tls \
  openshell-jwt-keys

When server.tls.clientCaSecretName="", the chart intentionally omits client_ca_path and the tls-client-ca mount, even with built-in PKI or cert-manager. That is expected; do not treat a missing tls-client-ca pod mount as a defect (openshell-server-client-ca may still exist from PKI). User auth is OIDC or trusted proxy (server.auth.allowUnauthenticatedUsers=true); supervisor transport still uses openshell-client-tls.

In cert-manager installs, certManager.enabled=true makes cert-manager own TLS generation. The Helm chart should still render the openshell-certgen pre-install/pre-upgrade hook in JWT-only mode to create openshell-jwt-keys, even if pkiInitJob.enabled remains true. If the gateway pod is pending with MountVolume.SetUp failed for volume "sandbox-jwt" and openshell-jwt-keys is absent, inspect the rendered templates/certgen.yaml output and the hook Job logs; cert-manager creates TLS Secrets but does not create the sandbox JWT signing Secret.

If the gateway exits with failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pem, verify that openshell-jwt-keys contains signing.pem, public.pem, and kid, and that the gateway workload mounts the sandbox-jwt secret at /etc/openshell-jwt. The sandbox JWT mount is required even when local Helm values disable TLS.

If certManager.serverIssuerRef points the server certificate at an external Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted cert on an OpenShift Route with TLS passthrough — see openshiftRoute.enabled), the chart creates two server certificates: an internal one (chart CA, internal SANs) and an external one (from the configured issuer, external SANs only). The gateway uses SNI to present the right cert.

Check the external Certificate/CertificateRequest/Challenge resources directly when the external secret never becomes Ready:

kubectl -n openshell get certificate,certificaterequest,challenge
kubectl -n openshell describe certificate openshell-server-external
oc -n openshell get route

ACME issuers reject certificate requests that include internal-only names (*.svc.cluster.local, localhost, loopback IPs) and require the commonName to also be a SAN — the external Certificate only requests the hostnames in certManager.serverDnsNames, for exactly this reason.

If sandbox supervisors fail their TLS handshake to the gateway with UnknownCA after configuring serverIssuerRef, the most likely cause is server.grpcEndpoint set to the external hostname. This forces supervisors to connect via the external hostname, receiving the ACME cert (via SNI) which they cannot verify against the chart CA. Remove server.grpcEndpoint or set it to the internal service name so supervisors receive the internal cert:

helm -n openshell get values openshell | grep -E 'grpcEndpoint|clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'
# server.grpcEndpoint should be unset or point to internal service name

Less commonly, UnknownCA can occur if the gateway's client-verification CA is misconfigured. The default clientCaFromServerTlsSecret=true is correct for all configurations — the internal server certificate is always signed by the chart CA (the same CA that signs the client cert), so its ca.crt is the right trust anchor. Supervisor Pods project only ca.crt from the copied Secret; tls.crt and tls.key are reserved for user clients and must not be visible in a sandbox. Only override this if you intentionally mount a separate client CA via server.tls.clientCaSecretName. Verify the mounted client CA matches the CA that signed the client certificate:

kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}' | jq .
# Should show items filter for ca.crt from openshell-server-tls

If gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path is set, the gateway should still render [openshell.gateway.gateway_jwt] and mount the sandbox-jwt Secret. SPIRE is used by both the gateway and sandbox supervisors for dynamic provider token grants. The gateway pod must mount the spiffe-workload-api CSI volume and set OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET; supervisor Pods must receive the matching Workload API socket from the Kubernetes driver config. The gateway verifies supervisor JWT-SVIDs from JWT bundles fetched through this Workload API socket, not from the SPIRE OIDC discovery endpoint. Verify that SPIRE is installed, the CSI driver is available, and the Kubernetes driver config includes provider_spiffe_workload_api_socket_path:

helm -n openshell get values openshell | grep provider_spiffe_workload_api_socket_path
kubectl get pods -A | grep -E 'spire|spiffe'
kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{.items[*].spec.containers[*].env[?(@.name==\"OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET\")].value}{\"\n\"}"

Sandbox pods using provider token grants should have an openshell.ai/sandbox-id annotation, an openshell.ai/managed-by=openshell label, supervisor env vars OPENSHELL_K8S_SA_TOKEN_FILE and OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, plus both the projected openshell-sa-token volume and the spiffe-workload-api CSI volume.

If grpcRoute.backendTLSPolicy.enabled=true, the Gateway proxy validates the backend pod's TLS certificate against a CA in a ConfigMap. Check that the ConfigMap exists and contains the correct CA, that enableMtls is disabled, and that the BackendTLSPolicy resource is present:

kubectl -n openshell get backendtlspolicy
kubectl -n openshell get configmap openshell-backend-ca -o yaml
helm -n openshell get values openshell | grep -E 'backendTLSPolicy|enableMtls|failOnTimeout|timeoutSeconds|caCertificateConfigMapName'

If the ConfigMap is missing after a cert-manager install, the post-install certgen hook may have timed out waiting for cert-manager to issue the server certificate. Check the certgen Job logs:

kubectl -n openshell get jobs | grep certgen
kubectl -n openshell logs job/openshell-certgen-backend-ca

Increase pkiInitJob.timeoutSeconds and run helm upgrade to retry. If the Gateway proxy reports TLS error: Secret is not supplied by SDS or similar backend TLS errors, the ConfigMap CA likely does not match the server certificate CA — verify both are from the same issuer.

Check the image references currently used by the gateway deployment:

GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)"
kubectl -n openshell get "${GATEWAY_DEPLOYMENT}" -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'

The gateway, sandbox, and supervisor images should use the same release tag. A stale runtime image can make sandbox behavior lag behind gateway policy or protocol changes. Kubernetes sandboxes take the configured supervisor and runtime images each time they start, so a sandbox that kept running through an upgrade uses its previous images until its next stop and start.

For vulnerability reports, record the running image digest and scan that exact artifact. The gateway includes a pinned Distroless base; the supervisor includes Alpine packages updated at image build time. A dependency or base-image fix only reaches deployed containers after rebuilding, publishing, and redeploying the images. Compare findings against the SBOM for that digest, not just its mutable latest or dev tag. For gateway base refreshes, verify the installed libc package revision in each platform's SBOM; the binary's glibc compatibility floor is not its runtime version.

For plaintext local evaluation, confirm the chart has:

helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint'

Expected shape:

server:
  disableTls: true
  grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080

Check service exposure:

kubectl -n openshell get svc openshell -o wide
kubectl -n openshell get endpoints openshell

For local port-forward testing:

kubectl -n openshell port-forward service/openshell 8080:8080

Leave the port forward running. In another terminal, register the local endpoint if needed and verify it:

openshell gateway add http://127.0.0.1:8080 --local --name local-kubernetes
openshell status

If the gateway is healthy but sandbox creation fails:

kubectl -n openshell get pods
kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)"
kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" -c openshell-gateway --tail=200

Check the configured sandbox namespace:

helm -n openshell get values openshell | grep sandboxNamespace

Then inspect sandbox resources in that namespace.

For a split release, the gateway values should have workspaceResources.enabled=false, and the target namespace should contain a separate openshell-workspace release:

helm -n openshell get values openshell | grep -A2 workspaceResources
helm -n <sandbox-namespace> status openshell-workspace
kubectl -n <sandbox-namespace> get serviceaccount,role,rolebinding,networkpolicy \
  -l app.kubernetes.io/instance=openshell-workspace
kubectl auth can-i create sandboxes.agents.x-k8s.io \
  --namespace <sandbox-namespace> \
  --as system:serviceaccount:openshell:openshell

If the gateway cannot create or watch sandboxes, verify the workspace RoleBinding subject matches the gateway ServiceAccount name and namespace. If SSH relay connections fail, verify the workspace NetworkPolicy selects the gateway's actual app.kubernetes.io/name and app.kubernetes.io/instance labels.

Check the configured sandbox service account when TokenReview bootstrap or sandbox registration fails. Helm creates a dedicated sandbox service account by default and writes it to [openshell.drivers.kubernetes].service_account_name; the selected Kubernetes compute driver rejects projected tokens from other service accounts. For an external driver, inspect its logs and confirm it advertises supports_sandbox_authentication; the gateway delegates the opaque credential over the driver socket and never interprets Kubernetes settings. Drivers that advertise sandbox authentication must return the same non-empty runtime identity from sandbox creation and credential authentication. A gateway log reporting a compute runtime identity mismatch indicates stale or re-created runtime resources; compare the live resource UID with the sandbox that the gateway provisioned. Restart also rejects multiple Sandbox resources with the same sandbox label and requires the persisted namespace and CR UID to remain unchanged. A generation-bound session-token rejection usually means the supervisor is presenting credentials from a runtime that was replaced; inspect the persisted generation before retrying bootstrap.

The Kubernetes driver serializes lifecycle mutations and runtime reconciliation per sandbox within one driver instance. A busy sandbox is checked again on the next reconciliation pass. If restart still loses its supervisor, compare the Sandbox and Pod UIDs and identify which gateway or external driver process performed cleanup; the local mutation gate does not coordinate separate processes.

helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'

The Kubernetes driver creates a sandbox workload Pod and a separate, directly managed supervisor Pod. The cluster CNI must enforce ingress and egress Kubernetes NetworkPolicy in every sandbox namespace; the Kubernetes API cannot attest enforcement. Run sandboxes only in a trusted namespace where tenants cannot create Pods, copy OpenShell role labels, or read the bootstrap Secret.

The workload Pod runs /openshell-sandbox. It has no gateway credentials and no direct egress. One namespace-wide workload NetworkPolicy is created before the suspended Sandbox resource. It denies all workload egress and allows supervisor Pods to reach sandbox TLS listeners. The driver then creates a per-sandbox Service, split immutable bootstrap Secrets, and a gated supervisor Pod before releasing either Pod. The supervisor Pod runs /openshell-supervisor. Both Pods use the same resolved non-root identity, request no capabilities, drop ALL, disable privilege escalation, and use RuntimeDefault seccomp. The supervisor and sandbox talk over per-sandbox TLS with server-certificate verification plus bootstrap-token client authentication; the supervisor owns gateway policy, provider credentials, DNS, and mediated upstream connections.

Each sandbox records which side opens that connection in the Sandbox annotation openshell.ai/sandbox-runtime-control-connection, bound to openshell.ai/sandbox-runtime-control-generation. A missing marker or a different current runtime generation means supervisor-initiated: the os-boundary-<id> Service selects the workload Pod and the supervisor dials it. With runtime-initiated (used for new generations), a separate os-supervisor-<id> Service selects the supervisor Pod, the sandbox runtime dials it, and a per-sandbox NetworkPolicy os-control-<id> admits only that connection at both the workload (egress) and the supervisor (ingress). The Service publishes the supervisor before it is Ready, because readiness follows the runtime's attach. If the runtime logs cannot reach the supervisor, check that the CNI applied both directions of os-control-<id>. Stop/start upgrades legacy sandboxes to runtime-initiated control, as does gateway recovery that recreates both Pods. Existing running generations keep their current connection until replaced. Check the mode first when a sandbox never becomes ready, especially after changing images:

kubectl -n <sandbox-namespace> get sandbox <sandbox-name> \
  -o jsonpath='{.metadata.annotations.openshell\.ai/sandbox-runtime-control-connection}{"\n"}'
kubectl -n <sandbox-namespace> get service os-boundary-<sandbox-id> -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n <sandbox-namespace> get service os-supervisor-<sandbox-id> -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n <sandbox-namespace> get networkpolicy os-control-<sandbox-id> -o yaml

A runtime-initiated sandbox needs a gateway, supervisor image, and sandbox runtime image that all support the mode. An older supervisor or runtime fails to decode its launch descriptor, so the sandbox cannot start or restart. Old gateways can restart sandboxes by creating their own compatible pair; os-boundary-<id> always keeps its workload selector. New gateways also create it for new sandboxes, alongside the supervisor Service. Upgrade the images together within each generation. There is no legacy rollout flag. Adding the supervisor endpoint on restart requires create and get on Services in the sandbox namespace. For a Forbidden restart error, check the gateway ServiceAccount with kubectl auth can-i create services --namespace <sandbox-namespace> --as system:serviceaccount:<gateway-namespace>:<gateway-service-account>. Do not edit the annotation, Service selector, or os-control-<id> policy: reconciliation suspends a sandbox whose Service or control policy no longer matches its recorded mode.

The supervisor loads its startup configuration through the polling RPCs and opens ConnectSupervisor once the workload runs. When the gateway runs with config_delivery_mode = "push", it accepts the session with config_push_enabled and a bootstrap update. The supervisor applies the bootstrap, stops polling, and receives later updates on that stream. Every update must contain both the sandbox configuration and the provider environment it names. A reconnect to a gateway in poll mode resumes polling.

If live policy or provider changes do not reach a sandbox in push mode, inspect gateway and supervisor logs for bootstrap build failures, updates missing a configuration part, apply failures, or the two-minute bootstrap timeout. Do not add gateway credentials to the workload Pod; configuration delivery terminates in the separate supervisor.

Inspect all driver-managed resources when a Kubernetes sandbox remains Starting or loses readiness:

kubectl -n <sandbox-namespace> get sandbox,pod,service,secret -l openshell.ai/sandbox-id=<sandbox-id>
kubectl -n <sandbox-namespace> get networkpolicy openshell-sandbox-workloads -o yaml
kubectl -n <sandbox-namespace> describe pod -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=supervisor
kubectl -n <sandbox-namespace> logs pod/<supervisor-pod> --tail=200
kubectl -n <sandbox-namespace> get pod -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload -o yaml
kubectl -n <sandbox-namespace> get networkpolicy -l openshell.ai/sandbox-id=<sandbox-id> -o yaml

Creation and recovery fail closed. A missing Secret leaves both pods inert; a missing or unobserved workload fence must prevent the driver from releasing the Sandbox; and readiness requires both Agent Sandbox readiness and an Available supervisor Pod. Its exec readiness check succeeds only after the supervisor has attached, confirmed enforcement, started or resumed the workload, and registered the gateway access plane. Use both Pod logs for bootstrap errors. An EPERM during enforcement setup means the runtime blocked a required unprivileged seccomp, task-memory, or Landlock operation. Do not add capabilities, gateway egress, or credentials to the workload Pod as a workaround.

If a Sandbox remains in the releasing bootstrap phase, inspect the supervisor Pod first. The gateway keeps the workload running during this phase so the supervisor can release that sandbox's runtime-control relationship cleanly. Check the Pod's deletion timestamp, termination grace period, events, and finalizers, and verify that the gateway ServiceAccount can delete Pods in the sandbox namespace:

kubectl -n <sandbox-namespace> get pod \
  -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=supervisor \
  -o yaml
kubectl auth can-i delete pods \
  --namespace <sandbox-namespace> \
  --as system:serviceaccount:openshell:openshell

Do not suspend or delete the workload Pod manually. The driver advances to suspending only after runtime control has been released, and then suspends the workload.

If a Sandbox remains in the suspending bootstrap phase, verify that the gateway ServiceAccount can create and delete Secrets in the sandbox namespace. In operator mode, those permissions come only from the openshell-workspace chart installed in the namespace. Recovery deletes the recorded generation's bootstrap Secrets by name before clearing the suspension annotations:

for verb in create delete; do
  kubectl auth can-i "$verb" secrets \
    --namespace <sandbox-namespace> \
    --as system:serviceaccount:openshell:openshell
done

Any no result can strand recovery before workload Pod creation. Upgrade the OpenShell chart rather than treating missing workload Pods as Pod failures.

Shared supervisor high availability

supervisor.autoscaling.enabled=true gives replica ownership to a separate memory HPA named <fullname>-supervisor; supervisor.replicas is then ignored. The default target is 512Mi average memory per Pod, with two to four members. Check kubectl describe hpa <fullname>-supervisor -n <namespace> and kubectl top pods -n <namespace> -l app.kubernetes.io/component=supervisor. An unknown memory target or ScalingActive=False can mean the resource metrics API (usually Metrics Server) is unavailable. The top-level autoscaling values control the gateway HPA separately. Scaling up adds capacity for new sandboxes and does not redistribute existing owners; check individual Pod memory as well as the average. See the published High Availability guide for configuration and enabling or disabling autoscaling on an existing release.

With supervisor_topology = "gateway", the gateway always coordinates the shared supervisor as a pool, and the <release>-supervisor Deployment runs supervisor.replicas members (default 1, which has no failover). The chart renders only [openshell.gateway.supervisor_pool] recovery_timeout_secs (the driver's shared_supervisor table carries no pool settings), rolls one member at a time, sets no affinity by default (set supervisor.affinity to control supervisor scheduling; top-level affinity applies only to the gateway), gives each member a 60-second grace period to leave, and adds a <release>-supervisor PodDisruptionBudget with maxUnavailable: 1 unless supervisor.podDisruptionBudget.enabled is false (default true). If a node drain stalls on supervisor evictions, check the budget's allowed disruptions; disable it only for the maintenance window. Each member takes its stable member ID from OPENSHELL_SUPERVISOR_POOL_MEMBER_ID (the Pod name) and its runtime address from OPENSHELL_SUPERVISOR_POOL_ADVERTISE_IP (the Pod IP, which sandbox runtimes must be able to reach). Only the Deployment's replica count depends on supervisor.replicas: scaling it must not change the gateway ConfigMap or the supervisor Pod template, so neither the gateway nor the remaining members restart. A member removed by scaling down announces that it is leaving (pool member is leaving in its log), and the gateway logs cause=leaving as it moves each of its sandboxes. A scale step that restarts the gateway or every supervisor Pod means something besides supervisor.replicas changed in the release values.

kubectl -n openshell get deployment,poddisruptionbudget,pod -l app.kubernetes.io/component=supervisor -o wide
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E -A2 'supervisor_topology|shared_supervisor|supervisor_pool'
kubectl -n openshell logs -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
  -c openshell-gateway --prefix --tail=500 | grep -E 'supervisor pool|taking over a shared sandbox instance'
kubectl -n openshell logs -l app.kubernetes.io/name=openshell-supervisor --prefix --tail=500 \
  | grep -Ei 'takeover|sandbox instance'
kubectl -n <sandbox-namespace> logs -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload \
  --tail=200 | grep -E 'sandbox-supervisor-|Supervisor takeover'

Each sandbox instance runs on one member, and a live owner keeps it. A member's console log names an instance by its route (the Sandbox's openshell.ai/sandbox-runtime-supervisor-session annotation) at the end of admitted assigned sandbox instance <route>, removing withdrawn sandbox instance <route>, and relinquishing a sandbox instance ... <route>; the member whose latest such line for a route is an admission supervises it. Other lines that mention the route, such as SSH relay socket paths, say nothing about ownership. Gateway console fields are styled, so strip ANSI escapes (for example sed 's/\x1b\[[0-9;]*m//g') before matching cause=leaving. The gateway logs shared supervisor pool enabled; reconciling pool membership and ownership at startup. When an owner's process exits, its node is lost, or it leaves, the gateway logs taking over a shared sandbox instance from a failed supervisor with from, to, term, and cause (expired: the member's record expired; leaving: it announced leaving; restarted: its process changed, and a member whose process restarted re-adopts its own instances first), revokes the previous owner's credentials for that sandbox, and grants the instance to a live member. Runtimes dial the supervisor Service, and a member that does not own the instance redirects the runtime to the owner's Pod IP on the runtime port, so the workload must reach every supervisor Pod IP. The replacement and the sandbox runtime both log Supervisor takeover confirmed; previous owner fenced. The sandbox keeps its Pod, generation, PIDs, and memory; open proxied connections and exec or SSH streams break and must reconnect.

The sandbox runtime writes these OCSF findings to the workload Pod log:

• sandbox-supervisor-connection-lost: the runtime lost its supervisor and

froze the workload.

• sandbox-supervisor-recovery-expired: no replacement confirmed within the

recovery timeout, so the runtime terminated the workload.

• sandbox-supervisor-takeover-denied: the runtime refused a supervisor

without a valid grant for the current term, such as a fenced owner returning.

The recovery timeout is the gateway's [openshell.gateway.supervisor_pool] recovery_timeout_secs (Helm supervisor.recoveryTimeoutSeconds, 30 to 300 seconds, default 60). Each pool supervisor offers it to a sandbox runtime when it attaches. A graceful leave hands instances over at once. A lost node takes up to about 22 seconds to detect, before the replacement starts: the gateway closes the member's silent stream after 5 seconds, the member's record expires 15 seconds after its last refresh, and a reconcile pass runs every 2 seconds. A short timeout terminates workloads that would have recovered. A gateway replica that just started waits 30 seconds before it takes over instances whose owner looks failed, so members cut off by a gateway restart reconnect first. With one replica, or when no live member takes over before the timeout, the workload ends in sandbox-supervisor-recovery-expired.

supervisor pool reconciliation failed; retrying means a gateway replica could not read or write the pool's store records: supervisor_pool_member (one per member, refreshed every 2 seconds by the gateway replica serving its newest stream, independently of reconciliation; older than 15 seconds means dead), supervisor_instance_assignment (one per instance the pool must run, written by the replica that launched it; it holds the handle of the launch material, which the gateway's credential store keeps), supervisor_instance_owner (one per instance, written with compare-and-swap, naming the owner and its term), supervisor_pool_capacity (one per member, listing the routes whose owner record names it; changed in the same transaction as each owner record, so a member is never charged more than 256 instances across replicas), and supervisor_instance_report (one per instance and ownership term, the owner's latest report of that term, recorded by the replica serving it). supervisor pool member refresh failed or did not finish in time means the replica could not keep a member's record fresh; if it persists for 15 seconds the member's instances move. Every gateway replica serves the pool from these records, so a sandbox launched through one replica reaches a member connected to another. could not load a supervisor pool instance's launch material points at the credential store instead. Check the database. To read the records on PostgreSQL without changing them:

SELECT object_type, id,
       (extract(epoch FROM now()) * 1000)::bigint - updated_at_ms AS age_ms,
       convert_from(payload, 'UTF8') AS record
FROM objects
WHERE object_type IN ('supervisor_pool_member', 'supervisor_instance_assignment',
                      'supervisor_instance_owner', 'supervisor_pool_capacity',
                      'supervisor_instance_report')
ORDER BY object_type, id;

Do not edit or delete these records; the gateway fences takeovers with them.

Corporate upstream proxy

When the deployment routes sandbox egress through a corporate HTTP forward proxy, the operator-owned settings render under [openshell.drivers.kubernetes] from the Helm upstreamProxy values. Absent proxy configuration preserves direct-dial egress; any present-but-invalid value fails closed at gateway startup (validate_upstream_proxy_config) rather than silently reverting to a direct connection. Confirm the rendered configuration first:

kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
helm -n openshell get values openshell | grep -A12 upstreamProxy

Both http://host:port and https://host:port forward proxies are supported; plain-HTTP egress is out of scope and always dials directly. The credential Secret named by proxy_auth_secret_name must exist in the sandbox namespace with the key named by proxy_auth_secret_key, and Kubernetes will not create keys longer than 253 bytes or named ./...

An https:// proxy with a private CA, or a TLS-intercepting proxy, also needs upstreamProxy.caBundle.configMapName. That ConfigMap lives in the gateway's release namespace, not the sandbox namespace, and the gateway reads it and stages the bundle into each sandbox's supervisor bootstrap Secret. A missing ConfigMap leaves the gateway Pod in ContainerCreating, like oidc-ca and vault-ca:

kubectl -n openshell get configmap <proxy-ca-configmap> -o jsonpath='{.data}' >/dev/null && echo "CA ConfigMap present"
kubectl -n openshell describe pod -l app.kubernetes.io/name=openshell | grep -A5 'ContainerCreating\|MountVolume'
kubectl -n <sandbox-namespace> get secret <supervisor-bootstrap-secret> -o jsonpath='{.data.upstream-proxy-ca\.pem}' | head -c 20

An empty last command with proxy_ca_bundle set in the rendered TOML means the bundle never reached the sandbox; check the gateway logs for a proxy_ca_bundle error, which fails closed at startup rather than falling back to direct egress.

The proxy arguments and credential mount belong only to the separate supervisor Pod. The workload Pod must never receive them. The credential is projected read-only as openshell-upstream-proxy-auth at /run/openshell/upstream-proxy-auth and passed by file path; it must never appear in environment variables, annotations, or command arguments.

kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
kubectl -n <sandbox-namespace> get pod <supervisor-pod> -o jsonpath='{.spec.containers[0].command}' | grep -- '--upstream-'
kubectl -n <sandbox-namespace> get pod <supervisor-pod> -o jsonpath='{.spec.containers[0].command}' | grep -- '--upstream-proxy-ca-bundle'
kubectl -n <sandbox-namespace> get pod <supervisor-pod> -o jsonpath='{.spec.containers[0].volumeMounts}' | grep upstream-proxy-auth
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20

A missing Secret or wrong key leaves the pod stuck with a MountVolume.SetUp failed / secret ... not found event. If the pod starts but egress still fails, the corporate proxy itself is the next suspect: policy- approved TLS CONNECT requests that time out after policy evaluation usually mean the proxy URL is unreachable from the sandbox namespace, or a cluster-internal destination that should be direct is missing from no_proxy. Inspect the network supervisor logs for CONNECT and upstream-proxy decisions:

kubectl -n <sandbox-namespace> logs pod/<supervisor-pod> --tail=200 | grep -Ei 'upstream|connect|proxy'

Step 7: Check VM-Backed Gateways

Use the VM driver logs and host diagnostics available in the user's environment. Verify:

• The VM driver process is running and reachable by the gateway.
• The runtime rootfs exists and matches the expected architecture.
• mke2fs or mkfs.ext4, debugfs, and e2fsck from e2fsprogs are installed. Run openshell-gateway config preflight with the intended local VM configuration, service account, and environment to check the selected paths and versions before startup. A remote endpoint reports that host checks were not performed. Explicit sandbox_uid/sandbox_gid does not remove this prerequisite.
• A persisted overlay identity error is resolved from its owner marker, overlay

upper layer, prepared rootfs, explicit config, or current image. Do not assign 10001:10001 unless the persisted state reports that legacy identity.

• Host virtualization support is enabled.
• The sandbox supervisor can establish its authenticated gateway session.

For vm.mounts (virtiofs host bind mounts) failures, also verify:

• [openshell.drivers.vm] sets allow_driver_config = true and enable_bind_mounts = true; an external VM driver needs --enable-bind-mounts and allow_driver_config in --admission-config-json.
• [openshell.drivers.vm.resource_admission] sets enabled = false; bind mounts are rejected while resource admission is on.
• The libkrun runtime exports krun_add_virtiofs4; otherwise the driver reports "VM bind mounts need a libkrun runtime with virtiofs support". The driver extracts its runtime under $XDG_DATA_HOME/openshell/vm-runtime/<version>/ (default ~/.local/share). Check with nm -D <dir>/libkrun.so | grep krun_add_virtiofs4 on Linux or nm -gU <dir>/libkrun.dylib | grep krun_add_virtiofs4 on macOS.
• The sandbox launches on libkrun. GPU sandboxes and QEMU launches do not support VM mounts.
• On Linux with a non-root gateway, file creation in a read-write mount fails unless sandbox_uid/sandbox_gid match the gateway user's UID/GID. Compare id -u/id -g of the gateway user with the configured values.

Then run:

openshell status
openshell logs <sandbox-name>

Corporate upstream proxy

When VM sandbox egress routes through a corporate HTTP forward proxy, the operator-owned settings live under [openshell.drivers.vm] and the gateway forwards them to the openshell-driver-vm subprocess as --upstream-proxy, --upstream-no-proxy, --upstream-proxy-auth-file, --upstream-proxy-auth-allow-insecure, --upstream-proxy-connect-by-hostname, and --upstream-proxy-ca-bundle. Both the gateway and the driver validate them at startup, so any present-but-invalid value fails closed with an error naming the key rather than reverting to a direct dial. Confirm the configuration and the resulting driver argv first:

grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy'

Both libkrun and QEMU guests are NIC-less. Proxy settings, credentials, and private CA material stay with the host openshell-supervisor. For a proxy on the gateway host, use http://host.openshell.internal:<port>; the supervisor normalizes that name to host loopback. Inspect supervisor.log and supervisor.err.log under the sandbox state directory for connection or credential failures.

Common Failure Patterns

Reporting

When handing results back to the user, include:

• Active gateway endpoint and auth mode.
• Compute platform and driver.
• Gateway process or workload status.
• Recent gateway log summary.
• Missing or malformed TLS, OIDC/mTLS, or sandbox JWT material.
• Service exposure status.
• Sandbox workload status.
• The exact command that failed and the shortest fix.

Package Configuration Preflight

For a Debian, Ubuntu, or snap gateway that stops before certificate generation or daemon startup, validate the selected configuration without starting the service:

openshell-gateway config preflight [--path PATH | -- GATEWAY_ARGS...]

Without a path, preflight validates a nonempty OPENSHELL_GATEWAY_CONFIG or an auto-discovered XDG config; no config succeeds. An explicit missing path, legacy schema-v1 file, malformed TOML, symlink, or nonregular file fails before gateway startup. It also applies read-only effective-config checks for driver selection and configuration, sockets, rate limits, TLS, interceptors, and supervisor middleware. If the file omits the selector, preflight validates configured tables for auto-detectable drivers without running socket or process-based detection probes. Arguments after -- validate the effective daemon invocation, including its command-line overrides. Preflight preserves every failed file. Do not advise users to delete or rewrite it automatically; back it up and follow the manual schema-v2 migration in the Gateway Configuration reference.