huawei-cloud-cce-cluster-management

Huawei Cloud CCE Cluster Management

Overview

Manage CCE (Cloud Container Engine) cluster lifecycle, including cluster creation/deletion/hibernation/awakening, node pool management, node scheduling control, and addon management.

The skill executes Huawei Cloud API calls through hcloud (KooCLI) and Kubernetes node operations (cordon/uncordon/drain/status) through kubectl cce (the kubectl-cce plugin). The plugin connects to the CCE API Gateway using AK/SK credentials — no cluster EIP or manual kubeconfig required. Two operations (create cluster, create node pool) fall back to the Python SDK because of a known hcloud metadata parsing defect — see cce-api-guide.md.

Dependency: This skill requires kubectl and the kubectl-cce plugin. Install them via the huawei-cloud-kubectl-cce-installer skill.

⛔ Security Constraints

Dangerous Operation Confirmation Mechanism

This skill strictly enforces a two-step confirmation mechanism for all dangerous operations to prevent accidental service disruption or data loss.

All dangerous operations require confirm=true parameter to execute. Otherwise, they return a preview and confirmation prompt.

Operations Requiring Confirmation

Workflow

Step 1: Preview Operation - Call without confirm parameter

# Example: Preview cluster deletion
python3 huawei-cloud.py huawei_delete_cce_cluster \
  region=cn-north-4 \
  cluster_id=xxx

Returns: operation preview, risk warning, confirmation example

Step 2: Confirm Execution - Call with confirm=true

# Example: Confirm and execute deletion
python3 huawei-cloud.py huawei_delete_cce_cluster \
  region=cn-north-4 \
  cluster_id=xxx \
  confirm=true

Credential Security

✅ This skill strictly follows these security rules:

1. No persistent credential storage - Never saves AK/SK, tokens, or certificates to disk
2. No long-term memory cache - AK/SK exists only during API call, released afterward
3. Only project ID memory cache - Non-sensitive project ID cached in process memory
4. No credential leakage - Never includes AK/SK in logs, responses, or errors
5. Temporary file cleanup - Temporary kubeconfig files are deleted immediately after use
6. Config-first credential passing - If hcloud CLI config has credentials, AK/SK are not passed as CLI arguments (avoids ps aux exposure). Environment variables are only set for subprocesses if not already present in the parent environment.

Credentials are resolved from parameters or environment variables (process-level, never written to disk):

• Permanent credentials: HW_ACCESS_KEY + HW_SECRET_KEY
• Temporary credentials (recommended for CI/CD / IAM temporary access keys): HW_ACCESS_KEY + HW_SECRET_KEY + HW_SECURITY_TOKEN
• HW_PROJECT_ID is optional — auto-fetched via hcloud IAM KeystoneListProjects when not provided (only for hcloud API calls; kubectl-cce node operations do not need it)
• Per-call ak / sk parameters override the environment variables for that single call

Security hardening tip: Set HW_ACCESS_KEY / HW_SECRET_KEY in the parent process environment (e.g., ~/.bashrc or systemd environment file) so they are inherited by subprocesses without explicit passing. The skill detects existing env vars and skips passing AK/SK as CLI arguments to hcloud, preventing credential exposure in process listings (ps aux). kubectl-cce reads these env vars directly from the process environment.

# Permanent
export HW_ACCESS_KEY  # your access key ID
export HW_SECRET_KEY  # your secret access key
export HW_REGION_NAME="cn-north-4"

# Temporary (add security token)
export HW_SECURITY_TOKEN  # your security token

Node Login Password Security

When creating nodes or node pools, the login credential is resolved with the following three-level priority:

1. `ssh_key` parameter — SSH key pair name (preferred when available). Mutually exclusive with password.
2. `password` parameter — raw node login password passed per call (8–26 chars, ≥3 of: uppercase / lowercase / digits / special).
3. `CCE_NODE_PASSWORD` environment variable — used when neither ssh_key nor password is provided.
4. Auto-generated random password — when none of the above is supplied, the skill generates a strong random password automatically.

⚠️ The auto-generated password is NEVER returned in the tool response (not in data, not in message, not in logs). To access the node afterwards, the user must reset the node password via the CCE console or the ECS API. The success message only contains a hint instructing the user to reset the password.

The raw password is never sent to the CCE API directly; the skill applies SHA-512 salted encryption + base64 encoding internally (see cce-cluster-parameters.md).

Prerequisites

CLI Tools

• `hcloud` (Huawei Cloud KooCLI 7.2+) — drives all Huawei Cloud API calls. Install:

  curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh
  hcloud version   # verify install

`kubectl` + `kubectl-cce` plugin — required for node scheduling operations (cordon/uncordon/drain/status). Install via the huawei-cloud-kubectl-cce-installer skill:

  # Check if already installed
  bash ../huawei-cloud-kubectl-cce-installer/scripts/install_kubectl_cce.sh --check

  # Install (after confirming the plan)
  sudo bash ../huawei-cloud-kubectl-cce-installer/scripts/install_kubectl_cce.sh --execute --bin-dir /usr/local/bin

The kubectl cce plugin connects through the CCE API Gateway using AK/SK credentials — no cluster EIP or manual kubeconfig needed.

Python Environment

• Python 3.8+
• Install SDK packages (retained for create-cluster / create-nodepool fallback) and helpers:

  pip install huaweicloudsdkcce huaweicloudsdkcore huaweicloudsdkiam passlib pyyaml

passlib provides SHA-512 salting. huaweicloudsdkcce + huaweicloudsdkcore + huaweicloudsdkiam are used by the two SDK fallback functions (create cluster, create node pool) and IAM project-ID resolution. pyyaml parses hcloud JSON output.

Environment Variables

The skill detects existing environment variables before passing credentials to subprocesses:

• hcloud CLI calls: if HW_ACCESS_KEY/HW_SECRET_KEY are already in os.environ, the skill skips passing --cli-access-key/--cli-secret-key as CLI arguments, preventing exposure in ps aux.
• kubectl-cce calls: the plugin reads HW_ACCESS_KEY/HW_SECRET_KEY from the process environment directly. If these variables are already set, the subprocess inherits them naturally — the skill does not set them again.

🔒 Security Tip: Set HW_ACCESS_KEY/HW_SECRET_KEY at the system or session level (e.g., in ~/.bashrc or a systemd environment file) so the skill never needs to pass them as explicit CLI arguments. This eliminates the risk of credential exposure in process listings (ps aux) and shell history.

# Set credentials at session level (the skill detects these and skips
# passing them as explicit CLI arguments to hcloud/kubectl-cce)
export HW_ACCESS_KEY  # your access key ID
export HW_SECRET_KEY  # your secret access key
export HW_REGION_NAME="cn-north-4"
# Optional, for temporary credentials:
export HW_SECURITY_TOKEN  # your security token
# Optional, for node login when ssh_key is not used:
export CCE_NODE_PASSWORD  # your node password (8-26 chars, at least 3 char categories)

IAM Permission Policies

Ensure the IAM user has the minimum required permissions:

参数确认

Before executing any command, confirm the following parameters with the user:

认证参数

集群参数

节点池参数

其他参数

Core Commands

Cluster Query

Cluster Management

Dynamic EIP Binding: huawei_bind_cce_cluster_eip supports dynamic EIP assignment. If eip_id is not provided, the skill automatically: (1) lists existing EIPs and finds an unbound one (status=DOWN), (2) if none available, creates a new EIP (traffic billing, 5Mbps, PER share type), (3) binds it to the cluster. The response includes eip_created (true/false), eip_id, eip_address, and public_endpoint (the External API URL).

Recommended defaults:

• Cluster type: Turbo (best performance with ENI network)
• Container network: eni for Turbo clusters (default in this skill)
• Cluster version: omit `cluster_version` to let the API pick the latest supported version; specify it only when the user requires a specific Kubernetes version
• Naming format: <env>-<app>-cluster (e.g., prod-web-cluster)

Node Pool Management

Recommended defaults:

• Naming format: <env>-<role>-pool (e.g., prod-worker-pool)
• Initial node count: 2 for HA, or 0 with autoscaling
• Enable autoscaling for dynamic scaling
• For Turbo clusters, use ENI-compatible flavors (e.g., c7.large.2)

Node Management

Node ID note: huawei_delete_cce_node requires the node UID (from huawei_list_cce_nodes → metadata.uid), not the node name. Using a name instead of UID will return an error.

Node scheduling operations (cordon, uncordon, drain, status) are executed via kubectl cce — the kubectl-cce plugin connects to the CCE API Gateway using AK/SK credentials. No cluster EIP or manual kubeconfig required. The plugin handles cordon, eviction, PodDisruptionBudget (PDB) compliance, and DaemonSet pod skipping natively.

huawei_cce_node_drain follows standard drain semantics: it first cordons the node, then evicts all resident pods (excluding DaemonSet pods) via the k8s Eviction API, which respects PodDisruptionBudget (PDB). Pods blocked by PDB will be reported in the failed_pods field.

Note: Prefer node pools for managed scaling. Direct node creation is for special cases.

Addon Management

Common addons:

• coredns - DNS service
• metrics-server - Monitoring metrics
• everest - Storage driver

Addon notes (from E2E verification): - huawei_get_cce_addon_detail, huawei_uninstall_cce_addon, and huawei_update_cce_addon all require the addon UID (from huawei_list_cce_addons → metadata.uid), not the addon name. Using a name instead of UID will return error CCE.03400001. - Addon status is in the status.status field (e.g., running, upgrading, abnormal), not spec.status. - After huawei_update_cce_addon, the addon may enter upgrading state. Wait for it to return to running before performing subsequent operations (e.g., uninstall).

Network Prerequisites

Use these tools to find VPC/subnet IDs before cluster creation.

Supported Regions

Output Format

All tools return JSON-formatted results containing:

• status: operation result (success / error)
• data: operation-specific response (cluster info, node list, addon details, etc.)
• message: human-readable description of the result
• warning: risk warning for dangerous operations (preview mode only)

Verification

See verification-method.md for detailed verification steps. Quick checklist:

1. hcloud version (should be 7.2+)
2. kubectl version --client
3. Verify HW_ACCESS_KEY / HW_SECRET_KEY env vars are set (add HW_SECURITY_TOKEN for temporary credentials)
4. hcloud CCE ListClusters --cli-region=cn-north-4 (connectivity test)
5. Test dangerous operation preview (call without confirm=true)

Best Practices

• Use environment variables (HW_ACCESS_KEY / HW_SECRET_KEY) for credentials — avoid hardcoding; add HW_SECURITY_TOKEN for temporary credentials
• Always preview dangerous operations before confirming with confirm=true
• Prefer Turbo clusters (container_network_type=eni) — the default — for high-performance workloads
• Omit cluster_version unless the user requires a specific Kubernetes version
• Resize node pools during low-traffic periods to minimize business impact
• Keep node pools at ≥2 nodes for production workloads to ensure redundancy
• Regularly check cluster health via huawei_list_cce_clusters

References

Notes

• Ensure AK/SK (and HW_SECURITY_TOKEN for temporary credentials) has correct IAM permissions
• Different regions may have different resource availability
• All dangerous operations require confirmation
• Deletion operations are irreversible
• Hibernate cluster stops all workloads - use during non-business hours
• Node drain uses kubectl drain natively: handles cordon + eviction + PDB compliance + DaemonSet skip automatically. Use --ignore-daemonsets --delete-emptydir-data flags.
• Turbo clusters recommended for best performance with ENI network
• Create cluster / create node pool use the Python SDK fallback due to a known hcloud metadata parsing defect; all other operations use hcloud CLI