huawei-cloud-modelarts-training-management

Huawei Cloud ModelArts Training Management

Full lifecycle management for ModelArts training jobs and related resources via hcloud CLI — 52 API interfaces across 8 functional domains.

Overview

This skill enables users to manage Huawei Cloud ModelArts training jobs and related resources through the hcloud CLI. It covers the complete training lifecycle including training job CRUD, algorithm management, training experiments, auto search (hyperparameter tuning), model import, and training image save.

Architecture

User Request → Agent → hcloud ModelArts <Operation> --cli-region={region} [--params] → Huawei Cloud ModelArts API

Applicable Scenarios

• Training Job Management: Create, list, show, stop, delete training jobs; query logs, metrics, engines, flavors, quotas
• Algorithm Management: Create, list, show, update, delete training algorithms; publish to gallery
• Training Job Tags: Create, show, delete tags for training jobs
• Training Experiments: Create, list, show, update, delete, check training experiments
• Training Job Events: Query job events, stages, tasks; list system events and scheduled events
• Model Import: Import, list, show, delete AI models; create ModelArts agency
• Auto Search: Query hyperparameter search trials, parameters analysis, yaml templates
• Training Image Save: Create and query training job image save tasks

Prerequisites

1. hcloud CLI installed and authenticated — Reference: https://support.huaweicloud.com/qs-hcli/hcli02003.html
2. Huawei Cloud AK/SK configured via hcloud (see Security & Credential Check below)
3. ModelArts service enabled in the target region
4. IAM permissions — See references/iam-policies.md

Security & Credential Check

⚠️ CRITICAL: This section defines the agent's security boundary. Follow it strictly.

🔒 Security Rules (MANDATORY)

• 🚫 NEVER read, echo, or print AK/SK values (e.g., cat ~/.hcloud/config.json, echo $HW_ACCESS_KEY is FORBIDDEN)
• 🚫 NEVER read or cat credential files such as ~/.hcloud/config.json or any file that may contain secrets
• 🚫 NEVER ask the user to input AK/SK directly in the conversation or command line
• 🚫 NEVER execute hcloud configure set --cli-access-key=... --cli-secret-key=... — credential configuration is the user's responsibility, done outside the agent session
• 🚫 NEVER pass --cli-access-key/--cli-secret-key on every API call — they will be exposed in process listings (ps aux) for the entire request duration
• ✅ ALWAYS use hcloud configure list to check credential status (presence check only, not values)
• ✅ If no valid profile exists, STOP and instruct the user to configure credentials outside this session
• ✅ Recommend using IAM user with minimal permissions instead of main account

✅ Agent Credential Check (the ONLY acceptable check)

# Step 1: Check credential status — presence only, not values
hcloud configure list

• ✅ Expected: Shows a valid profile with mode: AKSK
• ❌ If empty/invalid → Guide the user with the 3-step process below:

Step-by-step guidance (copy-paste for the user):

"No valid credentials found. Please follow these 3 steps in your terminal:" ① Get your AK/SK Log in to Huawei Cloud Console → IAM → Access Keys. If you don't have one, click "Create Access Key" and save it securely. ② Run this in your terminal (one-time setup): ``bash hcloud configure set --cli-access-key=<YOUR_AK> --cli-secret-key=<YOUR_SK> *(Replace <YOURAK>` and `<YOURSK> with the values from step ①)* **③ Come back and tell me "configured"**, I'll verify it works. *Note: The configure set command stores your credentials in ~/.hcloud/config.json. After that, all hcloud commands use them automatically — no need to pass AK/SK again.* *Agent: DO NOT execute any configure set` command yourself.*

🔧 User's Responsibility (for documentation, NOT agent execution)

The user configures credentials outside the agent session (in their own terminal):

# One-time setup — run this in your terminal, NOT in the agent chat
HISTCONTROL=ignorespace
 hcloud configure set --cli-access-key=<YOUR_AK> --cli-secret-key=<YOUR_SK>

⚠️ Note: hcloud CLI does NOT read HW_ACCESS_KEY/HW_SECRET_KEY environment variables (those are for Python SDK only). hcloud configure set is the only supported credential configuration method.

Workflow

Step 1: Identify the Operation

Determine which training operation the user needs based on their request:

Step 2: Execute CLI Command

hcloud ModelArts <Operation> --cli-region={region} [--key=value ...]

{region} must be replaced with the actual region, e.g., cn-north-4. Do NOT hardcode the region.

Step 2.5: ⚠️ Pre-Flight Check (MANDATORY for Write Operations)

Before executing ANY write operation (Create/Update/Delete/Stop/Change/Patch/Notify/Accept), you MUST: 1. Read [references/known-issues.md](references/known-issues.md) — Scan the relevant section for the target API 2. Apply documented workarounds — Especially for the following high-risk operations: | Operation | Critical Issue | Action Required | |-----------|--------------|----------------| | CreateTrainingJob | Must use v2 kind/metadata/spec/tasks format | Use --cli-jsonInput with v2 JSON structure | | CreateTrainingJob (dedicated pool) | flavor_id not accepted for dedicated pools | Omit flavor_id, specify pool_id instead | | CreateAlgorithm | JSON must use nested metadata + job_config structure | Use --cli-jsonInput with correct nesting (see known-issues #27) | | CreateAlgorithm | OBS code directory must exist and be accessible | Verify OBS path before calling API | | StopTrainingJob | Can only stop jobs in creating/waiting/running state | Check job status first | | CreateModel | source_location format must be obs://bucket/path/ | Verify OBS path format | | CreateModel | model_type must be a valid value | Check known-issues #22 for valid values | | CreateSaveImageJob | Requires job in running state; SWR namespace required | Verify job status and SWR config | | ChangeAlgorithm | Must include --metadata.name parameter | Add metadata.name to request | | NotifyTrainingJobInformation | Parameter names differ from API docs | Check known-issues #24 | | UpdateTrainingJob | Not a valid CLI operation | Use ChangeTrainingJob instead | Skipping this step will cause preventable failures. Every issue in the table was discovered through real API testing.

Step 3: Pricing Inquiry (for chargeable operations)

Before executing chargeable write operations (CreateTrainingJob on public resource pool, CreateTrainingExperiment), perform a BSS pricing inquiry to show estimated costs to the user.

📖 For detailed pricing inquiry procedures, read [references/pricing-inquiry.md](references/pricing-inquiry.md)

Chargeable Operations

Inquiry Workflow

1. Detect chargeable operation (CreateTrainingJob without pool_id)
2. Extract flavor_id from spec.resource.flavor_id parameter
3. Get project ID via IAM KeystoneListAuthProjects
4. Call BSS ListOnDemandResourceRatings to query on-demand price
5. Calculate estimated cost: unit_price × node_count × estimated_duration
6. Display price table and cost estimate to user
7. Proceed to Step 4 (user confirmation) after showing pricing info

Quick Pricing Query

Use the helper script for quick price queries:

# Query single flavor
bash scripts/query-pricing.sh --region cn-north-4 --flavor modelarts.bm.gpu.v100

# Query multiple flavors for comparison
bash scripts/query-pricing.sh --region cn-north-4 --flavor modelarts.bm.gpu.v100 --flavor modelarts.cpu.8u32g --flavor modelarts.bm.ascend910

BSS Inquiry Parameters (Fixed)

Step 4: Handle Write Operations

For all write operations (Create/Update/Delete/Stop/Change/Patch/Notify/Accept/Batch), prompt the user for confirmation before execution. For chargeable operations, confirmation includes reviewing the pricing information from Step 3.

Step 5: Generate Console URL (after CreateTrainingJob)

After successfully creating a training job, generate and display the ModelArts console URL so the user can directly access the job details in the web console.

URL Format

Training job detail page (after CreateTrainingJob):

https://console.huaweicloud.com/modelarts/?region={region}#/training/detail/{job_id}

Training job list page (after ListTrainingJobs):

https://console.huaweicloud.com/modelarts/?region={region}#/training

Important: Use #/training/detail/{job_id} for detail and #/training for list (new console paths). Do NOT use #/trainingJobs/details/{job_id} or #/trainingJobs — those are deprecated old console paths.

Workflow

1. CreateTrainingJob returns job_id in the response
2. Construct console URL: https://console.huaweicloud.com/modelarts/?region={region}#/training/detail/{job_id}
3. Display the URL to the user with a summary of the created job

Example Output

✅ Training job created successfully!

  Job Name:    my-training-job
  Job ID:      39cefbeb-0d86-46cb-a55a-fe6a62f3529b
  Status:      creating
  Region:      cn-north-4

  🔗 Console URL:
  https://console.huaweicloud.com/modelarts/?region=cn-north-4#/training/detail/39cefbeb-0d86-46cb-a55a-fe6a62f3529b

Note: The console URL uses hash routing (#/training/detail/{job_id}). Do NOT use the old path #/trainingJobs/details/{job_id} — it redirects to a deprecated console page. The URL is valid as long as the training job exists and the user has ModelArts access permissions for the region.

KooCLI Command Format Standard

hcloud ModelArts <Operation> --cli-region={region} [--key=value ...]

Note: --project_id is auto-resolved from authentication credentials if omitted. Include it explicitly only when targeting a specific project.

Complex parameters: For complex nested parameters (e.g., config, spec), use --cli-jsonInput=/path/to/file.json. The JSON file must wrap body in {"body": {...}} envelope.

Core Commands

All 52 CLI command examples across 8 functional domains are documented in a separate reference file.

📖 For detailed command syntax, parameters, and examples, read [references/cli-command-examples.md](references/cli-command-examples.md)

Quick Index

When executing any command, always refer to the reference file for exact parameter names, required/optional flags, and usage patterns.

Capability Boundary（能力边界）

This skill covers only ModelArts training management — 52 APIs across 8 functional domains (see Core Commands above). The following ModelArts capabilities are NOT supported:

When receiving requests for the above unsupported capabilities, explicitly inform the user that this skill does not support them, and suggest using the ModelArts console (https://console.huaweicloud.com/modelarts/) or the relevant dedicated skill.

Parameter Confirmation

SDK Fallback

If a CLI operation fails due to a CLI bug, fall back to SDK:

from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmodelarts.v2.modelarts_client import ModelArtsClient
from huaweicloudsdkmodelarts.v2.region.modelarts_region import ModelArtsRegion

credentials = BasicCredentials(ak="{AK}", sk="{SK}", project_id="{project_id}")
client = ModelArtsClient.new_builder() \
    .with_credentials(credentials) \
    .with_region(ModelArtsRegion.value_of("{region}")) \
    .build()

Reference Documents

Known Issues Summary

• Complex nested params: Use --cli-jsonInput with JSON file for CreateTrainingJob, CreateAlgorithm, CreateModel, etc.
• `--cli-jsonInput` syntax: Use file path directly (no @ prefix), JSON must be wrapped in {"body": {...}}
• Training job logs: ShowObsUrlOfTrainingJobLogs returns a temporary OBS URL (valid for 5 minutes)
• StopTrainingJob: Can only stop jobs in creating, waiting, or running state
• Auto search: Trial early stop only works on running trials

See references/known-issues.md for full details.

Notes

• All write operations (Create/Update/Delete/Stop/Change/Patch/Notify/Accept) require user confirmation before execution
• Pricing inquiry: CreateTrainingJob (public pool) and CreateTrainingExperiment trigger BSS pricing inquiry before execution — see references/pricing-inquiry.md
• Dedicated resource pool: Training jobs using a dedicated resource pool (pool_id specified) do NOT require pricing inquiry — the pool is already billed
• Region is not hardcoded — uses {region} placeholder
• project_id is auto-resolved when omitted
• No hardcoded AK/SK in any file — credentials configured via hcloud configure set by the user outside the agent session
• Agent must use hcloud configure list for credential presence check only — NEVER read/echo AK/SK values
• If no valid credentials found, agent must STOP and guide user through the 3-step configuration process (see Prerequisites section)
• SDK fallback available when CLI encounters bugs
• Complex nested parameters use --cli-jsonInput with JSON file