huawei-cloud-mrs-host-fault-diagnose

Huawei Cloud MRS Host Fault Diagnosis Skill

Overview

This skill diagnoses Huawei Cloud MRS (MapReduce Service) cluster faults. Given a service name and/or node name, it progressively localizes the root cause: quick log scan first, host troubleshooting when host issues are found, detailed investigation when no conclusion is reached.

Architecture: Caller (Agent) -> check_api_mode.py (Python, scripts/) determines the API mode -> either lakewatch_api_client.py -> LakeWatch API -> MRS cluster (node resource data, logs, MRS Manager proxy) or manager_api_client.py -> MRS Manager REST API (28443). Per-layer knowledge base (fault_layer/ + scenarios/ + propagation.md in lakewatch mode; fault_layer_manager/ + scenarios_manager/ + propagation_manager.md in manager mode) drives the diagnosis flow; per-component config under components/ is shared by both modes; three fault layers (host -> instance -> service) with propagation chain tracing.

Note on language: This SKILL.md and the documents under references/ are written in English per the repository spec. The knowledge base documents under fault_layer/, fault_layer_manager/, scenarios/, scenarios_manager/, components/, propagation.md, and propagation_manager.md are also in English. Commands and code blocks are English throughout.

Applicable Scenarios:

• A service is reported unhealthy and the root cause must be localized
• An instance is reported faulty on a specific node
• A host is reported unreachable or abnormal
• Progressive fault triage from quick scan to deep investigation

Typical Use Cases:

• "KrbServer出问题了，帮忙诊断一下" (service fault, no node specified)
• "8-5-225-6上的KrbServer挂了" (instance fault, service + node specified)
• "8-5-225-6出问题了" (host fault, node only)
• "MRS集群KrbServer启动失败，集群ID xxx"
• "DBService停止异常，节点8-5-225-6"

Critical Constraints

Important constraints: 1. Read-only: This skill only runs information-gathering commands (view logs, query status, collect resource data). It MUST NOT run any start/stop, modify, or delete operations. 2. User confirmation for repair: The skill only provides executable repair suggestions; it MUST NOT directly execute any repair operation. All repair actions require user confirmation. 3. Strict execution: Diagnose strictly according to the knowledge base content under this skill directory. Fabricating diagnostic commands outside the knowledge base is prohibited.

Prerequisites

1. Python Requirements

• Python >= 3.7
• Dependencies: pyyaml (YAML parsing), cryptography (Windows AES password encryption only)
• Linux uses CryptoAPI for password encryption (no cryptography dependency)
• Verify installation: python3 --version (Linux) / python --version (Windows)

This skill does NOT require KooCLI (hcloud). It calls the LakeWatch API through scripts/lakewatch_api_client.py (lakewatch mode) or the MRS Manager REST API through scripts/manager_api_client.py (manager mode). For the client setup, see CLI Installation Guide.

2. LakeWatch Credential Configuration

• A valid LakeWatch service account (username + password)
• The password MUST be encrypted with --encrypt-password and stored in scripts/lakewatch_api_config.yaml (auth.encrypted_password). Never store the plaintext password.
• Security Rules:
• Never expose the LakeWatch password in conversation or command output
• Never ask the user to input the plaintext password in conversation; use the interactive --encrypt-password flow
• The token is cached locally with owner-only file permissions (Win: %TEMP%\lakewatch_token\, Linux: /tmp/lakewatch_token/)

3. MRS Manager Credential Configuration (Manager Mode)

Manager mode is enabled when scripts/manager_api_config.yaml exists and auth.encrypted_password is set.

• A valid MRS Manager account (username + password)
• Configure the Manager floating IP in server.host (port default 28443). To obtain it, run grep float_ip /opt/huawei/Bigdata/om-server/OMS/workspace/conf/oms.ini on the OMS node, or ask the cluster administrator.
• The password MUST be encrypted with python3 scripts/manager_api_client.py --encrypt-password and stored in scripts/manager_api_config.yaml (auth.encrypted_password). Never store the plaintext password.
• Security Rules:
• Never expose the Manager password in conversation or command output
• Never ask the user to input the plaintext password in conversation; use the interactive --encrypt-password flow
• Windows AES ciphertext requires the .aes_key file to be migrated together to decrypt on another machine; Linux SCC ciphertext is not portable across clusters
• See MRS Manager API Client for the full client usage.

4. Access Permissions

• Lakewatch mode: Reachability to the LakeWatch service endpoint (configured in scripts/lakewatch_api_config.yaml server.host/port); the LakeWatch account must have permission to call the MRS Manager proxy and collect node resource/log data on the target cluster
• Manager mode: The script runtime environment must be able to reach the Manager port 28443; the Manager account needs read permissions on alarm, host, instance, and log APIs
• See IAM Policies for the access model and required roles

5. Dependent Skill: huawei-cloud-mrs-host-alarm-diagnose

This skill references the per-alarm diagnosis knowledge base from the huawei-cloud-mrs-host-alarm-diagnose skill (sibling directory under skills/bigdata/mrs/). When the fault diagnosis flow encounters a known alarm (12006/12007/25000/25500/27001), it loads the corresponding document: ../huawei-cloud-mrs-host-alarm-diagnose/alarms/<alarm_id>.md in lakewatch mode, or ../huawei-cloud-mrs-host-alarm-diagnose/alarm_manager/<alarm_id>.md in manager mode.

• If the alarm skill exists, load the referenced document and follow its diagnosis flow
• If NOT exist, inform the user and proceed with the generic fault diagnosis flow
• The dependency is document-level reference only (loading markdown by relative path), NOT a direct skill call. Both skills share the same LakeWatch/Manager API clients and config format.

Command Format Standard

This skill uses the LakeWatch API client instead of KooCLI. The unified command format is:

# Linux
python3 <skill_dir>/scripts/lakewatch_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2'

# Windows
python <skill_dir>/scripts/lakewatch_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2'

Windows (PowerShell) quote rule: every " inside a value must be replaced with """ (including " inside [] and {}), otherwise the server returns {"message":"Unknown exception","success":false,"code":"500"}:

# Correct on Windows
-p 'keywords=["""ERROR"""]'
-p 'env={"""PID""":"""123"""}'

# Wrong on Windows (will fail)
-p 'keywords=["ERROR"]'

Linux (bash) quote rule: keep " as-is inside the value, wrap the whole value in single quotes:

# Correct on Linux
-p 'keywords=["ERROR","Exception"]'
-p 'env={"PID":"123"}'

For the full API catalog, parameters, and the token/encryption mechanism, see LakeWatch API Client.

MRS Manager API Client (Manager Mode)

When check_api_mode.py reports manager, use manager_api_client.py instead of the LakeWatch client. The unified command format is:

# Linux
python3 <skill_dir>/scripts/manager_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2' --json

# Windows
python <skill_dir>/scripts/manager_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2' --json

For the full API catalog, authentication modes, metric names, and password encryption mechanism, see MRS Manager API Client.

Workflow

Step 0: Determine the API Mode

Run the mode check script to determine whether diagnosis is based on MRS Manager or LakeWatch:

• Windows: python scripts/check_api_mode.py
• Linux: python3 scripts/check_api_mode.py

The script checks whether scripts/manager_api_config.yaml exists and whether encrypted_password is filled in, and returns a JSON result:

{"mode": "manager", "reason": "..."}     // manager-based
{"mode": "lakewatch", "reason": "..."}   // lakewatch-based

Rules:

• manager_api_config.yaml does not exist -> default lakewatch
• File exists but encrypted_password is empty -> default lakewatch
• File exists and encrypted_password is not empty -> manager

The mode determines which knowledge base directories to load throughout the workflow:

In the rest of this SKILL.md, <FAULT_LAYER> denotes fault_layer (lakewatch) or fault_layer_manager (manager), <SCENARIOS> denotes scenarios (lakewatch) or scenarios_manager (manager), and <PROPAGATION> denotes propagation.md (lakewatch) or propagation_manager.md (manager).

Step 1: Determine Fault Entry

Extract fault information from the user input and determine the diagnosis entry:

Step 2: Locate the Fault Object

Mode note: The command blocks below show lakewatch-mode commands. In manager mode, use the corresponding manager_api_client.py commands — see Core Commands -> Manager Mode Commands and <SCENARIOS>/data_collection.md for the per-mode equivalents.

Entry A: Service Fault (has servicename, no nodename)

Load components/<service_name>.md for component config. Query OMS primary/standby nodes, check process on each node:

# lakewatch mode
python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

Manager mode equivalents: get_oms_info (OMS primary/standby nodes); get_host_process (process status).

Decision:

Entry B: Instance Fault (has servicename + nodename)

Load components/<service_name>.md. Directly check process on that node:

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

Manager mode equivalent: get_host_process (process status).

Decision:

Entry C: Host Fault (has nodename, no servicename)

Query OMS primary/standby nodes, query node IP, ping the faulty node from OMS active node:

# lakewatch mode
python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

# lakewatch mode
python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=<cluster_id>' \
  -p 'node_name=<node_name>'

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=ping-check' \
  -p 'env={"TARGET_IP":"<target_ip>"}' \
  -p 'node_name=<oms_active_node>'

Manager mode equivalents: get_oms_info (OMS primary/standby); get_hosts -p 'hostname=<node_name>' (node IP); check_remote (remote connectivity — no dedicated ping-check API).

Decision:

Step 3: Quick Log Scan

For the faulty node, quickly scan three layers of logs (Controller -> NodeAgent -> component), looking for clear ERROR:

# lakewatch mode - Controller log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=/var/log/Bigdata/controller' \
  -p 'log_file_name=exe.log*' \
  -p 'keywords=["<service_name>","ERROR","fail","timeout","Exception"]' \
  -p 'log_type=local' \
  -p 'node_name=<oms_active_node>'

# lakewatch mode - NodeAgent script log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=/var/log/Bigdata/nodeagent/scriptlog' \
  -p 'log_file_name=*.log*' \
  -p 'keywords=["<service_name>","ERROR","fail","exit"]' \
  -p 'log_type=local' \
  -p 'node_name=<node_name>'

Manager mode equivalents (see <SCENARIOS>/data_collection.md): browse_log (Controller exe.log, NodeAgent script.log); start_log_search + get_log_search_progress (keyword search).

If service_name is known, also check the component's own log (path from components/<service_name>.md):

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=<log_directory>' \
  -p 'log_file_name=<log_file_name>' \
  -p 'keywords=["ERROR","Exception","FATAL","fail","OOM"]' \
  -p 'log_type=local' \
  -p 'node_name=<node_name>'

Decision:

Step 4: Detailed Investigation

When the quick log scan yields no conclusion, collect complete data:

1. Load Data Collection to collect process/port/HA/resource/alarm/framework logs
2. Load Instance Fault Diagnosis for instance-level diagnosis (includes scenario identification)
3. If needed, load Service Fault Diagnosis for service-level diagnosis
4. If host issue is found, load Host Fault Diagnosis for host-level diagnosis

Step 5: Propagation Chain Tracing

Load Propagation Chain to trace the root cause propagation path and impact scope.

Step 6: Output Diagnosis Conclusion

## Diagnosis Result

| Item | Content |
|------|---------|
| Diagnosis time | [time] |
| Cluster ID | [cluster_id] |
| Faulty component | [service_name] |
| Faulty node | [node_name] |

### Diagnosis Process

| Step | Result |
|------|--------|
| Instance status | [which nodes normal/abnormal] |
| Quick log scan | [found/not found clear ERROR] |
| Host troubleshooting | [normal/abnormal: ...] |
| Detailed investigation | [process/port/HA/resource results] |

### Propagation Path

[root cause] -> [propagation] -> [symptom] (single-layer root cause if no propagation)

### Root Cause Analysis

**Root cause layer**: [host/instance/service]
**Root cause type**: [specific reason]

### Repair Suggestion

| Priority | Operation | Description | Needs user confirmation |
|----------|-----------|-------------|-------------------------|
| 1 | [operation] | [description] | Yes |

Core Commands

Query OMS Primary/Standby Nodes

python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

Query Node IP

python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=<cluster_id>' \
  -p 'node_name=<node_name>'

Collect Node Resource Data

# Process basic info
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

# Port check
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=port-check' \
  -p 'env={"PORT":"<port>"}' \
  -p 'node_name=<node_name>'

# HA resource status
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=ha-resource-status' \
  -p 'node_name=<node_name>'

# Disk space / Memory / CPU load
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=disk-space' \
  -p 'node_name=<node_name>'

Supported strategy_name values include: system-load, memory-usage, disk-space, disk-io, network-io, file-handle, port-check, high-cpu-processes, high-memory-process, zombie-process, dns-check, network-connectivity-test, process-basic-info, process-file-descriptor, jstack-thread-dump, disk-health-check, disk-smart-info, ha-resource-status, omm-process-tree, and more. See LakeWatch API Client for the full list.

Collect Alarm Log Data

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=<log_directory>' \
  -p 'log_file_name=<log_file_name>' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local'

When the log time format is non-standard ISO (e.g. [2026-07-07 20:54:25,171]), pass time_pattern:

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=2026/07/07 20:54:00 GMT+08:00' \
  -p 'log_directory=/var/log/Bigdata/omm/oms/pms' \
  -p 'log_file_name=pms*.log' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local' \
  -p 'time_pattern=^\[([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})||ymdHMS'

Proxy MRS Manager GET API

# Query cluster services
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/services'

# Query host processes
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/hosts/<node_name>/processes'

# Query active alarms
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/alarms'

target_url MUST NOT start with /. The proxy requires Agent >= 1.0.5 and reported OMS node info. Only GET is supported currently.

Manager Mode Commands (Manager Mode)

When check_api_mode.py reports manager, use manager_api_client.py for the equivalent queries:

# Query OMS primary/standby nodes
python3 manager_api_client.py -a get_oms_info --json

# Query cluster services
python3 manager_api_client.py -a get_cluster_services \
  -p 'cluster_id=<cluster_id>' --json

# Query host detail (disk/memory/CPU usage)
python3 manager_api_client.py -a get_host_detail \
  -p 'hostname=<node_name>' --json

# Query host process status
python3 manager_api_client.py -a get_host_process \
  -p 'hostname=<node_name>' --json

# Query service instances (HA status)
python3 manager_api_client.py -a get_instances \
  -p 'cluster_id=<cluster_id>' \
  -p 'service_name=<service_name>' \
  -p 'hostname=<node_name>' --json

# Query host monitor metrics (dev_ prefix)
python3 manager_api_client.py -a get_host_metrics \
  -p 'hostname=<node_name>' \
  -p 'metric_names=dev_cpu_surp_avg,dev_load_one_min' --json

# Check remote node connectivity (replaces ping-check/network-connectivity-test)
python3 manager_api_client.py -a check_remote \
  -p 'remote_ip=<target_ip>' \
  -p 'remote_port=22' \
  -p 'remote_user_name=omm' \
  -p 'remote_client_path=/opt/huawei/Bigdata/nodeagent' --json

# Browse a log file (file_name must be a full path)
python3 manager_api_client.py -a browse_log \
  -p 'hostname=<node_name>' \
  -p 'file_name=/var/log/Bigdata/controller/exe.log' \
  -p 'start_line=1' \
  -p 'end_line=500' \
  -p 'search=<service_name>' --json

# Search logs by keyword (returns task_id, then poll progress)
python3 manager_api_client.py -a start_log_search \
  -p 'cluster_id=<cluster_id>' \
  -p 'key_word=ERROR' \
  -p 'start_time=<alarm_time>' \
  -p 'end_time=<current_time>' \
  -p 'services=<component>:<service_name>:<role_name>' \
  -p 'min_log_level=WARN' --json

python3 manager_api_client.py -a get_log_search_progress \
  -p 'search_id=<task_id>' --json

start_log_search services format: component:service:role (e.g. HDFS:HDFS:NameNode); start_time/end_time format: yyyy-MM-ddTHH:mm:ss. See MRS Manager API Client for metric names and full parameter rules.

Parameter Confirmation

Output Format

The diagnosis report is output in Markdown, containing:

• Diagnosis result table: diagnosis time, cluster ID, faulty component, faulty node
• Diagnosis process: step-by-step results (instance status, quick log scan, host troubleshooting, detailed investigation)
• Propagation path: root cause -> propagation -> symptom (single-layer if no propagation)
• Root cause analysis: root cause layer (host/instance/service) + root cause type
• Repair suggestion table: priority, operation, description, needs-user-confirmation (all repair actions require user confirmation)

See the template in the Workflow -> Step 6 section.

Verification Method

See Verification Method for the installation, configuration, and function verification steps.

Best Practices

1. Determine entry first: Based on user-provided information (servicename, nodename), determine whether the entry is service fault, instance fault, or host fault before starting diagnosis.
2. Progressive investigation: Always start with quick log scan (Step 3); only escalate to detailed investigation (Step 4) when no clear conclusion is reached.
3. Substitute placeholders: Replace <cluster_id>, <alarm_time>, <node_name>, <target_ip>, <process_name>, etc. with actual user-provided values; never hardcode them.
4. Quote parameters: Always wrap -p values in single quotes; on Windows PowerShell, escape " as """ to avoid code:500 errors.
5. Time format: alarm_time must follow yyyy/MM/dd HH:mm:ss GMT+X:XX; for non-standard log time formats, pass time_pattern.
6. Summarize results: Use a summarization tool to condense command output before analysis; large raw outputs should not be analyzed directly.
7. Reflect after diagnosis: After completing the checks, reflect on whether the root cause is confirmed; if not, re-check for missed steps.
8. Read-only: All commands are read-only; repair steps are suggestions only and require user confirmation before execution.
9. Command failure handling: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.

References

Notes

• Security: This skill is read-only. It never exposes the LakeWatch or MRS Manager password; passwords are encrypted via --encrypt-password and stored in the corresponding config YAML. Repair steps are suggestions only.
• No KooCLI: This skill does not use hcloud; it calls the LakeWatch API through lakewatch_api_client.py or the MRS Manager REST API through manager_api_client.py. Do not mix in hcloud commands.
• Mode switching: Run check_api_mode.py (Step 0) to determine the mode. In manager mode use manager_api_client.py and the fault_layer_manager/ + scenarios_manager/ + propagation_manager.md knowledge base; in lakewatch mode use lakewatch_api_client.py and fault_layer/ + scenarios/ + propagation.md. Do not mix clients across modes.
• Command failure: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.
• Known limitations: The access_manager_get proxy only supports GET requests (PUT is not yet available on the Agent side); collect_alarm_log_data requires log_directory to be under /var/log/; some strategy_name values require extra env parameters; in manager mode browse_log requires the full log file path, start_log_search services must follow component:service:role, and get_alarms may return 500 on some Manager versions (fall back to Controller exe.log browsing).
• Cross-skill dependency: This skill references alarm diagnosis documents from the huawei-cloud-mrs-host-alarm-diagnose skill (../huawei-cloud-mrs-host-alarm-diagnose/alarms/<id>.md in lakewatch mode, alarm_manager/<id>.md in manager mode). See Prerequisites section 5 for the dependency declaration and handling rules. If the alarm skill is not installed, inform the user and proceed with the generic fault diagnosis flow.