🚀 Listed in the CNCF Landscape and eBPF Foundation Emerging Project. If you like this project, give it a star on GitHub! ❤️

News and Milestone

HUATUO Adds Huawei Ascend NPU Monitoring Support

2026-07-29 HUATUO Community AINPUAscendObservabilityFault Detection

In large-scale AI training, a single hardware failure can slow or interrupt an entire job. Minder, an NSDI 2025 paper, analyzed seven months of production data. It found that some training jobs used more than 10,000 GPUs and experienced an average of about two unexpected software or hardware failures per day. Hardware failures within a node accounted for 55.8% of all failures, and ECC errors accounted for 38.9%. Meta’s statistics from training the Llama 3 model family show that more than 66% of interruptions were related to hardware failures.

Hardware reliability is a key factor in the stable operation of large-scale AI clusters. To address this issue, PR #198, contributed by the Huawei Ascend team, has been merged into HUATUO. The change adds in-depth observability for Huawei Ascend NPUs. It uses DCMI, PCIe, and HCCN collectors to continuously monitor device health, compute workloads, memory reliability, and communication links. This monitoring helps AI clusters quickly detect and diagnose issues during training and inference.

HUATUO ecosystem overview

Support Overview

HUATUO is an open source, operating-system-level observability project initiated by Didi and incubated by the China Computer Federation (CCF). It provides kernel-level monitoring, abnormal event tracing, automated diagnostics, and continuous performance analysis for cloud-native general-purpose computing, AI computing, and infrastructure.

HUATUO hardware fault observability framework

HUATUO’s Ascend NPU support covers key fault domains across devices, memory, interconnects, and communications. It helps reduce monitoring tool fragmentation, inconsistent metric definitions, and disconnected alerting systems in heterogeneous environments. It provides a unified observability interface for AI training and inference in cloud-native environments.

Ascend support in HUATUO

Operational Benefits

This feature converts device data from DCMI, hccn_tool, and PCIe sysfs into a consistent set of Prometheus metrics. Hardware issues can then be monitored continuously, correlated across components, and reported through automated alerts.

Typical symptom Correlated signals Possible investigation focus
Reduced compute performance on one NPU Utilization, AI Core frequency, power, and temperature Overheating, frequency throttling, or uneven workload distribution
Intermittent job failures HBM single-bit and double-bit ECC errors and isolated page count HBM reliability issues and an increasing number of bad memory pages
Slower multi-NPU communication A mismatch between current and maximum PCIe speed or width Link negotiation downgrade, slot issues, or hardware faults
Collective communication timeouts HCCN link state, RoCE packet errors, retries, out-of-order packets, and CNPs Congestion, packet loss, or link faults
Cross-node performance fluctuations Optical module temperature, voltage, transmit and receive power, LOS, and SNR Degradation of an optical module or optical link

New Observability Capabilities

With the Huawei Ascend team’s contribution, HUATUO can collect key Ascend NPU metrics through DCMI, PCIe, and HCCN.

HUATUO Ascend NPU collection architecture

At startup, the DCMI collector dynamically loads /usr/local/dcmi/libdcmi.so, the default path on Linux. After loading the library, the collector registers the following core interfaces. Shared initialization, shutdown, and retry mechanisms manage the collection lifecycle:

dcmi_init
dcmi_get_device_health
dcmi_get_device_power_info
dcmi_get_device_temperature
dcmi_get_device_voltage
dcmi_get_device_utilization_rate
dcmi_get_device_frequency
dcmi_get_device_network_health
dcmi_get_device_hbm_info
dcmi_get_device_ecc_info
dcmi_get_device_pcie_info_v2
dcmi_get_device_logic_id
dcmi_get_device_phyid_from_logicid

After enabling the collectors, you can query the following data in HUATUO:

Collection path Monitoring scope Key metrics
DCMI Device status and compute workload Health, power, temperature, voltage, utilization, and clock frequency
DCMI HBM status and reliability Capacity, usage, temperature, bandwidth utilization, ECC errors, and isolated page count
PCIe Host-side links Maximum and current link speed, and maximum and current link width
HCCN Inter-NPU networking and RoCE communication Link state, transmit and receive rates, packet errors, retries, out-of-order packets, CNPs, and optical module information

Configuration and Deployment

Enable Data Collection

ascend_npu is included in the global BlackList by default. Before enabling the collector, remove it from the blacklist and enable the required collection capabilities:

# Remove "ascend_npu" from the global BlackList
BlackList = ["netdev_hw", "metax_gpu"]

[MetricCollector.AscendNPU]
    EnableDCMI = true
    EnablePCIe = true
    EnableHCCN = true

You can configure EnableDCMI, EnablePCIe, and EnableHCCN independently based on the device model, driver capabilities, and deployment environment.

Container Deployment

When running HUATUO in a container, mount the Ascend driver and DCMI library from the host into the container. The following command is an example; adjust the paths for your installation:

docker run --privileged --pid=host --cgroupns=host --network=host \
        -v /sys:/sys \
        -v /proc:/proc \
        -v /run:/run \
        -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
        -v /usr/local/dcmi:/usr/local/dcmi:ro \
        huatuo/huatuo-bamai:latest

Conclusion

Huawei Ascend NPU support extends HUATUO’s observability coverage for AI infrastructure. It also provides a unified interface for fault analysis in heterogeneous computing environments.

Moving from reactive troubleshooting to full-lifecycle monitoring, continuous alerting, and root cause analysis is an important step toward improving AI cluster reliability. HUATUO will continue to expand its metric documentation, recommended alert thresholds, and scenario-based troubleshooting procedures. This work will help teams deploy in-depth observability capabilities in production environments.

Tags