Tutorials

• 1: Compile
• 2: Deploy
• 2.1: Daemonset
• 2.2: Docker
• 2.3: Systemd
• 3: Add Cases
• 3.1: Add Custom Collection
• 3.2: Add Autotracing
• 3.3: Add Event
• 3.4: Add Metrics

1 - Compile

1. Build with the Official Image

To isolate the developer’s local environment and simplify the build process, we provide a containerized build method. You can directly use docker build to produce an image containing the core collector huatuo-bamai, BPF objects, tools, and more. Run the following in the project root directory:

docker build --network host -t huatuo/huatuo-bamai:latest .

2. Build a Custom Image

Dockerfile.dev:

FROM golang:1.23.0-alpine AS base
# Speed up Alpine package installation if needed
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk add --no-cache \
                make \
                clang15 \
                libbpf-dev \
                bpftool \
                curl \
                git

ENV PATH=$PATH:/usr/lib/llvm15/bin

# build huatuo components
FROM base AS build
ARG BUILD_PATH=${BUILD_PATH:-/go/huatuo-bamai}
ARG RUN_PATH=${RUN_PATH:-/home/huatuo-bamai}
WORKDIR ${BUILD_PATH}

2.1 Build the Dev Image

docker build --network host -t huatuo/huatuo-bamai-dev:latest -f ./Dockerfile.dev .

2.2 Run the Dev Container

docker run -it --privileged --cgroupns=host --network=host \
  -v /path/to/huatuo:/go/huatuo-bamai \
  huatuo/huatuo-bamai-dev:latest sh

2.3 Compile Inside the Container

Run:

make

Once the build completes, all artifacts are generated under ./_output.

3. Build on a Physical Machine or VM

The collector depends on the following tools. Install them based on your local environment:

• make
• git
• clang15
• libbpf
• bpftool
• curl

Due to significant differences across local environments, build issues may occur.
To avoid environment inconsistencies and simplify troubleshooting, we strongly recommend using the Docker build approach whenever possible.

2 - Deploy

The HUATUO collector huatuo-bamai runs on physical machines or VMs.
We provide both binary packages and Docker images, and you can deploy them in any custom way, such as:

• systemd and DaemonSet deployments are recommended for production.
• Docker / Compose is suitable for development and quick validation scenarios.

Binary Download

The latest binary package provided is v2.1.0; the master branch is for reference only.

• v2.1.0: https://github.com/ccfos/huatuo/releases/tag/v2.1.0

Mirror Download

Docker images are stored on Docker Hub by default (https://hub.docker.com/u/huatuo).

2.1 - Daemonset

HUATUO provides the simplest DaemonSet deployment option to minimize setup complexity.

Deploying the HUATUO collector via DaemonSet involves the following steps:

1. Download the Collector Configuration File

curl -L -o huatuo-bamai.conf https://github.com/ccfos/huatuo/raw/main/huatuo-bamai.conf

Modify this configuration file according to your environment, such as kubelet connection settings and Elasticsearch settings.

2. Create a ConfigMap

kubectl create configmap huatuo-bamai-config --from-file=./huatuo-bamai.conf

3. Deploy the Collector

kubectl apply -f huatuo-daemonset.minimal.yaml

Contents of huatuo-daemonset.minimal.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: huatuo
  namespace: default
  labels:
    app: huatuo
spec:
  selector:
    matchLabels:
      app: huatuo
  template:
    metadata:
      labels:
        app: huatuo
    spec:
      containers:
      - name: huatuo
        image: docker.io/huatuo/huatuo-bamai:latest
        resources:
          limits:
            cpu: '1'
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 512Mi
        securityContext:
          privileged: true
        volumeMounts:
        - name: proc
          mountPath: /proc
        - name: sys
          mountPath: /sys
        - name: run
          mountPath: /run
        - name: var
          mountPath: /var
        - name: etc
          mountPath: /etc
        - name: huatuo-local
          mountPath: /home/huatuo-bamai/huatuo-local
        - name: huatuo-bamai-config-volume
          mountPath: /home/huatuo-bamai/conf/huatuo-bamai.conf
          subPath: huatuo-bamai.conf
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: run
        hostPath:
          path: /run
      - name: var
        hostPath:
          path: /var
      - name: etc
        hostPath:
          path: /etc
      - name: huatuo-local
        hostPath:
          path: /var/log/huatuo/huatuo-local
          type: DirectoryOrCreate
      - name: huatuo-bamai-config-volume
        configMap:
          name: huatuo-bamai-config
      hostNetwork: true
      hostPID: true

2.2 - Docker

Run Only the Collector

Start the Container

docker run --privileged --cgroupns=host --network=host -v /sys:/sys -v /proc:/proc -v /run:/run huatuo/huatuo-bamai:latest

⚠️ This uses the default configuration file inside the container. The internal default configuration does not connect to Elasticsearch. For a complete setup, mount your own huatuo-bamai.conf using -v, and update the config according to your environment (kubelet access, Elasticsearch settings, local log storage path, etc.).

Deploy All Components (Docker Compose)

For local development and validation, using Docker Compose is the most convenient approach.
You can quickly launch a full environment containing the collector, Elasticsearch, Prometheus, Grafana, and other components.

docker compose --project-directory ./build/docker up

It is recommended to install Docker Compose using the plugin method: https://docs.docker.com/compose/install/linux/

2.3 - Systemd

Install by RPM

OpenCloudOS currently provides the v2.1.0 RPM package; the master is for reference only.

Tencent OpenCloudOS provides an official HUATUO package:
https://mirrors.opencloudos.tech/epol/9/Everything/x86_64/os/Packages/huatuo-bamai-2.1.0-2.oc9.x86_64.rpm
This allows HUATUO to be quickly installed and enabled on OpenCloudOS.

• x86_64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/x86_64/os/Packages/huatuo-bamai-2.1.0-2.oc9.x86_64.rpm

• arm64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/aarch64/os/Packages/huatuo-bamai-2.1.0-2.oc9.aarch64.rpm

• Install HUATUO on OC8

sudo rpm -ivh huatuo-bamai*.rpm

Other RPM-based operating systems can install HUATUO the same way.
As usual, you must update the config file according to your environment (e.g., kubelet connection, Elasticsearch settings).

Full OpenCloudOS installation guide:
https://mp.weixin.qq.com/s/Gmst4_FsbXUIhuJw1BXNnQ

Install by Binary Package

The latest binary package provided is v2.1.0; the master branch is for reference only.

You can also download the binary package and configure/manage it manually.
Again, update the configuration file based on your actual environment (kubelet connection, Elasticsearch settings, etc.).

• v2.1.0: https://github.com/ccfos/huatuo/releases/tag/v2.1.0

3 - Add Cases

3.1 - Add Custom Collection

The framework provides convenient APIs, including module startup, data storage, container information, BPF-related (load, attach, read, detach, unload), etc. You can implement custom collection logic and flexibly choose the appropriate collection mode and storage method.

Tracing Type

Based on your scenarios, you can implement the ITracingEvent interface in the core/autotracing or core/events directory to complete tracing-type collection.

// ITracingEvent represents a tracing/event
type ITracingEvent interface {
    Start(ctx context.Context) error
}

example:

type exampleTracing struct{}

// Register callback
func init() {
    tracing.RegisterEventTracing("example", newExample)
}

// Create tracing
func newExample() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleTracing{},
        Internal:    10, // Interval for enable tracing again (in seconds)
        Flag:        tracing.FlagTracing, // mark as tracing type
    }, nil
}

// Implement ITracingEvent
func (t *exampleTracing) Start(ctx context.Context) error {
    // do something
    ...

    // Save data to ES and local file
    storage.Save("example", ccontainerID, time.Now(), tracerData)
}

// Implement Collector interface for Prometheus format output (optional)
func (c *exampleTracing) Update() ([]*metric.Data, error) {
    // from tracerData to prometheus.Metric 
    ...

    return data, nil
}

Metric Type

Implement the Collector interface in the path core/metrics to complete metric-type collection.

type Collector interface {
    // Get new metrics and expose them via prometheus registry.
    Update() ([]*Data, error)
}

example:

type exampleMetric struct{}

// Register callback
func init() {
    tracing.RegisterEventTracing("example", newExample)
}

// Create Metric
func newExample() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &filenrCollector{
            metric: []*metric.Data{
                metric.NewGaugeData("name1", 0, "description of example_name1", nil),
                metric.NewGaugeData("name2", 0, "description of example_name2", nil),                
            },
        },
        Flag: tracing.FlagMetric, // mark as Metric type
    }, nil
}

// Implement Collector interface for Prometheus format output
func (c *exampleMetric) Update() ([]*metric.Data, error) {
    // do something
    ...

    return data, nil
}

The path core of the project includes multiple useful examples of the three collection modules, covering BPF code, map data interaction, container information, and more. For further details, refer to the corresponding code implementations.

3.2 - Add Autotracing

Overview

• Type：Exception event-driven（tracing/autotracing）
• Function：Automatically tracks system abnormal states and triggers context information capture when exceptions occur
• Characteristics：
  • When system abnormalities occur, autotracing automatically triggers and captures relevant context information
  • Event data is stored locally in real-time and also sent to remote ES, while you can also generate Prometheus metrics for observation
  • Suitable for significant performance overhead， such as triggering capture when detecting metrics rising above certain thresholds or rising too rapidly
• Already Integrated：abnormal usage tracking (cpu idle), D-state tracking (dload), container internal/external contention (waitrate), sudden memory allocation (memburst), disk abnormal tracking (iotracer)

How to Add Autotracing

AutoTracing only requires implementing the ITracingEvent interface and completing registration to add events to the system.

There is no implementation difference between AutoTracing and Event in the framework; they are only differentiated based on practical application scenarios.

// ITracingEvent represents a autotracing or event
type ITracingEvent interface {
    Start(ctx context.Context) error
}

1. Create Structure

type exampleTracing struct{}

2. Register Callback Function

func init() {
    tracing.RegisterEventTracing("example", newExample)
}

func newExample() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleTracing{},
        Internal:    10, // Interval in seconds before re-enabling tracing
        Flag:        tracing.FlagTracing, // Mark as tracing type; | tracing.FlagMetric (optional)
    }, nil
}

3. Implement ITracingEvent

func (t *exampleTracing) Start(ctx context.Context) error {
    // detect your care about 
    ...

    // Store data to ES and locally
    storage.Save("example", ccontainerID, time.Now(), tracerData)
}

Additionally, you can optionally implement the Collector interface to output in Prometheus format:

func (c *exampleTracing) Update() ([]*metric.Data, error) {
    // from tracerData to prometheus.Metric 
    ...

    return data, nil
}

The core/autotracing directory in the project has integrated various practical autotracing 示examples, along with rich underlying interfaces provided by the framework, including BPF program and map data interaction, container information, etc. For more details, refer to the corresponding code implementations.

3.3 - Add Event

Overview

• Type: Exception event-driven（tracing/event）
• Function：Continuously runs in the system and captures context information when preset thresholds are reached
• Characteristics:
  • Unlike autotracing, event runs continuously rather than being triggered only when exceptions occur.
  • Event data is stored locally in real-time and also sent to remote ES. You can also generate Prometheus metrics for observation.
  • Suitable for continuous monitoring and real-time analysis, enabling timely detection of abnormal behaviors in the system. The performance impact of event type collection is negligible.
• Already Integrated: Soft interrupt abnormalities（softirq）、abnormal memory allocation（oom）、soft lockups（softlockup）、D-state processes（hungtask）、memory reclaim（memreclaim）、abnormal packet loss（dropwatch）、network inbound latency (net_rx_latency), etc.

How to Add Event Metrics

Simply implement the ITracingEvent interface and complete registration to add events to the system.

There is no implementation difference between AutoTracing and Event in the framework; they are only differentiated based on practical application scenarios.

// ITracingEvent represents a tracing/event
type ITracingEvent interface {
    Start(ctx context.Context) error
}

1. Create Event Structure

type exampleTracing struct{}

2. Register Callback Function

func init() {
    tracing.RegisterEventTracing("example", newExample)
}

func newExample() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleTracing{},
        Internal:    10, // Interval in seconds before re-enabling tracing
        Flag:        tracing.FlagTracing, // Mark as tracing type; | tracing.FlagMetric (optional)
    }, nil
}

3. Implement the ITracingEvent Interface

func (t *exampleTracing) Start(ctx context.Context) error {
    // do something
    ...

    // Store data to ES and locally
    storage.Save("example", ccontainerID, time.Now(), tracerData)
}

Additionally, you can optionally implement the Collector interface to output in Prometheus format:

func (c *exampleTracing) Update() ([]*metric.Data, error) {
    // from tracerData to prometheus.Metric 
    ...

    return data, nil
}

The core/events directory in the project has integrated various practical events examples, along with rich underlying interfaces provided by the framework, including BPF program and map data interaction, container information, etc. For more details, refer to the corresponding code implementations.

3.4 - Add Metrics

Overview

The Metrics type is used to collect system performance and other indicator data. It can output in Prometheus format, serving as a data provider through the /metrics (curl localhost:<port>/metrics) .

• Type：Metrics collection

• Function：Collects performance metrics from various subsystems

• Characteristics：

  • Metrics are primarily used to collect system performance metrics such as CPU usage, memory usage, network statistics, etc. They are suitable for monitoring system performance and support real-time analysis and long-term trend observation.
  • Metrics can come from regular procfs/sysfs collection or be generated from tracing types (autotracing, event).
  • Outputs in Prometheus format for seamless integration into the Prometheus observability ecosystem.

• Already Integrated：

  • cpu (sys, usr, util, load, nr_running…)
  • memory（vmstat, memory_stat, directreclaim, asyncreclaim…）
  • IO (d2c, q2c, freeze, flush…)
  • Network（arp, socket mem, qdisc, netstat, netdev, socketstat…）

How to Add Statistical Metrics

Simply implement the Collector interface and complete registration to add metrics to the system.

type Collector interface {
    // Get new metrics and expose them via prometheus registry.
    Update() ([]*Data, error)
}

1. Create a Structure

Create a structure that implements the Collector interface in the core/metrics directory:

type exampleMetric struct{
}

2. Register Callback Function

func init() {
    tracing.RegisterEventTracing("example", newExample)
}

func newExample() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleMetric{},
        Flag: tracing.FlagMetric, // Mark as Metric type
    }, nil
}

3. Implement the Update Method

func (c *exampleMetric) Update() ([]*metric.Data, error) {
    // do something
    ...
	return []*metric.Data{
		metric.NewGaugeData("example", value, "description of example", nil),
	}, nil

}

The core/metrics directory in the project has integrated various practical Metrics examples, along with rich underlying interfaces provided by the framework, including BPF program and map data interaction, container information, etc. For more details, refer to the corresponding code implementations.