The Container Recovered, but the Problem Remains

A service’s memory usage keeps climbing until the OOM killer terminates it. After the container restarts, traffic returns to normal and the memory graph drops. The engineer investigating the alert has the container name, the timestamp, and the peak memory usage. The missing detail is what was in memory at that moment.

In a service that handles requests, an expanding cache, a queue backing up behind a slow downstream service, and unintentionally retained objects can all produce similar graphs. The first response is often to raise the memory limit or wait for the problem to recur and then capture a heap dump.

If the limit is simply below the normal working set, raising it can solve the problem. If memory is leaking, adding capacity usually just buys time. Heap dumps can also be triggered automatically, but this requires advance configuration and accounting for the pauses, CPU usage, and storage overhead of generating them.

Knowing which process the OOM killer ultimately selected does not tell you which object types or allocation paths caused the growth. When a container contains multiple processes, connecting container memory usage to one process’s runtime memory also requires selecting a target and interpreting the measurements.

HUATUO’s memsnapshot aims to collect evidence before the process exits, while memory pressure is building: after receiving a memory notification, it rechecks usage and, if the threshold is met, reads the running process to save a memory summary that can guide the investigation. The saved evidence remains available even if the process later restarts. Notification timing, the rate of memory growth, and the collection cooldown all affect whether collection happens in time, so a snapshot before every OOM event is not guaranteed.

Select a Container, Then a Process

Collection starts with container memory usage. HUATUO obtains the memory cgroup paths of running containers and watches for memory notifications. It rereads usage and limits after receiving a notification, registering a watch, or detecting a change to the hard limit. A container becomes eligible for collection only when its usage ratio reaches the configured threshold, which defaults to 90%. Containers without a memory limit are excluded from this ratio check.

This threshold determines whether to collect when usage is checked. It does not mean that the kernel will send a notification at that level.

Select a container, then a process. When several containers trigger at once, the scheduler consolidates the candidates, rechecks their usage, and ranks them by usage ratio. Each batch attempts collection for only the highest-ranked candidate. Collection runs serially and is subject to a cooldown shared across the node, so a single trigger does not scan every container on the node.

Once a container is selected, HUATUO chooses a process from the direct members of its cgroup. The selection follows the scoring approach used by memcg OOM: it adds RSS, swap usage, and page table memory, then adds an oom_score_adj contribution scaled by the container’s memory limit. It selects the candidate with the highest score, excluding processes with oom_score_adj = -1000.

This score selects a diagnostic target; it does not guarantee that the kernel will eventually kill the same process. Before and after collection, HUATUO also checks the container instance, the identity of the cgroup directory, the PID, and the process start time. If the target exits or its identity changes, collection is canceled or the result is discarded.

What Evidence Does a Snapshot Preserve?

After selecting a target process, autotracing passes its PID and start time to memsnapshot to begin collecting process memory information. memsnapshot reads a process summary, identifies the language runtime, and invokes the corresponding collector, returning results in a common format.

Start with the process’s memory usage. memsnapshot reads RSS, resident anonymous memory, resident file mappings, resident shared memory, swap usage, and page table memory from /proc/PID/status to build the process_memory summary. Values are in bytes. Fields that cannot be read are omitted; their absence does not mean zero. This summary does not depend on support for a particular language.

The following example shows the process_memory field in a result. All values are illustrative and expressed in bytes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "process_memory": {
    "status": "complete",
    "virtual_bytes": 6442450944,
    "rss_bytes": 1610612736,
    "rss_anon_bytes": 1509949440,
    "rss_file_bytes": 83886080,
    "rss_shmem_bytes": 16777216,
    "swap_bytes": 33554432,
    "page_table_bytes": 4194304
  }
}

Evidence from the language runtime. memsnapshot identifies the target language and runtime layout, then uses process_vm_readv to read the required memory ranges into the HUATUO process. The corresponding collector parses and aggregates the data. The application does not need to generate a heap dump, and HUATUO does not copy the process’s entire memory. The next section explains what each language collector reads.

The design aims to minimize extra work in the application process. Among open-source tools, Memray can track Python and native allocations, while async-profiler can sample Java heap allocations. These capabilities observe allocation behavior over time. memsnapshot reads existing runtime information under memory pressure to provide evidence from a single collection. The tools differ in what they measure and how the results are used.

During collection, HUATUO does not inject collection code into the application or explicitly pause the process. HUATUO handles most of the CPU work and temporary buffer allocation for parsing and aggregation in its own process. It reuses existing profiles, limits scan ranges, runs collection serially, and applies cooldown intervals to control additional CPU and memory use.

Reading memory from outside the process still consumes node CPU and memory bandwidth. Accessing nonresident pages can also trigger page faults and I/O. Runtime structures may change during collection, so the result is not a consistent heap snapshot. Language-specific collection has a default budget of 2 seconds, enforced through cooperative checks; this is not a hard limit on the entire operation. These choices explain how overhead is controlled. Whether this approach uses fewer resources than other tools, and how much it affects application latency, still requires measurement under the same workload.

Each collector places its results in snapshot, including the runtime version, collection status, elapsed time, and top entries. Reasons for incomplete or unavailable results are recorded as well. Collection is bounded by a time budget and an output size limit. Even if the language is unsupported or collection fails, a process summary that has already been read may still be retained.

memsnapshot does not save results itself. After the result returns to the autotracing module, the module rechecks process identity and container ownership. It then associates the container’s usage and limit, the target process, and the collection result, and writes them according to the storage configuration. A saved record therefore provides both the memory pressure at the time and diagnostic evidence for the corresponding process.

Other events can reuse the same collection capability. memsnapshot does not depend on the Before-OOM trigger conditions, nor does it select containers, schedule tasks, or write to storage. Once another event has selected a target process, it can pass the PID, start time, collection budget, and maximum number of result entries to collector.Capture to obtain the same process summary and language snapshot. The event is responsible for its trigger policy, any required checks of target ownership, and saving the result, without having to reimplement runtime parsing.

Investigating Memory Through Allocation Stacks and Object Types

The implementation and configuration details below are based on HUATUO 36175d6e. Collectors depend on specific runtime versions, symbols, and memory layouts. Support for a language does not imply support for every version or build configuration; check the actual collection results for the target runtime when deploying.

Go: Follow allocation call stacks back to application code. The collector uses information such as runtime.mbuckets and runtime.MemProfileRate to locate the runtime’s existing memory profile. It reads allocation and free records along with their call stacks, applies the sampling rate to estimate in-use memory and object counts, then aggregates by allocation stack and resolves symbols.

The call stack records the call chain at the time of allocation. Even if the top frame is a standard library function such as JSON encoding, following the callers can lead back to application code. Results depend on runtime sampling and when the statistics are updated. If the application has disabled memory sampling, the collector reports that collection is unavailable; it does not enable sampling on the application’s behalf.

The following excerpt shows the language result within tracer_data. Names and values are illustrative, common fields are omitted, and only some frames of the Go call stack are shown:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
  "language": "go",
  "snapshot": {
    "status": "complete",
    "entries": [
      {
        "kind": "allocation_site",
        "name": "encoding/json.mapEncoder.encode",
        "bytes": 1572864,
        "objects": 8192,
        "average_bytes": 192,
        "stack": [
          "encoding/json.mapEncoder.encode",
          "encoding/json.(*encodeState).reflectValue",
          "encoding/json.(*encodeState).marshal",
          "encoding/json.(*Encoder).Encode",
          "main.serve.func2"
        ]
      }
    ]
  }
}

This record attributes an estimated 1.5 MiB of in-use memory to one allocation stack. stack proceeds from the allocation site toward its callers. Following the JSON encoding path leads to the application entry point main.serve.func2, where you can inspect how the relevant objects are used and released.

Java: Identify object types in G1 heap regions. The current implementation targets HotSpot G1 and does not yet support JVMs with compact object headers enabled. The collector obtains field offsets and structure layouts from VM metadata exported by libjvm.so, locates G1 regions, and uses Klass information in object headers to identify class names and object sizes.

For regular regions, it samples windows of memory, identifies valid objects, and estimates the distribution of object types. For Humongous objects spanning multiple regions, it primarily reads the initial object header and class metadata without copying the entire object’s contents. Results summarize object counts and shallow sizes by class; they do not include the full graph of referenced objects.

The following excerpt shows the corresponding language result. Names and values are illustrative:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
  "language": "java",
  "snapshot": {
    "status": "partial",
    "reason": "Illustrative: regular regions are sampled, so type statistics include estimates",
    "entries": [
      {
        "kind": "object_type",
        "name": "com.example.order.PendingOrder",
        "bytes": 201326592,
        "objects": 1048576,
        "average_bytes": 192
      }
    ]
  }
}

In this record, the estimated shallow sizes of PendingOrder objects total 192 MiB, suggesting that the pending queue and its processing rate are worth investigating. This total excludes other objects referenced by these objects. The partial status is a reminder to interpret the result in the context of the sampling scope.

Python: Summarize objects through GC tracking structures. The collector locates _PyRuntime through symbols in the executable or libpython, finds the interpreter’s GC structures for the CPython version, and follows the linked lists to read objects. It identifies types through ob_type and aggregates counts and sizes.

The size estimate includes the objects themselves, some management overhead, and some buffers directly owned by lists and dictionaries. It does not recursively count referenced objects. Application-defined types may also appear in the result, helping investigate accumulated tasks and request contexts. Objects not tracked by GC and native memory held by extension modules are not fully covered, so the result should not be treated as an inventory of the entire Python heap.

The following excerpt shows the corresponding language result. Names and values are illustrative:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "language": "python",
  "snapshot": {
    "status": "complete",
    "entries": [
      {
        "kind": "object_type",
        "name": "dict",
        "bytes": 67108864,
        "objects": 262144,
        "average_bytes": 256
      }
    ]
  }
}

This record reports an estimated total of 64 MiB for the dict objects found in this scan. complete means only that collection finished within this collector’s scope. It does not mean that all Python objects were covered, nor does it identify which application code owns these dictionaries.

Container memory usage, RSS for a single process, and byte counts in language entries use different accounting scopes and cannot be directly added or subtracted. Processes written in C/C++ or using unsupported runtimes may still yield a process summary, but that does not imply support for native heap analysis. Source code for the language providers

Using Memory Snapshots on a Node

memory_threshold_snapshot is enabled by default. First, confirm that HUATUO can obtain container metadata, access the host’s /proc and memory cgroups, and has permission to load BPF programs and read the target process’s memory. The target container must have a finite memory limit and a dedicated memory cgroup.

Adjust collection parameters as needed. By default, HUATUO reads huatuo-bamai.conf from its working directory at startup. If --config specifies a configuration file, edit that file. All four settings below show their default values. If the configuration section already exists, edit it instead of adding a duplicate:

1
2
3
4
5

[AutoTracing.MemoryThresholdSnapshot]
    ThresholdPercent = 90
    IntervalTracing = 300
    RunTracingToolTimeout = 2
    MaxMemoryObjectEntries = 10

After changing the configuration, restart HUATUO and check that the global BlackList does not contain memory_threshold_snapshot. When upgrading, use the current AutoTracing.MemoryThresholdSnapshot configuration rather than older fields such as EventTracing.BeforeOOMMemsnap.

Verify that collection can trigger before waiting for a production incident. After startup, check the logs for watch initialization and registration, then use controlled memory growth to confirm that a collection record is produced. On cgroup v2, also check memory.high: HUATUO does not set it automatically, and configuring a 90% threshold alone does not guarantee a notification at 90%. Before adjusting this level, assess its effects on memory reclaim, throttling, and application latency.

Find results in the configured storage backend. Snapshots use the existing [Storage] configuration. With LocalFile storage, look for the memory_threshold_snapshot file in the Path directory configured under [Storage.LocalFile]. For example, after setting Path to /var/log/huatuo/snapshots, you can read it directly:

1

less /var/log/huatuo/snapshots/memory_threshold_snapshot

When inspecting a record, first check the target process, snapshot.status, reason, and runtime_version to determine whether the result is usable. Then examine the allocation stacks or object types in entries. output_truncated indicates that the output was truncated. If language-specific collection fails, process_memory may still provide a process summary. victim_pid identifies the collection target; it does not mean the process has been killed by the OOM killer.

To disable collection, add memory_threshold_snapshot to the global BlackList, preserving any existing entries, then restart HUATUO.

After an OOM event, the hardest evidence to recover is what was in memory before the process exited. The allocation stacks, object types, and process summaries preserved here can help identify which code, queue, or cache to investigate next. They may not reveal the root cause directly, but they give us more evidence to work with.