Overview

Requests are timing out and TCP retransmission counters are climbing. Which connections are affected, and which containers do they belong to? Node-level metrics alone rarely tell you where to look next. HUATUO tcpshark traces retransmissions in the kernel, identifies the connection phase and owning container, and correlates retransmissions with local packet drops through dropwatch. This walkthrough shows how to turn a retransmission alert into a focused investigation.

Why tcpshark?

tcpshark instruments the following kernel paths:

• tcp/tcp_retransmit_skb: SKB retransmissions from the regular retransmission queue.
• tcp/tcp_retransmit_synack: Server-side SYN-ACK retransmissions during connection establishment.
• tcp_send_loss_probe: Tail Loss Probes (TLPs).

Each event includes the connection four-tuple, TCP state, sequence number, congestion control state, retransmission counters, and socket metadata. Userspace processing adds the phase and tcp_reason fields. Use these fields to narrow your investigation:

1. Map the host’s network namespaces to containers.
2. Identify the affected connections and traffic direction within each container.
3. Determine whether retransmissions occur during connection establishment, data transfer, or teardown.

Using tcpshark

Set --mode retransmit and point --bpf-path to your copy of tcp_retransmit.o. This example captures events for 60 seconds, filters regular retransmissions involving 10.0.0.1 and ports 443–444, enables TLP tracing, and outputs newline-delimited JSON (NDJSON):

1
2
3
4
5
6

sudo ./tcpshark --mode retransmit \
  --enable-tlp \
  --bpf-path bpf/tcp_retransmit.o \
  --filter "tcp and host 10.0.0.1 and portrange 443-444" \
  --duration 60 \
  --output json

Use these options to control what tcpshark captures and how many events it emits:

• --duration 60 stops collection after 60 seconds. Choose a duration that covers the incident window, or set it to 0 to run until tcpshark receives a termination signal.
• --enable-tlp is disabled by default. Enabling it also attaches to tcp_send_loss_probe and emits TLP events.
• --filter accepts tcpdump-style filter expressions.
• --max-events-per-second 100 caps the event rate at 100 events per second in the BPF program.

When run as a standalone tool, tcpshark does not include container_id in its output. huatuo-bamai resolves container ownership by checking the socket’s memory cgroup first, then the network namespace cookie, and finally the network namespace inode number (inum). In the examples below, the retransmitting connections belong to the tcpshark-container-demo container.

Troubleshooting TCP Retransmissions

An event labeled tcp_reason="RTO" does not, by itself, prove network packet loss. Start by identifying the affected connection and its phase, then examine the retransmission path, owning container, and any evidence of packet drops.

Step 1: Identify the Connection

Consider the first event in the first screenshot:

1. 10.0.0.2:33342 > 10.0.0.1:443: The tcp_saddr, tcp_sport, tcp_daddr, and tcp_dport fields identify the connection and traffic direction.
2. phase="connect", tcp_state="SYN_SENT", and tcp_flags="SYN" indicate that connection establishment is in progress.
3. tcp_reason="RTO": In the first two events, icsk_retransmits increases from 1 to 2, while tcp_seq stays the same. The endpoint is retrying the same SYN as connection establishment stalls.
4. net_namespace_inum=4026532512: This identifies the network namespace associated with the event.

Step 2: Identify the Connection Phase and Retransmission Path

phase classifies retransmissions as connect, data, or close. tcp_reason further classifies them using the event type, sk_state, ca_state, and reordering counters.

Interpret these classifications in context:

• RTO often has a significant impact on latency. Look for recurring RTOs, repeated retransmissions with the same sequence number, and events that coincide with application latency spikes.
• fast_retransmit means the socket was in the Recovery state. It does not reveal the full sequence of duplicate ACKs or SACK/RACK decisions that led to recovery.
• unknown means the snapshot lacks enough information to classify the event. The event is still valid and may be useful to the investigation.

To display only RTO events, filter the JSON output:

1
2
3
4
5
6
7

sudo ./tcpshark --mode retransmit \
  --enable-tlp \
  --bpf-path bpf/tcp_retransmit.o \
  --filter "tcp and host 10.0.0.1 and portrange 443-444" \
  --duration 60 \
  --output json |
  jq --unbuffered -cC 'select(.tcp_reason == "RTO")'

Step 3: Look for Persistent Patterns

These patterns warrant closer investigation:

• icsk_retransmits keeps increasing.
• Multiple affected connections share a destination address, port, or workload.
• RTOs coincide with increased application latency, timeouts, or reduced throughput.

Group events to see which retransmission patterns dominate. The following command saves the NDJSON output and counts events by connection phase, retransmission reason, and event type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

./_output/bin/tcpshark \
  --mode retransmit \
  --enable-tlp \
  --bpf-path ./_output/bpf/tcp_retransmit.o \
  --filter 'tcp and portrange 19991-19998' \
  --duration 90 \
  --output json |
  tee tcpshark.ndjson |
  jq -r \
    '[.phase, .tcp_reason, .event_type] | @tsv' |
  sort |
  uniq -c |
  sort -nr

Step 4: Enable Packet Drop Detection

Add --with-dropwatch to trace local packet drops and automatically correlate them with retransmissions. This helps establish whether packets are being dropped in the local network stack.

1
2
3
4
5
6

./_output/bin/tcpshark \
  --mode retransmit \
  --with-dropwatch \
  --bpf-path-dir ./_output/bpf \
  --filter 'tcp and port 19997' \
  --output json | jq --unbuffered -cC 'select(.drop_location == "host_software")'

Continuous Monitoring with huatuo-bamai

For ongoing monitoring, configure huatuo-bamai to trace TCP retransmissions on physical hosts and in containers. The following configuration sets the filter and event rate limit, and enables TLP tracing and packet drop correlation:

1
2
3
4
5

[EventTracing.TCPRetransmit]
Filter = "tcp and port 443"
EnableTLP = true
MaxEventsPerSecond = 100
EnableDropwatchCorrelation = true

The global BlackList disables the tcp_retransmit tracer by default. To enable it, remove its entry and restart huatuo-bamai. Correlation uses tcpshark’s built-in dropwatch source, so the standalone dropwatch tracer can remain in the BlackList. Enable that tracer only if you also need to record raw packet drop events separately.

For full details on options, fields, and classification rules, see the tcpshark documentation.

Project repository: github.com/ccfos/huatuo