English | 简体中文
This guide explains how to profile RustNet to identify performance bottlenecks.
The easiest way to profile CPU usage on Linux:
# 1. Install flamegraph tools
cargo install flamegraph
# 2. Build a release binary with debug symbols
# IMPORTANT: Debug symbols are required for meaningful flamegraphs!
CARGO_PROFILE_RELEASE_DEBUG=true cargo build --release
# Or add this to Cargo.toml temporarily:
# [profile.release]
# debug = true
# 3. Run with profiling (requires sudo for perf)
# Note: Use full path to flamegraph since sudo doesn't have your user's PATH
# IMPORTANT: Use -- before the command to profile
sudo -E ~/.cargo/bin/flamegraph -- ./target/release/rustnet
# Or specify interface and other args after the binary
sudo -E ~/.cargo/bin/flamegraph -- ./target/release/rustnet -i eth0
# Alternatively, preserve PATH for cleaner commands:
sudo env "PATH=$PATH" flamegraph -- ./target/release/rustnet
# 4. Open the generated flamegraph.svg in a browser
firefox flamegraph.svgIf you prefer to use perf directly:
# Build with debug symbols
CARGO_PROFILE_RELEASE_DEBUG=true cargo build --release
# Record performance data (run for 30-60 seconds, then Ctrl+C to stop)
sudo perf record -F 99 -g ./target/release/rustnet -i eth0
# Generate flamegraph (requires FlameGraph scripts)
# Install from: https://github.com/brendangregg/FlameGraph
perf script | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg
# Or view in perf's TUI
sudo perf reportIf RustNet is already running:
# Find the PID
ps aux | grep rustnet
# Profile the running process for 60 seconds
sudo -E ~/.cargo/bin/flamegraph -p <PID> --output rustnet-live.svg
# Or with perf directly
sudo perf record -F 99 -g -p <PID> sleep 60
sudo perf reportLook for:
- Wide bars at the bottom: Functions that consume a lot of total CPU time
- Tall stacks: Deep call chains (potential optimization targets)
- Hot spots: Functions with many samples (bright colors in some viewers)
Common hot spots:
packet_parser::parse_packet: Normal - this is the core packet processingDashMap::iteroriter_mut: If this is a large portion, consider reducing iteration frequencyclone: If excessive, reduce unnecessary cloning- System calls (
read,write,ioctl): Filesystem or network I/O overhead
Microbenchmarks for core operations live in benches/. Run them with:
| Benchmark | Command |
|---|---|
| Packet parsing | cargo bench --bench packet_parsing |
| DPI overhead and mixed parsing/tracking | cargo bench --bench dpi_overhead |
| Connection merge | cargo bench --bench connection_merge |
| Snapshot creation | cargo bench --bench snapshot |
| Rate tracking | cargo bench --bench rate_tracker |
| Tracker ingestion | cargo bench --bench tracker_ingest |
| All benchmarks | cargo bench |
| Struct sizes | cargo test --lib struct_sizes -- --nocapture |
Criterion produces HTML reports in target/criterion/ with statistical comparison between runs.
The dpi_overhead benchmark includes deterministic Ethernet/TCP fixtures for ordinary and database traffic, plus a mixed parsing/tracking workload. It needs no capture file. To also replay a controlled capture using its recorded link type, set RUSTNET_BENCH_PCAP=/path/to/capture.pcap; an unreadable, empty, or unsupported capture fails the benchmark.
For before/after comparisons, use the same benchmark source, capture, compiler, build settings, and machine on both revisions. Build both versions before timing them, then alternate repeated runs without other heavy jobs. Database recognition adds work that the base revision did not perform, so report that cost separately from ordinary traffic. These measurements cover parsing and tracking CPU time, not live capture loss or UI responsiveness.
The ignored release tests measure snapshot building, wire projection, serialization, and payload size, checking cached output against full serialization. The matrix varies active/history counts and metadata size; PCAP replay samples every 500 ms of trace time with five-second cleanup, without sleeping.
RUSTNET_PROFILE_CSV=/tmp/matrix.csv cargo test --locked --release --lib \
profile_headless_matrix -- --ignored --nocapture --test-threads=1
python3 scripts/summarize-headless-profile.py /tmp/matrix.csv
RUSTNET_PROFILE_PCAP=/path/to/capture.pcap RUSTNET_PROFILE_CSV=/tmp/replay.csv \
cargo test --locked --release --lib profile_headless_pcap \
-- --ignored --nocapture --test-threads=1
python3 scripts/summarize-headless-profile.py /tmp/replay.csvUse fresh CSV filenames; existing files are not overwritten. For a synthetic 30-minute trace, run python3 scripts/generate-headless-profile.py /tmp/churn.pcap (add --active 1000 --churn 0 for steady traffic).
Timings exclude packet capture, parsing/tracking, host enrichment, and output I/O. Replay uses the replay host's addresses and has no concurrent workers. The uncached reference runs first and warms memory, so its ratio is not an unbiased speedup measurement. Representative live-capture validation is still pending; synthetic replay results are not production CPU or memory measurements.
For consistent benchmarks:
# Run with consistent traffic; headless mode is required without a terminal
sudo -v
sudo ./target/release/rustnet --headless --interface eth0 > /dev/null &
sleep 1
PID=$(pgrep -n -x rustnet)
# Monitor CPU usage
top -p $PID
# Or use perf stat for detailed metrics
sudo perf stat -p $PID sleep 60
# Stop the application
sudo kill $PIDThe headless examples require v1.7.0 or later; v1.6.0 does not
include --headless. Run this example when no other rustnet process is active
so pgrep selects the process you started.
After making changes, compare before/after:
# Baseline (before changes)
sudo perf stat -r 3 ./target/release/rustnet-before --headless --duration 60 --output json > /dev/null
# After changes
sudo perf stat -r 3 ./target/release/rustnet --headless --duration 60 --output json > /dev/nullKey metrics to compare:
- CPU cycles
- Instructions per cycle (IPC)
- Cache misses
- Context switches
If your flamegraph only shows "rustnet (100%)" with no details:
Problem: Debug symbols are missing from the release build.
Solution:
# Rebuild with debug symbols
CARGO_PROFILE_RELEASE_DEBUG=true cargo build --release
# Or add to Cargo.toml:
[profile.release]
debug = true
# Then re-profile
sudo -E ~/.cargo/bin/flamegraph -- ./target/release/rustnetProblem: Running with insufficient permissions or perf can't access user-space symbols.
Solution:
# Check perf_event_paranoid setting
cat /proc/sys/kernel/perf_event_paranoid
# If it's > 1, temporarily lower it (requires root):
sudo sysctl kernel.perf_event_paranoid=1
# Or run as root
sudo -E ~/.cargo/bin/flamegraph -- ./target/release/rustnetProblem: Profiling session too short, not enough data collected.
Solution:
# Let rustnet run for at least 30-60 seconds before stopping
# The more network traffic, the better the profile
# For longer profiling:
timeout 60 sudo -E ~/.cargo/bin/flamegraph -- ./target/release/rustnetIf the TUI feels sluggish:
- Check refresh rate: Default is 500ms, can be adjusted with
--refresh-interval - Check connection count: High connection counts increase sorting overhead
- Profile the UI loop: Look for hot spots in
run_ui_loop,draw, orsort_connections - Monitor thread contention: Check if packet processing threads are blocking the snapshot provider