TRex Traffic Generator — Administrator & Developer Guide

1. Architecture Overview

TRex is Cisco's open-source, high-performance traffic generator built on top of Intel's Data Plane Development Kit (DPDK). It bypasses the kernel network stack entirely, driving NICs directly from userspace to achieve line-rate packet generation at speeds exceeding 200 Gbps. The system operates in three distinct traffic generation modes — Stateless (STL), Advanced Stateful (ASTF), and legacy Stateful (STF) — unified under a common STX framework.

1.1 High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Python Control Plane                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ TRex     │  │ STL      │  │ ASTF     │  │ EMU      │    │
│  │ Console  │  │ Client   │  │ Client   │  │ Client   │    │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘    │
│       │              │              │              │          │
│       └──────────────┴──────────────┴──────────────┘          │
│                          │ JSON-RPC                           │
└──────────────────────────┼───────────────────────────────────┘
                           │ ZMQ (REQ/REP + PUB/SUB)
┌──────────────────────────┼───────────────────────────────────┐
│              C++ Data Plane (DPDK userspace)                  │
│  ┌───────────────────────┴────────────────────────────┐      │
│  │              RPC Server (JSON-RPC)                  │      │
│  │              ZMQ Publisher (stats)                   │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │                  STX Framework                       │      │
│  │  ┌─────────┐  ┌──────────┐  ┌──────────┐           │      │
│  │  │   STL   │  │   ASTF   │  │   STF    │           │      │
│  │  │ Engine  │  │  Engine  │  │  Engine  │           │      │
│  │  └─────────┘  └──────────┘  └──────────┘           │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │  FreeBSD TCP/IP Stack (44bsd/)  │  Tunnel Support   │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │              DPDK + NIC Drivers                      │      │
│  │  (ixgbe, i40e, ice, mlx5, bnxt, virtio, etc.)      │      │
│  └─────────────────────────────────────────────────────┘      │
└──────────────────────────────────────────────────────────────┘
                           │
                    ┌──────┴──────┐
                    │  Physical   │
                    │  NICs       │
                    └─────────────┘

1.2 Component Breakdown

1.3 Traffic Generation Modes

Stateless (STL): Each port runs independent streams defined by packet templates and field engines. Supports per-stream rate control, multi-burst scheduling, flow statistics, and latency measurement. No connection state is maintained — pure packet replay with optional field manipulation.

Advanced Stateful (ASTF): Replays application-layer conversations (HTTP, FTP, etc.) using a full FreeBSD-derived TCP/IP stack embedded in userspace. Client and server sides run on different port pairs. Supports millions of concurrent flows with realistic TCP behavior including congestion control, retransmissions, and proper teardown.

Stateful (STF): Legacy mode. Template-based stateful replay without the full TCP stack. Maintained for backward compatibility; new deployments should use ASTF.

1.4 Communication Architecture

The control plane communicates with the data plane via ZeroMQ (ZMQ):

• REQ/REP channel: JSON-RPC commands (start, stop, configure, query). The Python client sends JSON-RPC requests; the C++ RPC server (src/rpc-server/) dispatches them.
• PUB/SUB channel: Asynchronous statistics and events. The C++ publisher (src/publisher/) pushes counters; the Python subscriber (trex_subscriber.py) consumes them.

Note

TRex has no REST API. All control-plane interaction is via the JSON-RPC protocol over ZMQ sockets. The Python client library (TRexClient) is the canonical interface — direct JSON-RPC usage is possible but undocumented.

2. System Requirements

2.1 Hardware Requirements

Supported architectures: x86_64 (primary), aarch64, ppc64le.

Supported NICs (from src/drivers/, 26 driver files):

Warning

DPDK requires NICs to be unbound from the kernel driver and bound to vfio-pci or igb_uio. Running TRex will make the bound interfaces unavailable to the host OS networking stack. Never bind your management interface.

2.2 Operating System

Docker images are provided for Ubuntu 16.04 and 16.10 (docker/ubuntu/).

2.3 Compiler Requirements

2.4 Kernel and System Configuration

Hugepages — required by DPDK for memory allocation:

# Allocate 1024 2MB hugepages (2 GB total)
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Or for 1GB hugepages (add to kernel boot params)
# GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4"

# Mount hugetlbfs
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge

IOMMU — required for vfio-pci:

# Add to kernel boot params for Intel
# GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt"

# Add to kernel boot params for AMD
# GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"

Environment variables:

3. Configuration Reference

3.1 Platform Configuration (YAML)

TRex platform configuration uses a YAML list format (version 2). Configuration files live in scripts/cfg/ with 15 examples for various NIC and topology combinations.

Canonical configuration structure:

- version: 2
  interfaces: ["03:00.0", "03:00.1"]    # PCI bus addresses of DPDK ports
  port_limit: 2                           # Number of ports to use
  port_info:
    - ip: 1.1.1.1                        # Port 0 source IP
      default_gw: 2.2.2.2               # Port 0 gateway
    - ip: 2.2.2.2                        # Port 1 source IP
      default_gw: 1.1.1.1               # Port 1 gateway
  port_bandwidth_gb: 40                   # Per-port bandwidth cap (Gbps)
  platform:
    master_thread_id: 0                   # CPU core for master thread
    latency_thread_id: 1                  # CPU core for latency thread
    dual_if:
      - socket: 0                        # NUMA socket
        threads: [2, 3, 4, 5]            # Worker threads for port pair
  c: 4                                    # Number of cores per port pair
  limit_memory: 2048                      # Memory limit (MB)
  enable_zmq_pub: true                    # Enable ZMQ stats publisher
  zmq_pub_port: 4500                     # ZMQ publisher port
  prefix: my_trex                         # Multi-instance prefix

3.2 Key Configuration Fields

Tip

Use dpdk_setup_ports.py to discover PCI addresses and auto-generate a configuration file. Running multiple TRex instances on the same machine requires unique prefix values and non-overlapping port/core assignments.

3.3 Port Info Configuration Patterns

Basic (ARP-resolved):

port_info:
  - ip: 10.0.0.1
    default_gw: 10.0.0.2
  - ip: 10.0.0.2
    default_gw: 10.0.0.1

Static MAC (no ARP):

port_info:
  - dest_mac: "00:00:00:01:00:00"
    src_mac: "00:00:00:02:00:00"
  - dest_mac: "00:00:00:02:00:00"
    src_mac: "00:00:00:01:00:00"

With VLAN:

port_info:
  - ip: 10.0.0.1
    default_gw: 10.0.0.2
    vlan: 100
  - ip: 10.0.0.2
    default_gw: 10.0.0.1
    vlan: 100

3.4 NUMA-Aware CPU Pinning

For maximum performance, pin data-plane threads to the same NUMA node as the NIC:

platform:
  master_thread_id: 0
  latency_thread_id: 16
  dual_if:
    - socket: 0
      threads: [1, 2, 3, 4, 5, 6, 7]
    - socket: 1
      threads: [17, 18, 19, 20, 21, 22, 23]

Verify NUMA topology:

lscpu | grep NUMA
# NUMA node0 CPU(s): 0-15
# NUMA node1 CPU(s): 16-31

cat /sys/bus/pci/devices/0000:03:00.0/numa_node
# 0

3.5 Docker Configuration

Docker configuration files in docker/ubuntu/16.04/ and docker/ubuntu/16.10/ include a trex_cfg.yaml that maps to the container's expected layout. When running TRex in Docker, the container requires:

• --privileged or appropriate device access to /dev/hugepages and PCI devices
• Host network mode (--net=host) or direct NIC passthrough
• Hugepage mount from host

4. Deployment & Scaling

4.1 Building from Source

TRex uses the waf build system. Two build targets exist:

DPDK build (production):

cd linux_dpdk
./configure
make
# Binary output: _t-rex-64

The DPDK build (linux_dpdk/ws_main.py) compiles the full DPDK framework alongside the TRex engine, producing a statically-linked binary that includes NIC drivers, memory management, and the data plane.

Simulator build (development/testing):

cd linux
./configure
make
# Binary output: for unit tests and simulation without hardware

The simulator build (linux/ws_main.py) produces a binary that can run STL/ASTF/STF simulations without physical NICs, used for profile validation and unit testing.

Build options:

Important

The build system (build_option class, 31+ methods in ws_main.py) handles cross-compilation for x86_64, aarch64, and ppc64le. Ensure the correct toolchain is installed and CXX is set if not using the system default compiler.

4.2 Installation Layout

After building or extracting a release:

trex-core/
├── _t-rex-64                    # Main binary (DPDK build)
├── t-rex-64                     # Symlink or wrapper script
├── scripts/
│   ├── cfg/                     # Configuration examples
│   ├── stl/                     # Stateless profiles (74)
│   ├── astf/                    # ASTF profiles (97)
│   ├── emu/                     # EMU profiles (23)
│   ├── cap2/                    # Packet captures (62)
│   ├── dpdk_setup_ports.py      # NIC setup utility
│   └── automation/
│       └── trex_control_plane/  # Python client library
├── external_libs/               # Bundled Python dependencies
└── doc/                         # Documentation

4.3 NIC Binding

Before running TRex, NICs must be bound to a DPDK-compatible driver:

# List NICs and their current drivers
sudo ./dpdk_setup_ports.py -s

# Expected output:
# 03:00.0 'Ethernet Controller X710' drv=i40e unused=vfio-pci
# 03:00.1 'Ethernet Controller X710' drv=i40e unused=vfio-pci

# Bind to vfio-pci
sudo ./dpdk_setup_ports.py -b vfio-pci 03:00.0 03:00.1

# Generate config file interactively
sudo ./dpdk_setup_ports.py -i

Caution

Issue #1135 reports that dpdk_setup_ports.py may fail with "no Driver_str" on certain NIC models. If this occurs, manually bind the driver with dpdk-devbind.py from the DPDK tools or specify the PCI address directly in the configuration file.

4.4 Starting TRex

Stateless mode:

sudo ./t-rex-64 -i --cfg scripts/cfg/my_config.yaml
# -i : interactive mode (starts RPC server, waits for client connections)
# Output: shows port info, starts listening on ZMQ ports

ASTF mode:

sudo ./t-rex-64 -i --astf --cfg scripts/cfg/my_config.yaml
# --astf : start in Advanced Stateful mode

Simulator mode (no hardware):

./bp-sim-64 --stl -f scripts/stl/bench.py -o output.pcap
# Runs a profile through the simulator, writes pcap output

4.5 Multi-Instance Deployment

Multiple TRex instances can run on the same server by isolating ports and cores:

# Instance 1: trex_a.yaml
- version: 2
  interfaces: ["03:00.0", "03:00.1"]
  prefix: trex_a
  c: 4
  platform:
    master_thread_id: 0
    latency_thread_id: 1
    dual_if:
      - socket: 0
        threads: [2, 3, 4, 5]

# Instance 2: trex_b.yaml
- version: 2
  interfaces: ["05:00.0", "05:00.1"]
  prefix: trex_b
  c: 4
  platform:
    master_thread_id: 8
    latency_thread_id: 9
    dual_if:
      - socket: 1
        threads: [10, 11, 12, 13]

Each instance must have a unique prefix, non-overlapping CPU cores, and separate NIC ports.

4.6 CI/CD

TRex uses Travis CI (.travis.yml) for continuous integration. The BUILD_NUM environment variable is set by Travis to track build numbers.

5. Operations

5.1 Connecting the Console

./trex-console

# Connect to a remote TRex instance
./trex-console -s 192.168.1.100

# Expected output:
# Connecting to TRex server at 192.168.1.100:4501
# Connected
# trex>

5.2 Python Client API (TRexClient)

The TRexClient class (93+ methods) is the primary interface for automation:

from trex.common.trex_client import TRexClient

# Connect
client = TRexClient(server='127.0.0.1')
client.connect()

# Acquire ports
client.acquire(port_id=[0, 1])

# Start traffic (STL example)
client.start(ports=[0], mult="10gbps", duration=60)

# Get statistics
stats = client.get_stats()
print(stats[0])  # Port 0 stats

# Stop and disconnect
client.stop(ports=[0])
client.disconnect()

Key TRexClient method categories (93+ methods):

5.3 Port Object (Port Class, 52+ Methods)

Each port is represented by a Port object with methods for:

5.4 Namespace Management (NSCmds)

The NSCmds class manages network namespaces for EMU (emulation) scenarios:

ns = NSCmds()
ns.add_node(mac="00:00:00:01:00:00", ipv4="10.0.0.1")
ns.set_dg(ipv4="10.0.0.254")
ns.set_vlan(vlans=[100])

5.5 Service Framework

The service framework (interactive/trex/common/services/, 10 files) provides pluggable services that run on ports in service mode:

• ARP resolution
• DHCP client
• ICMPv6 / IPv6 Neighbor Discovery
• IGMP
• Custom service plugins

5.6 Console Plugins

Console plugins (interactive/trex/console/plugins/, 8 files) extend the TRex console:

5.7 Monitoring Statistics

TRex publishes real-time statistics via ZMQ PUB/SUB. The Python subscriber (trex_subscriber.py) processes these asynchronously. Statistics include:

• Per-port TX/RX counters (packets, bytes, errors)
• Per-stream flow statistics
• Latency histograms (via HDR Histogram, src/hdrh/)
• CPU utilization per core
• Memory (hugepage) usage

# Get global stats
stats = client.get_stats()
# stats keys: 'global', port indices (0, 1, ...), 'flow_stats', 'latency'

print(f"TX: {stats['global']['tx_bps']:.2f} bps")
print(f"RX: {stats['global']['rx_bps']:.2f} bps")
print(f"CPU: {stats['global']['cpu_util']:.1f}%")

5.8 Gratuitous ARP

Warning

Issue #1162 reports that Gratuitous ARP is broken in TRex v3.06. If you rely on GARP for failover scenarios, test thoroughly or apply the fix from the issue tracker before deploying v3.06.

6. Performance

6.1 Performance Architecture

TRex achieves line-rate performance through:

1. DPDK kernel bypass: Zero-copy packet I/O directly to NIC hardware queues
2. Per-core packet processing: Each worker thread handles a subset of flows, no shared state contention
3. Hugepage memory: Pre-allocated memory pools eliminate allocation overhead
4. NUMA awareness: Threads pinned to the same NUMA node as their NIC
5. Batch processing: Packets processed in bursts (configurable RX/TX burst size)

6.2 Performance Tuning

Core allocation: More cores per port pair increases packet processing throughput. Each core can handle approximately 10-15 Mpps depending on packet size and profile complexity.

# High-throughput configuration: 7 cores per port pair
c: 7
platform:
  master_thread_id: 0
  latency_thread_id: 15
  dual_if:
    - socket: 0
      threads: [1, 2, 3, 4, 5, 6, 7]

Memory allocation: Insufficient hugepages cause startup failure. Calculate based on:

• Per-port memory pool: ~256 MB minimum
• Per-core packet buffers: ~64 MB
• ASTF TCP stack state: ~1 KB per concurrent connection

# Check hugepage allocation
cat /proc/meminfo | grep Huge
# HugePages_Total:    1024
# HugePages_Free:      512
# Hugepagesize:       2048 kB

CPU isolation: For consistent performance, isolate TRex cores from the Linux scheduler:

# Kernel boot params
# isolcpus=1-7,17-23 nohz_full=1-7,17-23 rcu_nocbs=1-7,17-23

6.3 Latency Measurement

TRex uses HDR Histogram (src/hdrh/, 10 files) for high-resolution latency measurement. A dedicated latency thread (configured via latency_thread_id) sends and receives timestamped probe packets separately from the data plane.

# Enable latency on a stream
stream = STLStream(
    packet=pkt,
    mode=STLTXCont(pps=1000),
    flow_stats=STLFlowLatencyStats(pg_id=7)
)

6.4 Packet Size Considerations

Issue #1205 reports packet truncation to 64 bytes on E810 (ICE) NICs. Verify minimum packet sizes are met for your NIC driver. Issue #1203 requests custom RX/TX burst sizes — the default burst size is compiled in; modification requires a rebuild.

7. Security

7.1 Privilege Model

TRex requires root privileges (or CAP_SYS_RAWIO, CAP_NET_ADMIN, CAP_IPC_LOCK) because DPDK needs:

• Access to PCI device files (/sys/bus/pci/)
• Hugepage allocation and mapping
• Direct hardware register access via vfio-pci or igb_uio

Caution

Running TRex as root on a shared system exposes all PCI devices to the process. In multi-tenant environments, use VFIO with proper IOMMU groups to isolate NIC access, and restrict SSH access to the TRex host.

7.2 Network Security Implications

TRex can generate arbitrary packets at line rate. This makes it a powerful tool that can cause network disruption if misconfigured:

• MAC spoofing: TRex sets arbitrary source MACs. Ensure upstream switches permit this or configure src_mac in port_info.
• IP spoofing: Source IPs in traffic profiles are user-defined. Restrict TRex to isolated test networks.
• Denial of service: Line-rate traffic generation can overwhelm any network device. Physical isolation of test networks is strongly recommended.
• SYN flood capability: Profiles like scripts/stl/syn_attack.py exist for testing purposes. Ensure these are only used in controlled environments.

7.3 Control Plane Security

The JSON-RPC server listens on ZMQ sockets (default ports 4500-4501). There is no built-in authentication or encryption:

• Network isolation: Bind TRex management to a dedicated management VLAN or loopback interface
• Firewall: Restrict access to ZMQ ports (4500-4501 and the ZMQ publisher port) to authorized clients only
• No TLS: ZMQ transport is plaintext. Do not expose control ports to untrusted networks

# Firewall example: restrict TRex control to management subnet
iptables -A INPUT -p tcp --dport 4500:4501 -s 10.0.100.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 4500:4501 -j DROP

7.4 Bundled Dependencies

TRex bundles the following external libraries, which should be tracked for CVEs:

Important

The bundled Scapy 2.4.3 is a known-old version. While it runs only on the client side (not in the data plane), ensure your Python environment does not have conflicting Scapy installations. Other bundled Python libraries include jsonrpclib, jsonpickle, requests, urllib3, pyyaml, nose, progressbar, texttable, and simpy.

7.5 File System Security

• Configuration files in scripts/cfg/ may contain MAC addresses and IP topology — treat as sensitive in production environments
• Packet capture files (scripts/cap2/) may contain real traffic patterns
• The TRex binary requires SUID root or equivalent capabilities

8. Developer Internals

8.1 Source Code Map

Core engine entry (src/main.cpp):

The main entry point parses command-line arguments and dispatches to one of five simulator modes:

8.2 STX Framework Architecture

The STX framework (src/stx/) provides a unified abstraction layer across all traffic modes. The src/stx/common/ directory (50 files) contains shared infrastructure:

• Port management and state machines
• Memory pool allocation
• Packet buffer management
• Statistics collection
• Timer wheel for scheduled events

Each mode extends STX with mode-specific logic:

• src/stx/stl/: Stream table, field engine VM, multi-burst scheduler
• src/stx/astf/: TCP template loading, connection pool, flow table
• src/stx/stf/: Legacy template-based replay

8.3 FreeBSD TCP/IP Stack (src/44bsd/)

The ASTF mode embeds a FreeBSD-derived TCP/IP stack (24 files, 347K characters) providing:

• Full TCP state machine (SYN, ESTABLISHED, FIN_WAIT, TIME_WAIT, etc.)
• Congestion control (Reno, NewReno variants)
• Retransmission timers
• Window scaling
• Selective acknowledgment (SACK)
• TCP segmentation

This stack runs entirely in userspace, integrated with DPDK's memory and packet buffer system. Each worker thread has its own stack instance — no locking required.

8.4 RPC Server (src/rpc-server/)

The JSON-RPC server (13 files) handles control-plane commands:

Client (Python) ──ZMQ REQ──▶ RPC Server ──dispatch──▶ Command Handler
                                                              │
                                                              ▼
                                                        STX Framework
                                                     (start/stop/config)

Commands are validated against JSON schemas (using valijson). Each command maps to a handler function that interacts with the STX framework.

8.5 Statistics Publisher (src/publisher/)

Two files implement the ZMQ PUB statistics channel:

• Periodic stats broadcast (configurable interval)
• Port counters, flow stats, latency histograms
• Global CPU and memory utilization
• Consumed by trex_subscriber.py on the client side

8.6 NIC Driver Layer (src/drivers/)

The 26 driver files extend or patch DPDK drivers for TRex-specific functionality:

• Custom RSS (Receive Side Scaling) configuration
• Hardware flow director rules
• MTU handling (see issue #1204 re: MTU set to 65518)
• Interrupt coalescing tuning
• Driver-specific workarounds

8.7 Platform Abstraction Layer (src/pal/)

PAL provides two implementations:

This allows the same core logic to compile for both simulation testing and production deployment.

8.8 Python Client Architecture

trex_control_plane/interactive/trex/
├── common/
│   ├── trex_client.py          # TRexClient (93+ methods)
│   ├── trex_conn.py            # JsonRpcClient connection
│   ├── trex_port.py            # Port (52+ methods)
│   ├── trex_ns.py              # NSCmds namespace management
│   ├── trex_subscriber.py      # Async ZMQ stats subscriber
│   ├── trex_events.py          # Event system
│   ├── trex_logger.py          # Logger (ScreenLogger)
│   ├── trex_types.py           # RC, VLAN, PortProfileID
│   ├── trex_psv.py             # Port state validation
│   ├── trex_exceptions.py      # Exception hierarchy
│   ├── trex_vlan.py            # VLAN handling
│   ├── trex_api_annotators.py  # API decorators
│   └── services/               # Service framework (10 files)
├── stl/                         # STL-specific client
├── astf/                        # ASTF-specific client
├── emu/                         # EMU client
├── wireless/                    # Wireless testing
├── console/
│   └── plugins/                 # Console plugins (8 files)
└── utils/                       # Utilities (9 files)

Exception hierarchy:

8.9 Traffic Profile System

STL Profiles (74 files in scripts/stl/):

STL profiles define packet templates and field engines. Key profile types:

STL profile classes (e.g., STLS1, STLBench, STLImix, STLPcap) extend a base class and implement get_streams() to return stream definitions.

ASTF Profiles (97 files in scripts/astf/):

ASTF profiles define client/server application templates:

EMU Profiles (23 files in scripts/emu/):

Network protocol emulation:

8.10 Build System Internals

The build_option class (31+ methods) in ws_main.py manages:

• Compiler detection and version validation
• DPDK library compilation and linking
• Platform-specific flags (x86_64, aarch64, ppc64le)
• Sanitizer integration (ASan, MSan)
• External library paths
• Static linking configuration

The build produces a single statically-linked binary (_t-rex-64) that embeds DPDK, all NIC drivers, the TCP/IP stack, and the RPC server.

8.11 Tunnel Support (src/tunnels/)

Seven files implement tunnel encapsulation/decapsulation:

• GTP-U (GPRS Tunneling Protocol) — mobile network testing
• GRE (Generic Routing Encapsulation)
• Custom tunnel handlers

8.12 Testing

C++ tests (src/gtest/, 11 files): Google Test-based unit and integration tests. Run via the GTEST simulator mode:

./bp-sim-64 --gtest

Python tests: Profile validation via simulator mode:

./bp-sim-64 --stl -f scripts/stl/bench.py -o /dev/null

9. Integration Points

9.1 Automation Integration

TRex is designed for integration with test automation frameworks:

# Robot Framework / pytest integration pattern
from trex.stl.trex_stl_client import STLClient

def run_traffic_test(server, duration, rate):
    client = STLClient(server=server)
    try:
        client.connect()
        client.reset()
        client.add_streams(profile, ports=[0])
        client.start(ports=[0], mult=rate, duration=duration)
        client.wait_on_traffic()
        stats = client.get_stats()
        return stats
    finally:
        client.disconnect()

9.2 Monitoring Integration

ZMQ PUB/SUB statistics can be consumed by external monitoring systems:

• Subscribe to the ZMQ publisher port (default 4500)
• Parse JSON-formatted statistics messages
• Feed into Grafana, Prometheus (via custom exporter), or ELK

9.3 Docker Integration

# Example: running TRex in Docker
FROM ubuntu:16.04
# ... install dependencies ...
COPY trex_cfg.yaml /etc/trex_cfg.yaml
CMD ["./t-rex-64", "-i", "--cfg", "/etc/trex_cfg.yaml"]

Docker configurations in docker/ubuntu/ provide reference Dockerfiles.

9.4 Known Integration Issues

10. Troubleshooting

10.1 Startup Failures

Error: DPDK map error

• Cause: Insufficient hugepages or incorrect IOMMU configuration (issue #919)
• Solution: Verify hugepage allocation (cat /proc/meminfo | grep Huge), check IOMMU is enabled in BIOS and kernel params
• Prevention: Pre-allocate hugepages in /etc/sysctl.conf and verify at boot

Error: ixgbe_dev_start failure

• Cause: NIC initialization failure, often due to firmware mismatch or driver conflict (issue #934)
• Solution: Update NIC firmware, ensure kernel ixgbe module is unloaded before binding to DPDK
• Prevention: Keep NIC firmware up to date, use dpdk_setup_ports.py for proper binding

Error: Unable to start in ASTF

• Cause: ASTF mode requires specific configuration and port pair setup (issue #1183)
• Solution: Ensure ports are configured in pairs with valid IP addresses and gateways. Verify --astf flag is passed.
• Prevention: Test ASTF configuration with the simulator build first

Error: no Driver_str in dpdk_setup_ports.py

• Cause: NIC model not recognized by the setup script (issue #1135)
• Solution: Manually bind the NIC using DPDK's dpdk-devbind.py, or update dpdk_setup_ports.py
• Prevention: Use well-supported NIC models (X710, ConnectX-5)

10.2 Runtime Issues

Error: Method not found ctx_stop_autotriggers

• Cause: Client/server version mismatch — the RPC method does not exist in the running TRex version (issue #1208)
• Solution: Ensure the Python client library version matches the TRex server version
• Prevention: Deploy client and server from the same release package

Error: Connection lost / TRexTimeoutError

• Cause: ZMQ connection timeout, TRex process crash, or firewall blocking ports
• Solution: Check TRex process is running (ps aux | grep t-rex), verify firewall rules for ZMQ ports, check system logs for crash dumps
• Prevention: Monitor TRex process health, use client.is_connected() in automation

Error: Packets truncated to 64 bytes

• Cause: E810 (ICE) driver issue (issue #1205)
• Solution: Set minimum packet size to 64 bytes in profiles, or use a NIC with a more mature driver
• Prevention: Test with your specific NIC model before production deployment

Note

TRex writes crash dumps and logs to the working directory. When reporting issues, include the log output, platform configuration YAML, and the output of dpdk_setup_ports.py -s.

10.3 Performance Issues

Symptom: TX rate not reaching target

• Cause: Insufficient cores, NUMA cross-socket access, or memory pressure
• Solution: Add more cores (c field), pin threads to correct NUMA node, increase hugepage allocation
• Prevention: Follow NUMA-aware configuration (Section 3.4)

Symptom: High latency variance

• Cause: Latency thread sharing a core with data plane, or CPU frequency scaling
• Solution: Dedicate a separate core for latency (latency_thread_id), disable CPU frequency scaling (cpupower frequency-set -g performance)
• Prevention: Include latency thread in initial platform configuration

10.4 Build Issues

Error: Compiler version too old

• Cause: GCC below 4.7.0 or sanitizer build with GCC below 4.9.0
• Solution: Install appropriate GCC version, set CXX environment variable
• Prevention: Use a supported OS with GCC 7.4+ from package manager

11. Quick Reference

11.1 Common Commands

11.2 Default Ports

11.3 File Locations

11.4 Configuration Checklist

• [ ] Hugepages allocated and mounted
• [ ] IOMMU enabled (for vfio-pci)
• [ ] NICs unbound from kernel driver
• [ ] NICs bound to vfio-pci or igb_uio
• [ ] YAML config created with correct PCI addresses
• [ ] CPU cores pinned to correct NUMA node
• [ ] Latency thread on dedicated core
• [ ] Management interface NOT bound to DPDK
• [ ] Firewall rules for ZMQ ports (if remote access needed)
• [ ] Test network physically isolated from production

For installation steps, profile creation, and usage walkthroughs, see the User Guide.