# TRex Traffic Generator — Administrator & Developer Guide

## 1. Architecture Overview

TRex is Cisco's open-source, high-performance traffic generator built on top of Intel's Data Plane Development Kit (DPDK). It bypasses the kernel network stack entirely, driving NICs directly from userspace to achieve line-rate packet generation at speeds exceeding 200 Gbps. The system operates in three distinct traffic generation modes — Stateless (STL), Advanced Stateful (ASTF), and legacy Stateful (STF) — unified under a common STX framework.

### 1.1 High-Level Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Python Control Plane                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ TRex     │  │ STL      │  │ ASTF     │  │ EMU      │    │
│  │ Console  │  │ Client   │  │ Client   │  │ Client   │    │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘    │
│       │              │              │              │          │
│       └──────────────┴──────────────┴──────────────┘          │
│                          │ JSON-RPC                           │
└──────────────────────────┼───────────────────────────────────┘
                           │ ZMQ (REQ/REP + PUB/SUB)
┌──────────────────────────┼───────────────────────────────────┐
│              C++ Data Plane (DPDK userspace)                  │
│  ┌───────────────────────┴────────────────────────────┐      │
│  │              RPC Server (JSON-RPC)                  │      │
│  │              ZMQ Publisher (stats)                   │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │                  STX Framework                       │      │
│  │  ┌─────────┐  ┌──────────┐  ┌──────────┐           │      │
│  │  │   STL   │  │   ASTF   │  │   STF    │           │      │
│  │  │ Engine  │  │  Engine  │  │  Engine  │           │      │
│  │  └─────────┘  └──────────┘  └──────────┘           │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │  FreeBSD TCP/IP Stack (44bsd/)  │  Tunnel Support   │      │
│  ├─────────────────────────────────────────────────────┤      │
│  │              DPDK + NIC Drivers                      │      │
│  │  (ixgbe, i40e, ice, mlx5, bnxt, virtio, etc.)      │      │
│  └─────────────────────────────────────────────────────┘      │
└──────────────────────────────────────────────────────────────┘
                           │
                    ┌──────┴──────┐
                    │  Physical   │
                    │  NICs       │
                    └─────────────┘
```

### 1.2 Component Breakdown

| Layer | Directory | Language | Purpose |
|-------|-----------|----------|---------|
| Main entry | `src/main.cpp` | C++ | Process init, mode dispatch (STF/STL/ASTF/GTEST) |
| STX framework | `src/stx/` | C++ | Unified framework for all traffic modes |
| STX common | `src/stx/common/` | C++ | 50 files of shared STX infrastructure |
| Stateless engine | `src/stx/stl/` | C++ | STL mode: per-stream profiles, field engine |
| ASTF engine | `src/stx/astf/` | C++ | Advanced stateful: TCP/UDP template replay |
| ASTF batch | `src/stx/astf_batch/` | C++ | Batch-mode ASTF processing |
| STF engine | `src/stx/stf/` | C++ | Legacy stateful mode |
| TCP/IP stack | `src/44bsd/` | C++ | Ported FreeBSD TCP/IP (24 files, 347K chars) |
| RPC server | `src/rpc-server/` | C++ | JSON-RPC command handler (13 files) |
| Stats publisher | `src/publisher/` | C++ | ZMQ PUB for real-time statistics (2 files) |
| Tunnels | `src/tunnels/` | C++ | GTP, GRE, and other tunnel encapsulation (7 files) |
| HDR Histogram | `src/hdrh/` | C++ | Latency measurement (10 files) |
| NIC drivers | `src/drivers/` | C++ | Driver patches and extensions (26 files) |
| PAL | `src/pal/` | C++ | Platform Abstraction (Linux, Linux-DPDK) |
| Utilities | `src/utils/`, `src/common/` | C++ | Helper functions (51 files total) |
| Python client | `scripts/automation/trex_control_plane/interactive/` | Python | Client library, console, services |
| STL profiles | `scripts/stl/` | Python | 74 stateless traffic profiles |
| ASTF profiles | `scripts/astf/` | Python | 97 advanced stateful profiles |
| EMU profiles | `scripts/emu/` | Python | 23 emulation profiles |
| Captures | `scripts/cap2/` | pcap | 62 packet capture files |
| Config examples | `scripts/cfg/` | YAML | 15 platform configuration examples |
| Build system | `linux_dpdk/ws_main.py`, `linux/ws_main.py` | Python (waf) | Build orchestration |
| Tests | `src/gtest/` | C++ | Google Test suite (11 files) |

### 1.3 Traffic Generation Modes

**Stateless (STL):** Each port runs independent streams defined by packet templates and field engines. Supports per-stream rate control, multi-burst scheduling, flow statistics, and latency measurement. No connection state is maintained — pure packet replay with optional field manipulation.

**Advanced Stateful (ASTF):** Replays application-layer conversations (HTTP, FTP, etc.) using a full FreeBSD-derived TCP/IP stack embedded in userspace. Client and server sides run on different port pairs. Supports millions of concurrent flows with realistic TCP behavior including congestion control, retransmissions, and proper teardown.

**Stateful (STF):** Legacy mode. Template-based stateful replay without the full TCP stack. Maintained for backward compatibility; new deployments should use ASTF.

### 1.4 Communication Architecture

The control plane communicates with the data plane via ZeroMQ (ZMQ):

- **REQ/REP channel:** JSON-RPC commands (start, stop, configure, query). The Python client sends JSON-RPC requests; the C++ RPC server (`src/rpc-server/`) dispatches them.
- **PUB/SUB channel:** Asynchronous statistics and events. The C++ publisher (`src/publisher/`) pushes counters; the Python subscriber (`trex_subscriber.py`) consumes them.

> [!NOTE]
> TRex has no REST API. All control-plane interaction is via the JSON-RPC protocol over ZMQ sockets. The Python client library (`TRexClient`) is the canonical interface — direct JSON-RPC usage is possible but undocumented.

---

## 2. System Requirements

### 2.1 Hardware Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | x86_64, 4+ cores | x86_64 with AES-NI, 8+ cores, NUMA-aware |
| RAM | 4 GB + hugepages | 16+ GB, 1 GB hugepages |
| NIC | DPDK-supported (ixgbe, i40e) | Intel X710, XXV710, E810, Mellanox ConnectX-5/6 |
| Disk | 2 GB for installation | SSD for profile/pcap storage |

**Supported architectures:** x86_64 (primary), aarch64, ppc64le.

**Supported NICs (from `src/drivers/`, 26 driver files):**

| Driver | NICs | Notes |
|--------|------|-------|
| ixgbe | Intel 82599, X520, X540, X550 | 10 GbE, mature driver |
| i40e | Intel X710, XL710, XXV710 | 25/40 GbE |
| ice | Intel E810 | 100 GbE, see known issues (#1202, #1205) |
| mlx5 | Mellanox ConnectX-4/5/6 | 25/50/100/200 GbE via ibverbs |
| bnxt | Broadcom NetXtreme | 10/25/50 GbE |
| virtio | QEMU virtio-net | Virtual environments |
| vmxnet3 | VMware | Virtual environments |
| igc | Intel I225/I226 | 2.5 GbE |
| ntacc | Napatech | SmartNIC |

> [!WARNING]
> DPDK requires NICs to be unbound from the kernel driver and bound to `vfio-pci` or `igb_uio`. Running TRex will make the bound interfaces unavailable to the host OS networking stack. Never bind your management interface.

### 2.2 Operating System

| OS | Status |
|----|--------|
| Ubuntu 16.04+ | Primary target |
| CentOS 7+ | Supported |
| Fedora | Supported |
| Other Linux x86_64 | Should work with kernel 3.x+ |

Docker images are provided for Ubuntu 16.04 and 16.10 (`docker/ubuntu/`).

### 2.3 Compiler Requirements

| Compiler | Minimum | Notes |
|----------|---------|-------|
| GCC | 4.7.0 | Basic compilation |
| GCC | 4.9.0 | Required for address/memory sanitizer (`--sanitized`) |
| GCC 6.2 | Supported | `--gcc6` build flag |
| GCC 7.4 | Supported | `--gcc7` build flag |
| GCC 8.3 | Supported | `--gcc8` build flag |

### 2.4 Kernel and System Configuration

**Hugepages** — required by DPDK for memory allocation:

```bash
# Allocate 1024 2MB hugepages (2 GB total)
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Or for 1GB hugepages (add to kernel boot params)
# GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4"

# Mount hugetlbfs
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
```

**IOMMU** — required for `vfio-pci`:

```bash
# Add to kernel boot params for Intel
# GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt"

# Add to kernel boot params for AMD
# GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"
```

**Environment variables:**

| Variable | Purpose | Example |
|----------|---------|---------|
| `CXX` | C++ compiler override | `CXX=g++-8` |
| `USER` | Build user identification | Set by system |
| `BUILD_NUM` | CI build number (Travis) | Set by CI |

---

## 3. Configuration Reference

### 3.1 Platform Configuration (YAML)

TRex platform configuration uses a YAML list format (version 2). Configuration files live in `scripts/cfg/` with 15 examples for various NIC and topology combinations.

**Canonical configuration structure:**

```yaml
- version: 2
  interfaces: ["03:00.0", "03:00.1"]    # PCI bus addresses of DPDK ports
  port_limit: 2                           # Number of ports to use
  port_info:
    - ip: 1.1.1.1                        # Port 0 source IP
      default_gw: 2.2.2.2               # Port 0 gateway
    - ip: 2.2.2.2                        # Port 1 source IP
      default_gw: 1.1.1.1               # Port 1 gateway
  port_bandwidth_gb: 40                   # Per-port bandwidth cap (Gbps)
  platform:
    master_thread_id: 0                   # CPU core for master thread
    latency_thread_id: 1                  # CPU core for latency thread
    dual_if:
      - socket: 0                        # NUMA socket
        threads: [2, 3, 4, 5]            # Worker threads for port pair
  c: 4                                    # Number of cores per port pair
  limit_memory: 2048                      # Memory limit (MB)
  enable_zmq_pub: true                    # Enable ZMQ stats publisher
  zmq_pub_port: 4500                     # ZMQ publisher port
  prefix: my_trex                         # Multi-instance prefix
```

### 3.2 Key Configuration Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `version` | int | 2 | Config format version (always 2) |
| `interfaces` | list[str] | required | PCI addresses of network ports |
| `port_limit` | int | len(interfaces) | Number of ports to activate |
| `port_info` | list[dict] | — | Per-port IP/MAC/gateway configuration |
| `port_info[].ip` | str | — | Source IP address for the port |
| `port_info[].default_gw` | str | — | Default gateway IP |
| `port_info[].dest_mac` | str | — | Override destination MAC (skip ARP) |
| `port_info[].src_mac` | str | — | Override source MAC |
| `port_bandwidth_gb` | int | — | Bandwidth cap in Gbps per port |
| `platform` | dict | — | CPU pinning and NUMA topology |
| `platform.master_thread_id` | int | 0 | Core for the master (control) thread |
| `platform.latency_thread_id` | int | — | Core for latency measurement thread |
| `platform.dual_if` | list[dict] | — | Per-port-pair thread assignment |
| `platform.dual_if[].socket` | int | — | NUMA socket for this port pair |
| `platform.dual_if[].threads` | list[int] | — | CPU cores for data plane threads |
| `c` | int | 1 | Cores per dual-interface (port pair) |
| `limit_memory` | int | — | DPDK memory limit in MB |
| `enable_zmq_pub` | bool | false | Enable ZMQ statistics publisher |
| `zmq_pub_port` | int | 4500 | ZMQ publisher port |
| `prefix` | str | — | Namespace prefix (multi-instance) |
| `memory` | dict | — | Memory allocation overrides |

> [!TIP]
> Use `dpdk_setup_ports.py` to discover PCI addresses and auto-generate a configuration file. Running multiple TRex instances on the same machine requires unique `prefix` values and non-overlapping port/core assignments.

### 3.3 Port Info Configuration Patterns

**Basic (ARP-resolved):**
```yaml
port_info:
  - ip: 10.0.0.1
    default_gw: 10.0.0.2
  - ip: 10.0.0.2
    default_gw: 10.0.0.1
```

**Static MAC (no ARP):**
```yaml
port_info:
  - dest_mac: "00:00:00:01:00:00"
    src_mac: "00:00:00:02:00:00"
  - dest_mac: "00:00:00:02:00:00"
    src_mac: "00:00:00:01:00:00"
```

**With VLAN:**
```yaml
port_info:
  - ip: 10.0.0.1
    default_gw: 10.0.0.2
    vlan: 100
  - ip: 10.0.0.2
    default_gw: 10.0.0.1
    vlan: 100
```

### 3.4 NUMA-Aware CPU Pinning

For maximum performance, pin data-plane threads to the same NUMA node as the NIC:

```yaml
platform:
  master_thread_id: 0
  latency_thread_id: 16
  dual_if:
    - socket: 0
      threads: [1, 2, 3, 4, 5, 6, 7]
    - socket: 1
      threads: [17, 18, 19, 20, 21, 22, 23]
```

Verify NUMA topology:

```bash
lscpu | grep NUMA
# NUMA node0 CPU(s): 0-15
# NUMA node1 CPU(s): 16-31

cat /sys/bus/pci/devices/0000:03:00.0/numa_node
# 0
```

### 3.5 Docker Configuration

Docker configuration files in `docker/ubuntu/16.04/` and `docker/ubuntu/16.10/` include a `trex_cfg.yaml` that maps to the container's expected layout. When running TRex in Docker, the container requires:

- `--privileged` or appropriate device access to `/dev/hugepages` and PCI devices
- Host network mode (`--net=host`) or direct NIC passthrough
- Hugepage mount from host

---

## 4. Deployment & Scaling

### 4.1 Building from Source

TRex uses the `waf` build system. Two build targets exist:

**DPDK build (production):**
```bash
cd linux_dpdk
./configure
make
# Binary output: _t-rex-64
```

The DPDK build (`linux_dpdk/ws_main.py`) compiles the full DPDK framework alongside the TRex engine, producing a statically-linked binary that includes NIC drivers, memory management, and the data plane.

**Simulator build (development/testing):**
```bash
cd linux
./configure
make
# Binary output: for unit tests and simulation without hardware
```

The simulator build (`linux/ws_main.py`) produces a binary that can run STL/ASTF/STF simulations without physical NICs, used for profile validation and unit testing.

**Build options:**

| Flag | Purpose |
|------|---------|
| `--sanitized` | Enable address/memory sanitizer (requires GCC 4.9+) |
| `--gcc6` | Build with GCC 6.2 |
| `--gcc7` | Build with GCC 7.4 |
| `--gcc8` | Build with GCC 8.3 |

> [!IMPORTANT]
> The build system (`build_option` class, 31+ methods in `ws_main.py`) handles cross-compilation for x86_64, aarch64, and ppc64le. Ensure the correct toolchain is installed and `CXX` is set if not using the system default compiler.

### 4.2 Installation Layout

After building or extracting a release:

```
trex-core/
├── _t-rex-64                    # Main binary (DPDK build)
├── t-rex-64                     # Symlink or wrapper script
├── scripts/
│   ├── cfg/                     # Configuration examples
│   ├── stl/                     # Stateless profiles (74)
│   ├── astf/                    # ASTF profiles (97)
│   ├── emu/                     # EMU profiles (23)
│   ├── cap2/                    # Packet captures (62)
│   ├── dpdk_setup_ports.py      # NIC setup utility
│   └── automation/
│       └── trex_control_plane/  # Python client library
├── external_libs/               # Bundled Python dependencies
└── doc/                         # Documentation
```

### 4.3 NIC Binding

Before running TRex, NICs must be bound to a DPDK-compatible driver:

```bash
# List NICs and their current drivers
sudo ./dpdk_setup_ports.py -s

# Expected output:
# 03:00.0 'Ethernet Controller X710' drv=i40e unused=vfio-pci
# 03:00.1 'Ethernet Controller X710' drv=i40e unused=vfio-pci

# Bind to vfio-pci
sudo ./dpdk_setup_ports.py -b vfio-pci 03:00.0 03:00.1

# Generate config file interactively
sudo ./dpdk_setup_ports.py -i
```

> [!CAUTION]
> Issue #1135 reports that `dpdk_setup_ports.py` may fail with "no Driver_str" on certain NIC models. If this occurs, manually bind the driver with `dpdk-devbind.py` from the DPDK tools or specify the PCI address directly in the configuration file.

### 4.4 Starting TRex

**Stateless mode:**
```bash
sudo ./t-rex-64 -i --cfg scripts/cfg/my_config.yaml
# -i : interactive mode (starts RPC server, waits for client connections)
# Output: shows port info, starts listening on ZMQ ports
```

**ASTF mode:**
```bash
sudo ./t-rex-64 -i --astf --cfg scripts/cfg/my_config.yaml
# --astf : start in Advanced Stateful mode
```

**Simulator mode (no hardware):**
```bash
./bp-sim-64 --stl -f scripts/stl/bench.py -o output.pcap
# Runs a profile through the simulator, writes pcap output
```

### 4.5 Multi-Instance Deployment

Multiple TRex instances can run on the same server by isolating ports and cores:

```yaml
# Instance 1: trex_a.yaml
- version: 2
  interfaces: ["03:00.0", "03:00.1"]
  prefix: trex_a
  c: 4
  platform:
    master_thread_id: 0
    latency_thread_id: 1
    dual_if:
      - socket: 0
        threads: [2, 3, 4, 5]

# Instance 2: trex_b.yaml
- version: 2
  interfaces: ["05:00.0", "05:00.1"]
  prefix: trex_b
  c: 4
  platform:
    master_thread_id: 8
    latency_thread_id: 9
    dual_if:
      - socket: 1
        threads: [10, 11, 12, 13]
```

Each instance must have a unique `prefix`, non-overlapping CPU cores, and separate NIC ports.

### 4.6 CI/CD

TRex uses Travis CI (`.travis.yml`) for continuous integration. The `BUILD_NUM` environment variable is set by Travis to track build numbers.

---

## 5. Operations

### 5.1 Connecting the Console

```bash
./trex-console

# Connect to a remote TRex instance
./trex-console -s 192.168.1.100

# Expected output:
# Connecting to TRex server at 192.168.1.100:4501
# Connected
# trex>
```

### 5.2 Python Client API (TRexClient)

The `TRexClient` class (93+ methods) is the primary interface for automation:

```python
from trex.common.trex_client import TRexClient

# Connect
client = TRexClient(server='127.0.0.1')
client.connect()

# Acquire ports
client.acquire(port_id=[0, 1])

# Start traffic (STL example)
client.start(ports=[0], mult="10gbps", duration=60)

# Get statistics
stats = client.get_stats()
print(stats[0])  # Port 0 stats

# Stop and disconnect
client.stop(ports=[0])
client.disconnect()
```

**Key TRexClient method categories (93+ methods):**

| Category | Methods |
|----------|---------|
| Connection | `connect()`, `disconnect()`, `is_connected()` |
| Port control | `acquire()`, `release()`, `reset()` |
| Traffic | `start()`, `stop()`, `pause()`, `resume()`, `update()` |
| Statistics | `get_stats()`, `get_port_stats()`, `clear_stats()` |
| Capture | `start_capture()`, `stop_capture()`, `fetch_capture_packets()` |
| Service | `set_service_mode()` |
| Configuration | `set_port_attr()`, `set_l2_mode()`, `set_l3_mode()` |
| ARP/ND | `arp()`, `scan6()` |
| VLAN | `set_vlan()`, `clear_vlan()` |
| Namespace | `set_namespace()`, `remove_all_ns()` |
| Ping | `ping_ip()` |

### 5.3 Port Object (Port Class, 52+ Methods)

Each port is represented by a `Port` object with methods for:

| Category | Methods |
|----------|---------|
| Lifecycle | `acquire()`, `release()`, `set_attr()` |
| Traffic | `start()`, `stop()`, `pause()`, `resume()`, `update()` |
| Layer config | `set_l2_mode()`, `set_l3_mode()` |
| State query | `is_acquired()`, `is_active()`, `get_profile_id()` |

### 5.4 Namespace Management (NSCmds)

The `NSCmds` class manages network namespaces for EMU (emulation) scenarios:

```python
ns = NSCmds()
ns.add_node(mac="00:00:00:01:00:00", ipv4="10.0.0.1")
ns.set_dg(ipv4="10.0.0.254")
ns.set_vlan(vlans=[100])
```

### 5.5 Service Framework

The service framework (`interactive/trex/common/services/`, 10 files) provides pluggable services that run on ports in service mode:

- ARP resolution
- DHCP client
- ICMPv6 / IPv6 Neighbor Discovery
- IGMP
- Custom service plugins

### 5.6 Console Plugins

Console plugins (`interactive/trex/console/plugins/`, 8 files) extend the TRex console:

| Plugin | Purpose |
|--------|---------|
| `plugin_IPv6ND` | IPv6 Neighbor Discovery |
| `plugin_bird` | BIRD routing daemon integration |
| `plugin_wireless` | Wireless AP simulation |
| `plugin_wlc` | Wireless LAN Controller testing |

### 5.7 Monitoring Statistics

TRex publishes real-time statistics via ZMQ PUB/SUB. The Python subscriber (`trex_subscriber.py`) processes these asynchronously. Statistics include:

- Per-port TX/RX counters (packets, bytes, errors)
- Per-stream flow statistics
- Latency histograms (via HDR Histogram, `src/hdrh/`)
- CPU utilization per core
- Memory (hugepage) usage

```python
# Get global stats
stats = client.get_stats()
# stats keys: 'global', port indices (0, 1, ...), 'flow_stats', 'latency'

print(f"TX: {stats['global']['tx_bps']:.2f} bps")
print(f"RX: {stats['global']['rx_bps']:.2f} bps")
print(f"CPU: {stats['global']['cpu_util']:.1f}%")
```

### 5.8 Gratuitous ARP

> [!WARNING]
> Issue #1162 reports that Gratuitous ARP is broken in TRex v3.06. If you rely on GARP for failover scenarios, test thoroughly or apply the fix from the issue tracker before deploying v3.06.

---

## 6. Performance

### 6.1 Performance Architecture

TRex achieves line-rate performance through:

1. **DPDK kernel bypass:** Zero-copy packet I/O directly to NIC hardware queues
2. **Per-core packet processing:** Each worker thread handles a subset of flows, no shared state contention
3. **Hugepage memory:** Pre-allocated memory pools eliminate allocation overhead
4. **NUMA awareness:** Threads pinned to the same NUMA node as their NIC
5. **Batch processing:** Packets processed in bursts (configurable RX/TX burst size)

### 6.2 Performance Tuning

**Core allocation:** More cores per port pair increases packet processing throughput. Each core can handle approximately 10-15 Mpps depending on packet size and profile complexity.

```yaml
# High-throughput configuration: 7 cores per port pair
c: 7
platform:
  master_thread_id: 0
  latency_thread_id: 15
  dual_if:
    - socket: 0
      threads: [1, 2, 3, 4, 5, 6, 7]
```

**Memory allocation:** Insufficient hugepages cause startup failure. Calculate based on:
- Per-port memory pool: ~256 MB minimum
- Per-core packet buffers: ~64 MB
- ASTF TCP stack state: ~1 KB per concurrent connection

```bash
# Check hugepage allocation
cat /proc/meminfo | grep Huge
# HugePages_Total:    1024
# HugePages_Free:      512
# Hugepagesize:       2048 kB
```

**CPU isolation:** For consistent performance, isolate TRex cores from the Linux scheduler:

```bash
# Kernel boot params
# isolcpus=1-7,17-23 nohz_full=1-7,17-23 rcu_nocbs=1-7,17-23
```

### 6.3 Latency Measurement

TRex uses HDR Histogram (`src/hdrh/`, 10 files) for high-resolution latency measurement. A dedicated latency thread (configured via `latency_thread_id`) sends and receives timestamped probe packets separately from the data plane.

```python
# Enable latency on a stream
stream = STLStream(
    packet=pkt,
    mode=STLTXCont(pps=1000),
    flow_stats=STLFlowLatencyStats(pg_id=7)
)
```

### 6.4 Packet Size Considerations

Issue #1205 reports packet truncation to 64 bytes on E810 (ICE) NICs. Verify minimum packet sizes are met for your NIC driver. Issue #1203 requests custom RX/TX burst sizes — the default burst size is compiled in; modification requires a rebuild.

---

## 7. Security

### 7.1 Privilege Model

TRex requires root privileges (or `CAP_SYS_RAWIO`, `CAP_NET_ADMIN`, `CAP_IPC_LOCK`) because DPDK needs:

- Access to PCI device files (`/sys/bus/pci/`)
- Hugepage allocation and mapping
- Direct hardware register access via `vfio-pci` or `igb_uio`

> [!CAUTION]
> Running TRex as root on a shared system exposes all PCI devices to the process. In multi-tenant environments, use VFIO with proper IOMMU groups to isolate NIC access, and restrict SSH access to the TRex host.

### 7.2 Network Security Implications

TRex can generate arbitrary packets at line rate. This makes it a powerful tool that can cause network disruption if misconfigured:

- **MAC spoofing:** TRex sets arbitrary source MACs. Ensure upstream switches permit this or configure `src_mac` in `port_info`.
- **IP spoofing:** Source IPs in traffic profiles are user-defined. Restrict TRex to isolated test networks.
- **Denial of service:** Line-rate traffic generation can overwhelm any network device. Physical isolation of test networks is strongly recommended.
- **SYN flood capability:** Profiles like `scripts/stl/syn_attack.py` exist for testing purposes. Ensure these are only used in controlled environments.

### 7.3 Control Plane Security

The JSON-RPC server listens on ZMQ sockets (default ports 4500-4501). There is no built-in authentication or encryption:

- **Network isolation:** Bind TRex management to a dedicated management VLAN or loopback interface
- **Firewall:** Restrict access to ZMQ ports (4500-4501 and the ZMQ publisher port) to authorized clients only
- **No TLS:** ZMQ transport is plaintext. Do not expose control ports to untrusted networks

```bash
# Firewall example: restrict TRex control to management subnet
iptables -A INPUT -p tcp --dport 4500:4501 -s 10.0.100.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 4500:4501 -j DROP
```

### 7.4 Bundled Dependencies

TRex bundles the following external libraries, which should be tracked for CVEs:

| Library | Version (bundled) | Purpose |
|---------|-------------------|---------|
| Scapy | 2.4.3 | Packet crafting (Python) |
| ZMQ (libzmq) | — | IPC / control plane |
| nlohmann JSON | — | C++ JSON parsing |
| yaml-cpp | — | YAML config parsing |
| valijson | — | JSON schema validation |
| BPF/bpfjit | — | Packet filtering |
| ibverbs | — | RDMA/InfiniBand support |
| libmnl | — | Netlink interface |

> [!IMPORTANT]
> The bundled Scapy 2.4.3 is a known-old version. While it runs only on the client side (not in the data plane), ensure your Python environment does not have conflicting Scapy installations. Other bundled Python libraries include `jsonrpclib`, `jsonpickle`, `requests`, `urllib3`, `pyyaml`, `nose`, `progressbar`, `texttable`, and `simpy`.

### 7.5 File System Security

- Configuration files in `scripts/cfg/` may contain MAC addresses and IP topology — treat as sensitive in production environments
- Packet capture files (`scripts/cap2/`) may contain real traffic patterns
- The TRex binary requires SUID root or equivalent capabilities

---

## 8. Developer Internals

### 8.1 Source Code Map

**Core engine entry (`src/main.cpp`):**

The main entry point parses command-line arguments and dispatches to one of five simulator modes:

| Mode | Simulator Class | Purpose |
|------|----------------|---------|
| STF | `SimStateful` | Legacy stateful simulation |
| STL | `SimStateless` | Stateless simulation |
| ASTF | `SimAstf` | Advanced stateful (full stack) |
| ASTF Simple | `SimAstfSimple` | Simplified ASTF (batch) |
| GTEST | `SimGtest` | Google Test runner |

### 8.2 STX Framework Architecture

The STX framework (`src/stx/`) provides a unified abstraction layer across all traffic modes. The `src/stx/common/` directory (50 files) contains shared infrastructure:

- Port management and state machines
- Memory pool allocation
- Packet buffer management
- Statistics collection
- Timer wheel for scheduled events

Each mode extends STX with mode-specific logic:
- `src/stx/stl/`: Stream table, field engine VM, multi-burst scheduler
- `src/stx/astf/`: TCP template loading, connection pool, flow table
- `src/stx/stf/`: Legacy template-based replay

### 8.3 FreeBSD TCP/IP Stack (src/44bsd/)

The ASTF mode embeds a FreeBSD-derived TCP/IP stack (24 files, 347K characters) providing:

- Full TCP state machine (SYN, ESTABLISHED, FIN_WAIT, TIME_WAIT, etc.)
- Congestion control (Reno, NewReno variants)
- Retransmission timers
- Window scaling
- Selective acknowledgment (SACK)
- TCP segmentation

This stack runs entirely in userspace, integrated with DPDK's memory and packet buffer system. Each worker thread has its own stack instance — no locking required.

### 8.4 RPC Server (src/rpc-server/)

The JSON-RPC server (13 files) handles control-plane commands:

```
Client (Python) ──ZMQ REQ──▶ RPC Server ──dispatch──▶ Command Handler
                                                              │
                                                              ▼
                                                        STX Framework
                                                     (start/stop/config)
```

Commands are validated against JSON schemas (using `valijson`). Each command maps to a handler function that interacts with the STX framework.

### 8.5 Statistics Publisher (src/publisher/)

Two files implement the ZMQ PUB statistics channel:

- Periodic stats broadcast (configurable interval)
- Port counters, flow stats, latency histograms
- Global CPU and memory utilization
- Consumed by `trex_subscriber.py` on the client side

### 8.6 NIC Driver Layer (src/drivers/)

The 26 driver files extend or patch DPDK drivers for TRex-specific functionality:

- Custom RSS (Receive Side Scaling) configuration
- Hardware flow director rules
- MTU handling (see issue #1204 re: MTU set to 65518)
- Interrupt coalescing tuning
- Driver-specific workarounds

### 8.7 Platform Abstraction Layer (src/pal/)

PAL provides two implementations:

| PAL | Target | Purpose |
|-----|--------|---------|
| Linux | Simulator builds | POSIX-based, no DPDK dependency |
| Linux-DPDK | Production builds | Full DPDK integration |

This allows the same core logic to compile for both simulation testing and production deployment.

### 8.8 Python Client Architecture

```
trex_control_plane/interactive/trex/
├── common/
│   ├── trex_client.py          # TRexClient (93+ methods)
│   ├── trex_conn.py            # JsonRpcClient connection
│   ├── trex_port.py            # Port (52+ methods)
│   ├── trex_ns.py              # NSCmds namespace management
│   ├── trex_subscriber.py      # Async ZMQ stats subscriber
│   ├── trex_events.py          # Event system
│   ├── trex_logger.py          # Logger (ScreenLogger)
│   ├── trex_types.py           # RC, VLAN, PortProfileID
│   ├── trex_psv.py             # Port state validation
│   ├── trex_exceptions.py      # Exception hierarchy
│   ├── trex_vlan.py            # VLAN handling
│   ├── trex_api_annotators.py  # API decorators
│   └── services/               # Service framework (10 files)
├── stl/                         # STL-specific client
├── astf/                        # ASTF-specific client
├── emu/                         # EMU client
├── wireless/                    # Wireless testing
├── console/
│   └── plugins/                 # Console plugins (8 files)
└── utils/                       # Utilities (9 files)
```

**Exception hierarchy:**

| Exception | Use Case |
|-----------|----------|
| `TRexError` | General errors (connection lost, invalid port, timeouts) |
| `TRexArgumentError` | Invalid method arguments |
| `TRexTypeError` | Type mismatches in API calls |
| `TRexTimeoutError` | Operation timeout (ZMQ, connection) |
| `TRexConsoleError` | Console-specific failures |
| `ASTFErrorBadIp` | Invalid IP address in ASTF profile |

### 8.9 Traffic Profile System

**STL Profiles (74 files in `scripts/stl/`):**

STL profiles define packet templates and field engines. Key profile types:

| Profile | Purpose |
|---------|---------|
| `bench.py` | Performance benchmarking |
| `burst.py` | Burst traffic patterns |
| `flow_stats*.py` | Flow statistics testing |
| `imix*.py` | Internet Mix traffic patterns |
| `pcap*.py` | PCAP-based replay |
| `syn_attack.py` | SYN flood (testing only) |
| `udp_*.py` | UDP traffic variants |
| `hlt/` | High-Level Traffic API profiles |

STL profile classes (e.g., `STLS1`, `STLBench`, `STLImix`, `STLPcap`) extend a base class and implement `get_streams()` to return stream definitions.

**ASTF Profiles (97 files in `scripts/astf/`):**

ASTF profiles define client/server application templates:

| Profile | Purpose |
|---------|---------|
| `http_eflow*.py` | HTTP traffic with extended flows |
| `ftp*.py` | FTP session simulation |
| `udp*.py` | UDP application patterns |
| `video_stream.py` | Video streaming simulation |

**EMU Profiles (23 files in `scripts/emu/`):**

Network protocol emulation:

| Profile | Purpose |
|---------|---------|
| `dhcp*.py` | DHCP client emulation |
| `dns*.py` | DNS query patterns |
| `mdns*.py` | mDNS (multicast DNS) |
| `icmp*.py` | ICMP echo/response |
| `ipfix*.py` | IPFIX/NetFlow export |

### 8.10 Build System Internals

The `build_option` class (31+ methods) in `ws_main.py` manages:

- Compiler detection and version validation
- DPDK library compilation and linking
- Platform-specific flags (x86_64, aarch64, ppc64le)
- Sanitizer integration (ASan, MSan)
- External library paths
- Static linking configuration

The build produces a single statically-linked binary (`_t-rex-64`) that embeds DPDK, all NIC drivers, the TCP/IP stack, and the RPC server.

### 8.11 Tunnel Support (src/tunnels/)

Seven files implement tunnel encapsulation/decapsulation:

- GTP-U (GPRS Tunneling Protocol) — mobile network testing
- GRE (Generic Routing Encapsulation)
- Custom tunnel handlers

### 8.12 Testing

**C++ tests (`src/gtest/`, 11 files):** Google Test-based unit and integration tests. Run via the GTEST simulator mode:

```bash
./bp-sim-64 --gtest
```

**Python tests:** Profile validation via simulator mode:

```bash
./bp-sim-64 --stl -f scripts/stl/bench.py -o /dev/null
```

---

## 9. Integration Points

### 9.1 Automation Integration

TRex is designed for integration with test automation frameworks:

```python
# Robot Framework / pytest integration pattern
from trex.stl.trex_stl_client import STLClient

def run_traffic_test(server, duration, rate):
    client = STLClient(server=server)
    try:
        client.connect()
        client.reset()
        client.add_streams(profile, ports=[0])
        client.start(ports=[0], mult=rate, duration=duration)
        client.wait_on_traffic()
        stats = client.get_stats()
        return stats
    finally:
        client.disconnect()
```

### 9.2 Monitoring Integration

ZMQ PUB/SUB statistics can be consumed by external monitoring systems:

- Subscribe to the ZMQ publisher port (default 4500)
- Parse JSON-formatted statistics messages
- Feed into Grafana, Prometheus (via custom exporter), or ELK

### 9.3 Docker Integration

```dockerfile
# Example: running TRex in Docker
FROM ubuntu:16.04
# ... install dependencies ...
COPY trex_cfg.yaml /etc/trex_cfg.yaml
CMD ["./t-rex-64", "-i", "--cfg", "/etc/trex_cfg.yaml"]
```

Docker configurations in `docker/ubuntu/` provide reference Dockerfiles.

### 9.4 Known Integration Issues

| Issue | Description | Impact |
|-------|-------------|--------|
| #919 | DPDK map error | Startup failure on certain memory configurations |
| #934 | `ixgbe_dev_start` failure | ixgbe NIC init failure |
| #1127 | X710 driver bind error | i40e NIC binding failure |
| #1162 | Gratuitous ARP broken in v3.06 | Network failover affected |
| #1183 | Unable to start in ASTF | ASTF mode startup failure |
| #1185 | 400G NIC issues | High-speed NIC support gaps |
| #1186 | IPv4 ID changed by TRex | Unexpected packet modification |
| #1202 | No ICE driver patches since v3.04 | E810 stability concerns |
| #1204 | MTU set to 65518 | Unexpected MTU behavior |
| #1205 | Packet truncate to 64B (E810) | Small packet loss on ICE |
| #1207 | Stop traffic abnormalities | Traffic stop behavior issues |
| #1208 | Method not found `ctx_stop_autotriggers` | RPC method missing |
| #1214 | Missing v3.08 tag | Release management gap |

---

## 10. Troubleshooting

### 10.1 Startup Failures

**Error:** `DPDK map error`
- **Cause:** Insufficient hugepages or incorrect IOMMU configuration (issue #919)
- **Solution:** Verify hugepage allocation (`cat /proc/meminfo | grep Huge`), check IOMMU is enabled in BIOS and kernel params
- **Prevention:** Pre-allocate hugepages in `/etc/sysctl.conf` and verify at boot

**Error:** `ixgbe_dev_start failure`
- **Cause:** NIC initialization failure, often due to firmware mismatch or driver conflict (issue #934)
- **Solution:** Update NIC firmware, ensure kernel ixgbe module is unloaded before binding to DPDK
- **Prevention:** Keep NIC firmware up to date, use `dpdk_setup_ports.py` for proper binding

**Error:** `Unable to start in ASTF`
- **Cause:** ASTF mode requires specific configuration and port pair setup (issue #1183)
- **Solution:** Ensure ports are configured in pairs with valid IP addresses and gateways. Verify `--astf` flag is passed.
- **Prevention:** Test ASTF configuration with the simulator build first

**Error:** `no Driver_str in dpdk_setup_ports.py`
- **Cause:** NIC model not recognized by the setup script (issue #1135)
- **Solution:** Manually bind the NIC using DPDK's `dpdk-devbind.py`, or update `dpdk_setup_ports.py`
- **Prevention:** Use well-supported NIC models (X710, ConnectX-5)

### 10.2 Runtime Issues

**Error:** `Method not found ctx_stop_autotriggers`
- **Cause:** Client/server version mismatch — the RPC method does not exist in the running TRex version (issue #1208)
- **Solution:** Ensure the Python client library version matches the TRex server version
- **Prevention:** Deploy client and server from the same release package

**Error:** Connection lost / TRexTimeoutError
- **Cause:** ZMQ connection timeout, TRex process crash, or firewall blocking ports
- **Solution:** Check TRex process is running (`ps aux | grep t-rex`), verify firewall rules for ZMQ ports, check system logs for crash dumps
- **Prevention:** Monitor TRex process health, use `client.is_connected()` in automation

**Error:** Packets truncated to 64 bytes
- **Cause:** E810 (ICE) driver issue (issue #1205)
- **Solution:** Set minimum packet size to 64 bytes in profiles, or use a NIC with a more mature driver
- **Prevention:** Test with your specific NIC model before production deployment

> [!NOTE]
> TRex writes crash dumps and logs to the working directory. When reporting issues, include the log output, platform configuration YAML, and the output of `dpdk_setup_ports.py -s`.

### 10.3 Performance Issues

**Symptom:** TX rate not reaching target
- **Cause:** Insufficient cores, NUMA cross-socket access, or memory pressure
- **Solution:** Add more cores (`c` field), pin threads to correct NUMA node, increase hugepage allocation
- **Prevention:** Follow NUMA-aware configuration (Section 3.4)

**Symptom:** High latency variance
- **Cause:** Latency thread sharing a core with data plane, or CPU frequency scaling
- **Solution:** Dedicate a separate core for latency (`latency_thread_id`), disable CPU frequency scaling (`cpupower frequency-set -g performance`)
- **Prevention:** Include latency thread in initial platform configuration

### 10.4 Build Issues

**Error:** Compiler version too old
- **Cause:** GCC below 4.7.0 or sanitizer build with GCC below 4.9.0
- **Solution:** Install appropriate GCC version, set `CXX` environment variable
- **Prevention:** Use a supported OS with GCC 7.4+ from package manager

---

## 11. Quick Reference

### 11.1 Common Commands

| Task | Command |
|------|---------|
| Start (stateless) | `sudo ./t-rex-64 -i --cfg config.yaml` |
| Start (ASTF) | `sudo ./t-rex-64 -i --astf --cfg config.yaml` |
| Console connect | `./trex-console` |
| Console (remote) | `./trex-console -s <ip>` |
| NIC status | `sudo ./dpdk_setup_ports.py -s` |
| Bind NIC | `sudo ./dpdk_setup_ports.py -b vfio-pci <pci>` |
| Simulate STL | `./bp-sim-64 --stl -f profile.py -o out.pcap` |
| Run tests | `./bp-sim-64 --gtest` |
| Build (DPDK) | `cd linux_dpdk && ./configure && make` |
| Build (sim) | `cd linux && ./configure && make` |
| Build (sanitized) | `cd linux_dpdk && ./configure --sanitized && make` |

### 11.2 Default Ports

| Port | Protocol | Purpose |
|------|----------|---------|
| 4500 | ZMQ | Statistics publisher (PUB/SUB) |
| 4501 | ZMQ | RPC server (REQ/REP) |

### 11.3 File Locations

| File/Directory | Purpose |
|----------------|---------|
| `scripts/cfg/` | Configuration examples (15 files) |
| `scripts/stl/` | Stateless profiles (74 files) |
| `scripts/astf/` | ASTF profiles (97 files) |
| `scripts/emu/` | EMU profiles (23 files) |
| `scripts/cap2/` | Packet captures (62 files) |
| `scripts/automation/trex_control_plane/` | Python client library |
| `src/` | C++ source (core engine) |
| `external_libs/` | Bundled Python dependencies |

### 11.4 Configuration Checklist

- [ ] Hugepages allocated and mounted
- [ ] IOMMU enabled (for vfio-pci)
- [ ] NICs unbound from kernel driver
- [ ] NICs bound to vfio-pci or igb_uio
- [ ] YAML config created with correct PCI addresses
- [ ] CPU cores pinned to correct NUMA node
- [ ] Latency thread on dedicated core
- [ ] Management interface NOT bound to DPDK
- [ ] Firewall rules for ZMQ ports (if remote access needed)
- [ ] Test network physically isolated from production

*For installation steps, profile creation, and usage walkthroughs, see the [User Guide](/cisco-system-traffic-generator/trex-core/user-guide/).*

---

_Generated May 2026 from cisco-system-traffic-generator/trex-core at 27e0153. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._