# TRex Traffic Generator — User Guide

TRex is Cisco's open-source, high-performance network traffic generator built on top of Intel DPDK. It generates realistic network traffic at line rate — up to 200+ Gbps on commodity hardware — for testing routers, firewalls, load balancers, IDS/IPS, and other network infrastructure. TRex operates in three distinct modes: Stateless (STL) for raw packet streams, Advanced Stateful (ASTF) for realistic TCP/UDP application traffic, and Emulation (EMU) for protocol-level client simulation.

This guide covers everything you need to install, configure, and operate TRex across all three modes, including the interactive console, Python automation API, traffic profiles, and real-world workflow recipes.

> [!NOTE]
> For architecture details, deployment planning, security hardening, performance tuning internals, and contributor workflows, see the **TRex Admin Guide**.

---

## Table of Contents

- [1. Quick Start](#1-quick-start)
- [2. Installation](#2-installation)
- [3. Configuration](#3-configuration)
- [4. Core Usage](#4-core-usage)
- [5. Common Tasks](#5-common-tasks)
- [6. Troubleshooting](#6-troubleshooting)
- [7. FAQ](#7-faq)

---

## 1. Quick Start

This section gets you from zero to generating traffic in the shortest path possible. You will need a Linux machine with at least two DPDK-compatible network interfaces.

### 1.1 Minimum Requirements

| Requirement | Minimum |
|---|---|
| OS | Linux (Ubuntu 18.04+, CentOS 7+) |
| Kernel | 2.6.32+ |
| CPU | x86_64 with SSE 4.2 |
| RAM | 4 GB (8+ GB recommended) |
| NICs | 2 DPDK-compatible ports (Intel X710, XL710, E810, ixgbe, i40e, etc.) |
| Hugepages | 1024 x 2MB (or equivalent 1GB pages) |

### 1.2 Five-Minute Quick Start

**Step 1: Download and extract TRex.**

```bash
wget --no-cache --no-check-certificate https://trex-tgn.cisco.com/trex/release/latest
tar -xzvf latest
cd v3.0x   # version directory
```

**Step 2: Set up hugepages and bind NICs.**

```bash
sudo ./dpdk_setup_ports.py -i
```

This interactive script walks you through configuring hugepages and binding network interfaces to DPDK-compatible drivers (`igb_uio`, `vfio-pci`).

**Step 3: Create a minimal config file.**

```yaml
# /etc/trex_cfg.yaml
- port_limit: 2
  version: 2
  interfaces: ["03:00.0", "03:00.1"]
  port_info:
    - dest_mac: "00:00:00:01:00:00"
      src_mac: "00:00:00:02:00:00"
    - dest_mac: "00:00:00:02:00:00"
      src_mac: "00:00:00:01:00:00"
```

Replace the PCI addresses and MAC addresses with your actual hardware values.

**Step 4: Start the TRex server in stateless mode.**

```bash
sudo ./t-rex-64 -i --cfg /etc/trex_cfg.yaml
```

**Step 5: Connect the console and send traffic.**

In a second terminal:

```bash
./trex-console

trex> start -f stl/bench.py -m 1gbps -p 0
trex> stats
trex> stop -a
```

You are now generating 1 Gbps of stateless traffic on port 0.

> [!TIP]
> Use `./t-rex-64 --help` to see all server flags and `trex-console -h` for console options. The console supports tab completion for commands and arguments.

---

## 2. Installation

### 2.1 Supported Platforms

TRex runs on Linux x86_64 systems. The primary build system is `waf`, with separate build configurations for DPDK-based builds (`linux_dpdk/ws_main.py`) and non-DPDK builds (`linux/ws_main.py`).

### 2.2 Pre-Built Binary Installation

The fastest path. Download the latest release tarball:

```bash
mkdir /opt/trex
cd /opt/trex
wget --no-cache --no-check-certificate https://trex-tgn.cisco.com/trex/release/latest
tar -xzvf latest
```

The extracted directory contains:
- `t-rex-64` — the main TRex binary
- `bp_sim` — alias/alternate name for the same binary
- `trex-console` — the interactive Python console
- `scripts/` — profiles, automation libraries, config examples
- `dpdk_setup_ports.py` — NIC binding and hugepage setup utility

### 2.3 Building from Source

If you need a custom build (e.g., specific DPDK patches, debug builds):

```bash
git clone https://github.com/cisco-system-traffic-generator/trex-core.git
cd trex-core

# DPDK-based build (production)
cd linux_dpdk
./b configure
./b build

# Non-DPDK build (simulation/testing)
cd linux
./b configure
./b build
```

The build system uses `waf`. The `CXX` environment variable selects the C++ compiler:

```bash
export CXX=g++-9
cd linux_dpdk && ./b configure && ./b build
```

> [!IMPORTANT]
> The `BUILD_NUM` environment variable is used internally for build numbering. The `USER` environment variable is read by the build system for paths. These are the only environment variables TRex uses — there are no runtime env-var knobs for traffic generation behavior.

### 2.4 Hugepage Configuration

TRex requires Linux hugepages for DPDK memory. Configure them before starting TRex:

**Temporary (until reboot):**

```bash
# 2MB hugepages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Mount hugepage filesystem
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
```

**Persistent (survives reboot):**

Add to `/etc/default/grub`:

```
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4"
```

Then run `update-grub` and reboot. Verify:

```bash
grep Huge /proc/meminfo
cat /proc/cmdline | grep huge
```

### 2.5 NIC Binding with dpdk_setup_ports.py

The `dpdk_setup_ports.py` script manages binding NICs to DPDK-compatible drivers:

```bash
# Interactive setup (recommended for first time)
sudo ./dpdk_setup_ports.py -i

# Show current NIC status
sudo ./dpdk_setup_ports.py -s

# Auto-configure based on existing /etc/trex_cfg.yaml
sudo ./dpdk_setup_ports.py
```

> [!WARNING]
> Binding a NIC to DPDK removes it from the Linux kernel network stack. Do not bind your management interface. Always keep at least one interface on the kernel driver for SSH/management access.

### 2.6 Python Client Library Installation

The TRex Python client library lives in `scripts/automation/trex_control_plane/`. To use it from external scripts:

```bash
# Add to your Python path
export PYTHONPATH=/opt/trex/v3.0x/scripts/automation/trex_control_plane/interactive

# Or install the package
cd scripts/automation/trex_control_plane
pip install .
```

Verify the import:

```python
from trex.stl.trex_stl_lib.api import *
print("STL API loaded successfully")
```

### 2.7 VM / Lab Installation

For testing in a VM where you lack physical NICs, TRex can run with virtual interfaces (e.g., virtio, vmxnet3). Key differences:

- Set `port_bandwidth_gb: 1` in your config (VMs cannot do line rate)
- Use `--software` mode if the virtual NIC lacks DPDK support
- Hugepages still required inside the VM

```yaml
- port_limit: 2
  version: 2
  interfaces: ["03:00.0", "03:00.1"]
  port_bandwidth_gb: 1
  port_info:
    - ip: 1.1.1.1
      default_gw: 2.2.2.2
    - ip: 2.2.2.2
      default_gw: 1.1.1.1
```

---

## 3. Configuration

### 3.1 Config File Location and Format

TRex configuration is a YAML file, typically stored at `/etc/trex_cfg.yaml`. You can specify an alternate location with the `--cfg` flag:

```bash
sudo ./t-rex-64 -i --cfg /path/to/my_config.yaml
```

Example config files are in `scripts/cfg/`. The config is a YAML list (note the leading dash) where each item configures a TRex instance.

### 3.2 Complete Config Reference

| Key | Type | Default | Description |
|---|---|---|---|
| `port_limit` | int | — | Number of ports (2 or 4) |
| `version` | int | 2 | Config file version (always 2) |
| `interfaces` | list[str] | — | PCI addresses, e.g. `["03:00.0","03:00.1"]` |
| `port_info` | list[obj] | — | Per-port MAC or IP configuration |
| `port_bandwidth_gb` | int | auto | Port bandwidth: 1 (VM), 10 (10G NIC), 40 (XL710) |
| `enable_zmq_pub` | bool | false | Enable ZMQ publisher for stats |
| `zmq_pub_port` | int | 4500 | ZMQ publisher port |
| `zmq_rpc_port` | int | 4501 | ZMQ RPC port (console connects here) |
| `telnet_port` | int | 4502 | Telnet access port |
| `prefix` | str | — | Instance prefix for multi-instance setups |
| `limit_memory` | int | — | Memory limit in MB |
| `c` | int | — | Number of datapath cores |
| `scan_only_1g` | bool | false | Only scan 1G interfaces |

**Port info — MAC-based (loopback or direct cable):**

```yaml
port_info:
  - dest_mac: "00:00:00:01:00:00"
    src_mac:  "00:00:00:02:00:00"
  - dest_mac: "00:00:00:02:00:00"
    src_mac:  "00:00:00:01:00:00"
```

**Port info — IP-based (routed setup through DUT):**

```yaml
port_info:
  - ip: 10.0.0.1
    default_gw: 10.0.0.2
  - ip: 10.1.0.1
    default_gw: 10.1.0.2
```

### 3.3 Platform Section — CPU Pinning

For production deployments, pin TRex threads to specific CPU cores to avoid NUMA crossings and ensure deterministic performance:

```yaml
- port_limit: 2
  version: 2
  interfaces: ["03:00.0", "03:00.1"]
  port_info:
    - ip: 10.0.0.1
      default_gw: 10.0.0.2
    - ip: 10.1.0.1
      default_gw: 10.1.0.2
  platform:
    master_thread_id: 0
    latency_thread_id: 1
    dual_if:
      - socket: 0
        threads: [2, 3, 4, 5]
```

| Platform Key | Type | Description |
|---|---|---|
| `platform.master_thread_id` | int | CPU core for the master (control) thread |
| `platform.latency_thread_id` | int | CPU core for the latency measurement thread |
| `platform.dual_if[].socket` | int | NUMA socket for this interface pair |
| `platform.dual_if[].threads` | list[int] | CPU cores for datapath threads |

> [!TIP]
> Run `lscpu` and `cat /proc/cpuinfo` to identify NUMA topology. Place TRex threads on the same NUMA node as the NICs for best performance. Use `lstopo` from the `hwloc` package for a visual map.

### 3.4 Memory Section

```yaml
memory:
  dp_flows: 1048576
```

| Key | Type | Description |
|---|---|---|
| `memory.dp_flows` | int | Number of datapath flow entries |

### 3.5 Multi-Instance Configuration

Run multiple TRex instances on the same server using the `prefix` key and different ports:

```yaml
# Instance A — /etc/trex_cfg_a.yaml
- port_limit: 2
  version: 2
  prefix: instance_a
  interfaces: ["03:00.0", "03:00.1"]
  zmq_pub_port: 4500
  zmq_rpc_port: 4501
  telnet_port: 4502
  port_info:
    - dest_mac: "00:00:00:01:00:00"
      src_mac:  "00:00:00:02:00:00"
    - dest_mac: "00:00:00:02:00:00"
      src_mac:  "00:00:00:01:00:00"
```

```yaml
# Instance B — /etc/trex_cfg_b.yaml
- port_limit: 2
  version: 2
  prefix: instance_b
  interfaces: ["05:00.0", "05:00.1"]
  zmq_pub_port: 4510
  zmq_rpc_port: 4511
  telnet_port: 4512
  port_info:
    - dest_mac: "00:00:00:03:00:00"
      src_mac:  "00:00:00:04:00:00"
    - dest_mac: "00:00:00:04:00:00"
      src_mac:  "00:00:00:03:00:00"
```

### 3.6 Example Configs by Scenario

**Lab / VM — Two virtual NICs, minimal resources:**

```yaml
- port_limit: 2
  version: 2
  interfaces: ["03:00.0", "03:00.1"]
  port_bandwidth_gb: 1
  limit_memory: 1024
  c: 1
  port_info:
    - ip: 1.1.1.1
      default_gw: 2.2.2.2
    - ip: 2.2.2.2
      default_gw: 1.1.1.1
```

**Production — Four XL710 40G ports, pinned CPUs:**

```yaml
- port_limit: 4
  version: 2
  interfaces: ["03:00.0", "03:00.1", "05:00.0", "05:00.1"]
  port_bandwidth_gb: 40
  enable_zmq_pub: true
  zmq_pub_port: 4500
  port_info:
    - ip: 10.0.0.1
      default_gw: 10.0.0.2
    - ip: 10.1.0.1
      default_gw: 10.1.0.2
    - ip: 10.2.0.1
      default_gw: 10.2.0.2
    - ip: 10.3.0.1
      default_gw: 10.3.0.2
  platform:
    master_thread_id: 0
    latency_thread_id: 1
    dual_if:
      - socket: 0
        threads: [2, 3, 4, 5, 6, 7]
      - socket: 1
        threads: [8, 9, 10, 11, 12, 13]
```

---

## 4. Core Usage

### 4.1 The TRex Binary (`t-rex-64`)

The `t-rex-64` binary is the TRex server/daemon. It manages DPDK, NIC ports, and the datapath. All traffic generation commands flow through this process.

#### 4.1.1 Server Startup Flags

| Flag | Description |
|---|---|
| `-i` | Interactive mode — wait for console/API connections |
| `--cfg <file>` | Path to YAML config file |
| `-f <file>` | Traffic profile file (for direct/batch mode) |
| `-o <file>` | Output capture file (.erf format) |
| `-d <seconds>` | Duration to run |
| `-v [1-3]` | Verbosity level (1=basic, 2=detailed, 3=debug) |
| `--pcap` | Enable pcap export |
| `--ipv6` | Enable IPv6 mode |
| `--sl` | Stateless mode |
| `--tcp_cfg` | ASTF mode |
| `--ut` | Unit test mode |
| `--cores <N>` | Number of datapath cores (1-8) |
| `--core_index <N>` | Use a specific core index |
| `--limit <N>` | Packet limit (stop after N packets) |
| `--dry` | Dry run — parse profile but do not send traffic |
| `--full` | Full ASTF simulation |

**ASTF simulator-specific flags:**

| Flag | Description |
|---|---|
| `--shaper-rate <bps>` | Traffic shaper rate |
| `--shaper-size <bytes>` | Traffic shaper buffer size |
| `--rtt <usec>` | Simulated round-trip time |
| `--drop <rate>` | Simulated packet drop rate |

#### 4.1.2 Starting TRex in Different Modes

**Stateless interactive (most common):**

```bash
sudo ./t-rex-64 -i --sl --cfg /etc/trex_cfg.yaml
```

**ASTF interactive:**

```bash
sudo ./t-rex-64 -i --tcp_cfg --cfg /etc/trex_cfg.yaml
```

**Batch mode (run a profile for a duration, then exit):**

```bash
sudo ./t-rex-64 -f stl/bench.py -d 60 --cfg /etc/trex_cfg.yaml
```

**Dry run (validate a profile without sending traffic):**

```bash
sudo ./t-rex-64 -f stl/bench.py --dry
```

**Packet capture to file:**

```bash
sudo ./t-rex-64 -f stl/bench.py -d 30 -o capture.erf --pcap
```

### 4.2 The TRex Console (`trex-console`)

The interactive console is a Python-based CLI that connects to a running TRex server via ZMQ.

```bash
./trex-console
```

#### 4.2.1 Console Connection Options

```bash
# Connect to remote TRex server
./trex-console -s 192.168.1.100

# Connect to specific RPC port
./trex-console -s 192.168.1.100 -p 4501

# Connect in read-only mode
./trex-console --readonly
```

#### 4.2.2 Essential Console Commands

**Server & Port Management:**

| Command | Description |
|---|---|
| `connect` | Connect/reconnect to TRex server |
| `disconnect` | Disconnect from server |
| `acquire` | Acquire port ownership |
| `release` | Release port ownership |
| `reset` | Reset all ports to idle |
| `portattr` | Show/set port attributes |
| `l1_config` | Layer 1 port configuration |

**Traffic Control:**

| Command | Description |
|---|---|
| `start` | Start traffic on port(s) |
| `stop` | Stop traffic on port(s) |
| `pause` | Pause traffic |
| `resume` | Resume paused traffic |
| `update` | Update multiplier on running traffic |
| `push` | Push a pcap file |

**Monitoring & Stats:**

| Command | Description |
|---|---|
| `stats` | Show port statistics |
| `streams` | Show configured streams |
| `tui` | Full-screen text UI dashboard |
| `clear` | Clear statistics |

**Capture & Debug:**

| Command | Description |
|---|---|
| `capture` | Start/stop/show packet captures |
| `resolve` | Resolve ARP for port(s) |
| `ping` | Ping from TRex port |
| `scan6` | IPv6 neighbor discovery scan |

**Utility:**

| Command | Description |
|---|---|
| `history` | Show command history |
| `verbose` | Set verbosity level |
| `quit` | Exit the console |

#### 4.2.3 Console Start Command Syntax

The `start` command is the primary way to begin traffic generation:

```
start -f <profile> [-m <multiplier>] [-p <ports>] [-d <duration>] [--force]
      [-t <key>=<val> ...] [--total] [--dry]
```

**Multiplier formats:**

| Format | Example | Meaning |
|---|---|---|
| Raw | `-m 1000` | 1000 pps per stream |
| pps | `-m 10kpps` | 10,000 packets/sec |
| bps | `-m 1gbps` | 1 Gbps |
| percentage | `-m 50%` | 50% of line rate |

**Examples:**

```bash
# Start a profile at 1 Gbps on port 0
trex> start -f stl/bench.py -m 1gbps -p 0

# Start on both ports at 10,000 pps for 60 seconds
trex> start -f stl/bench.py -m 10kpps -p 0 1 -d 60

# Start with profile tunables
trex> start -f stl/bench.py -m 1gbps -t size=256 vm=var1

# Update running traffic to 2 Gbps
trex> update -m 2gbps -p 0

# Stop all ports
trex> stop -a
```

### 4.3 Console Plugins

The console supports plugins for specialized testing scenarios.

#### 4.3.1 IPv6 Neighbor Discovery Plugin

```bash
trex> plugins load IPv6ND
trex> ipv6nd start -p 0
```

Sends IPv6 Router Solicitation and processes Router Advertisements. Useful for testing IPv6 network equipment.

#### 4.3.2 BIRD Routing Plugin

```bash
trex> plugins load bird
```

Integrates with the BIRD routing daemon for testing routing protocols (BGP, OSPF, etc.) alongside traffic generation.

#### 4.3.3 Wireless Testing Plugins

```bash
# Generic wireless testing
trex> plugins load wireless

# Wireless LAN Controller testing
trex> plugins load wlc
```

These plugins add specialized commands for testing wireless network infrastructure, including association, authentication, and data forwarding.

### 4.4 Python API

The Python API provides full programmatic control over TRex. Use it for automated testing, CI/CD integration, and custom traffic generation scripts.

#### 4.4.1 STL API

```python
from trex.stl.trex_stl_lib.api import *

# Connect to TRex server
client = STLClient(server="127.0.0.1")
client.connect()

# Acquire ports
client.acquire(ports=[0, 1])
client.reset(ports=[0, 1])

# Load a profile
profile = STLProfile.load("stl/bench.py", size=64)
client.add_streams(profile.get_streams(), ports=[0])

# Start traffic at 1 Gbps for 30 seconds
client.start(ports=[0], mult="1gbps", duration=30)

# Wait for traffic to finish
client.wait_on_traffic(ports=[0])

# Get stats
stats = client.get_stats()
print(f"TX packets: {stats[0]['opackets']}")
print(f"RX packets: {stats[1]['ipackets']}")
print(f"TX rate: {stats[0]['tx_bps']:.2f} bps")

# Disconnect
client.disconnect()
```

#### 4.4.2 ASTF API

```python
from trex.astf.trex_astf_lib.api import *

client = ASTFClient(server="127.0.0.1")
client.connect()
client.acquire(ports=[0, 1])
client.reset()

# Load an ASTF profile
client.load_profile("astf/http_simple.py")

# Start the ASTF traffic
client.start(mult=1, duration=60)
client.wait_on_traffic()

# Get ASTF-specific stats
stats = client.get_traffic_stats()
print(f"Total flows: {stats['total']['m_active_flows']}")

client.stop()
client.disconnect()
```

#### 4.4.3 Key TRexClient Methods

The `TRexClient` base class provides 93+ methods. The most commonly used:

| Method | Description |
|---|---|
| `connect()` | Connect to TRex server |
| `disconnect()` | Disconnect |
| `acquire(ports)` | Acquire port ownership |
| `release(ports)` | Release ports |
| `reset(ports)` | Reset ports to defaults |
| `add_streams(streams, ports)` | Add stream objects to ports |
| `remove_all_streams(ports)` | Remove all streams |
| `start(ports, mult, duration)` | Start traffic |
| `stop(ports)` | Stop traffic |
| `update(ports, mult)` | Change rate on running traffic |
| `pause(ports)` | Pause traffic |
| `resume(ports)` | Resume traffic |
| `wait_on_traffic(ports)` | Block until traffic finishes |
| `get_stats()` | Get port statistics |
| `clear_stats(ports)` | Clear statistics |
| `set_port_attr(...)` | Set port attributes (promiscuous, etc.) |
| `push_remote(pcap, ports)` | Push a pcap file to hardware |
| `set_rx_queue(port, size)` | Enable RX packet queue |
| `get_rx_queue_pkts(port)` | Get captured RX packets |
| `ping_ip(port, dst_ip)` | Ping from a TRex port |
| `resolve(ports)` | ARP resolution |

#### 4.4.4 Exception Handling

```python
from trex.stl.trex_stl_lib.api import *

client = STLClient()
try:
    client.connect()
    client.acquire(ports=[0])
    client.start(ports=[0], mult="1gbps")
except TRexError as e:
    print(f"General TRex error: {e}")
except TRexTimeoutError as e:
    print(f"Connection/operation timeout: {e}")
except TRexArgumentError as e:
    print(f"Invalid argument: {e}")
except TRexTypeError as e:
    print(f"Type mismatch: {e}")
finally:
    client.disconnect()
```

---

## 5. Common Tasks

### 5.1 Stateless (STL) Workflows

#### Recipe 1: Simple Single-Stream UDP at Fixed Rate

```bash
# Start TRex in stateless mode
sudo ./t-rex-64 -i --sl --cfg /etc/trex_cfg.yaml

# In console:
trex> start -f stl/udp_1pkt_simple.py -m 100kpps -p 0
trex> tui
# Watch real-time stats, press 'q' to exit TUI
trex> stop -a
```

#### Recipe 2: Benchmark with Variable Packet Sizes

```bash
# 64-byte packets at maximum rate
trex> start -f stl/bench.py -m 100% -p 0 -t size=64

# IMIX (Internet Mix) distribution
trex> start -f stl/bench.py -m 50% -p 0 -t size=imix
```

IMIX generates a realistic distribution of packet sizes (64B, 594B, 1518B) in proportions matching real-world internet traffic.

#### Recipe 3: Multi-Stream with Field Engine (Variable Source IPs)

```bash
trex> start -f stl/bench.py -m 1gbps -p 0 -t vm=var1
```

The `vm=var1` tunable enables the field engine to vary the source IP address across a range, creating multiple flows for load balancer and firewall testing.

#### Recipe 4: Bidirectional Traffic

```bash
# Send from both port 0 and port 1
trex> start -f stl/bench.py -m 1gbps -p 0 1
```

#### Recipe 5: Pcap Replay

Push a recorded pcap file as a traffic stream:

```bash
trex> push --port 0 -f /path/to/capture.pcap -m 1gbps --duration 60
```

Or use the pcap profile:

```bash
trex> start -f stl/pcap.py -m 1gbps -p 0 -t pcap_file=/path/to/capture.pcap
```

#### Recipe 6: Flow Stats and Latency Measurement

```bash
# With per-flow statistics
trex> start -f stl/bench.py -m 1gbps -p 0 -t flow=fs pg_id=7

# With flow-stats latency (measures per-flow latency)
trex> start -f stl/bench.py -m 100kpps -p 0 -t flow=fsl pg_id=7
```

> [!NOTE]
> Flow-stats latency (`fsl`) adds timestamping to each packet group. This has slightly higher overhead than basic flow stats (`fs`). Use `-t flow=no-fs` to disable flow stats entirely for maximum throughput.

#### Recipe 7: Gradual Rate Ramp-Up

```bash
trex> start -f stl/bench.py -m 100kpps -p 0
# ... wait a few seconds
trex> update -m 500kpps -p 0
# ... wait
trex> update -m 1mpps -p 0
# ... wait
trex> update -m 5mpps -p 0
```

#### Recipe 8: Cache-Based Streams (High PPS)

For maximum packets-per-second, use cached streams that pre-compute packets in memory:

```bash
trex> start -f stl/bench.py -m 100% -p 0 -t cache_size=1024
```

#### Recipe 9: Duration-Limited Test Run

```bash
# Run for exactly 120 seconds, then stop
trex> start -f stl/bench.py -m 1gbps -p 0 -d 120
```

#### Recipe 10: Capture Packets for Analysis

```bash
# Start a capture on port 1 (RX side)
trex> capture monitor start --rx 1 --limit 1000

# Start traffic
trex> start -f stl/bench.py -m 10kpps -p 0

# Save captured packets
trex> capture monitor save /tmp/captured.pcap
trex> capture monitor stop
```

### 5.2 Advanced Stateful (ASTF) Workflows

ASTF mode generates realistic TCP and UDP application-layer traffic. TRex acts as both client and server, establishing full TCP connections, performing handshakes, and transferring application data.

#### Recipe 11: HTTP Traffic Generation

```bash
# Start TRex in ASTF mode
sudo ./t-rex-64 -i --tcp_cfg --cfg /etc/trex_cfg.yaml

# In console:
trex> start -f astf/http_simple.py -m 1
trex> stats
trex> stop
```

The multiplier (`-m`) in ASTF mode controls the rate of new connections per second relative to the profile's base CPS.

#### Recipe 12: HTTP with Custom Response Size

```bash
trex> start -f astf/http_simple.py -m 1 -t size=4096
```

#### Recipe 13: FTP Traffic

```bash
trex> start -f astf/ftp_simple.py -m 1
```

#### Recipe 14: UDP Application Traffic

```bash
trex> start -f astf/udp1.py -m 10
```

#### Recipe 15: ASTF with TCP Tuning

```bash
# Custom MSS and window size
trex> start -f astf/http_simple.py -m 1 -t mss=1460 win=65535

# With TCP keepalive
trex> start -f astf/http_simple.py -m 1 -t tcp_ka=60

# With HTTP pipelining
trex> start -f astf/http_simple.py -m 1 -t pipe=4
```

#### Recipe 16: ASTF Simulation (Offline)

Test ASTF profiles without NICs using the simulator:

```bash
sudo ./t-rex-64 -f astf/http_simple.py --tcp_cfg --full -o sim_output.erf
```

With network impairments:

```bash
sudo ./t-rex-64 -f astf/http_simple.py --tcp_cfg --full \
    --rtt 50000 --drop 0.01 -o sim_output.erf
```

#### Recipe 17: ASTF Traffic Shaping Simulation

```bash
sudo ./t-rex-64 -f astf/http_simple.py --tcp_cfg --full \
    --shaper-rate 1000000000 --shaper-size 65536 -o shaped_output.erf
```

### 5.3 Emulation (EMU) Workflows

EMU mode emulates network clients at the protocol level — DHCP, DNS, ARP, IGMP, and more. Each emulated client has its own MAC, IP, and protocol state.

#### Recipe 18: DHCP Client Emulation

```python
# Using EMU profile
from trex.emu.trex_emu_profile import *

# Or via console:
trex> start -f emu/dhcp_profile.py -t clients=100
```

Common EMU tunables:

| Tunable | Description |
|---|---|
| `--ns` | Namespace identifier |
| `--clients` | Number of emulated clients |
| `--mac` | Base MAC address |
| `--4` | IPv4 mode |
| `--6` | IPv6 mode |
| `--dg` | Default gateway |

#### Recipe 19: DNS Client Emulation

```bash
trex> start -f emu/dns_profile.py -t clients=50
```

#### Recipe 20: mDNS (Multicast DNS) Emulation

```bash
trex> start -f emu/mdns_profile.py -t clients=20
```

#### Recipe 21: IGMP Multicast Group Joins

```bash
trex> start -f emu/igmpv4_profile.py -t clients=100
```

#### Recipe 22: IPFIX Flow Export Emulation

```bash
trex> start -f emu/ipfix_profile.py
```

#### Recipe 23: ARP Emulation

```bash
trex> start -f emu/arp_profile.py -t clients=200
```

#### Recipe 24: AppSim (Application Simulation)

```bash
trex> start -f emu/appsim_profile.py
```

### 5.4 Profile Writing

#### Recipe 25: Writing a Custom STL Profile

STL profiles are Python classes that inherit from the TRex profile base:

```python
from trex.stl.trex_stl_lib.api import *

class STLS1(object):
    def get_streams(self, tunables, **kwargs):
        size = tunables.get('size', 64)

        # Build a UDP packet
        pkt = STLPktBuilder(
            pkt=Ether() / IP(src="16.0.0.1", dst="48.0.0.1") /
                UDP(dport=12, sport=1025) /
                Raw(b'\x00' * (size - 42))
        )

        # Single continuous stream
        return [STLStream(
            packet=pkt,
            mode=STLTXCont(pps=1000)
        )]

    def get_tunables(self):
        return {'size': 64}


def register():
    return STLS1()
```

Run it:

```bash
trex> start -f my_profile.py -m 1gbps -p 0 -t size=128
```

#### Recipe 26: Multi-Stream Profile with Bursts

```python
from trex.stl.trex_stl_lib.api import *

class STLMultiBurst(object):
    def get_streams(self, tunables, **kwargs):
        pkt_small = STLPktBuilder(
            pkt=Ether() / IP(src="16.0.0.1", dst="48.0.0.1") /
                UDP() / Raw(b'\x00' * 22)
        )
        pkt_large = STLPktBuilder(
            pkt=Ether() / IP(src="16.0.0.1", dst="48.0.0.1") /
                UDP() / Raw(b'\x00' * 1400)
        )

        return [
            # Stream 1: continuous small packets
            STLStream(
                packet=pkt_small,
                mode=STLTXCont(pps=5000),
                name="small_continuous"
            ),
            # Stream 2: burst of 100 large packets every second
            STLStream(
                packet=pkt_large,
                mode=STLTXMultiBurst(
                    pkts_per_burst=100,
                    ibg=1000000.0,  # inter-burst gap in usec
                    count=0  # infinite
                ),
                name="large_bursts"
            ),
        ]

def register():
    return STLMultiBurst()
```

#### Recipe 27: Profile with Field Engine (IP Range Sweep)

```python
from trex.stl.trex_stl_lib.api import *

class STLVarIP(object):
    def get_streams(self, tunables, **kwargs):
        pkt = Ether() / IP(src="16.0.0.1", dst="48.0.0.1") / UDP() / Raw(b'\x00' * 22)

        vm = STLScVmRaw([
            STLVmFlowVar(name="src_ip",
                         min_value="10.0.0.1",
                         max_value="10.0.0.254",
                         size=4, op="inc"),
            STLVmWrFlowVar(fv_name="src_ip", pkt_offset="IP.src"),
            STLVmFixIpv4(offset="IP")
        ])

        return [STLStream(
            packet=STLPktBuilder(pkt=pkt, vm=vm),
            mode=STLTXCont(pps=1000)
        )]

def register():
    return STLVarIP()
```

#### Recipe 28: Writing a Custom ASTF Profile

```python
from trex.astf.trex_astf_lib.api import *

class Prof1(object):
    def get_profile(self, tunables, **kwargs):
        size = tunables.get('size', 1024)

        # Client program: connect, send request, receive response
        prog_c = ASTFProgram()
        prog_c.send(b"GET /index.html HTTP/1.1\r\nHost: test\r\n\r\n")
        prog_c.recv(size)

        # Server program: receive request, send response
        prog_s = ASTFProgram()
        prog_s.recv(1)
        prog_s.send(b"HTTP/1.1 200 OK\r\nContent-Length: " +
                     str(size).encode() + b"\r\n\r\n" + b"A" * size)

        # Create the template
        template = ASTFTCPClientTemplate(
            program=prog_c,
            ip_gen=ASTFIPGenDist(
                ip_range=ASTFIPRange(ip_start="16.0.0.1",
                                      ip_end="16.0.0.254"),
                dist="seq"
            ),
            port=80
        )

        server_template = ASTFTCPServerTemplate(program=prog_s)
        cap = ASTFCapInfo(cps=100)

        return ASTFProfile(
            default_ip_gen=ASTFIPGenGlobal(
                ip_offset="1.0.0.0"),
            cap_list=[cap],
            templates=[ASTFTemplate(
                client_template=template,
                server_template=server_template)]
        )

def register():
    return Prof1()
```

### 5.5 Automation & Scripting

#### Recipe 29: Automated Throughput Test

```python
from trex.stl.trex_stl_lib.api import *
import time

def find_max_rate(server="127.0.0.1", profile="stl/bench.py",
                  target_loss_pct=0.1, duration=30):
    """Binary search for maximum rate with acceptable loss."""
    client = STLClient(server=server)
    client.connect()
    client.acquire(ports=[0, 1])

    low, high = 1, 100  # percentage of line rate
    best = 0

    try:
        while high - low > 1:
            mid = (low + high) // 2
            client.reset(ports=[0, 1])
            client.clear_stats()

            prof = STLProfile.load(profile)
            client.add_streams(prof.get_streams(), ports=[0])
            client.start(ports=[0], mult=f"{mid}%", duration=duration)
            client.wait_on_traffic(ports=[0])

            stats = client.get_stats()
            tx = stats[0]["opackets"]
            rx = stats[1]["ipackets"]
            loss_pct = ((tx - rx) / tx * 100) if tx > 0 else 100

            print(f"Rate: {mid}% | TX: {tx} | RX: {rx} | Loss: {loss_pct:.3f}%")

            if loss_pct <= target_loss_pct:
                best = mid
                low = mid
            else:
                high = mid

        print(f"Maximum rate with <={target_loss_pct}% loss: {best}%")
    finally:
        client.stop(ports=[0])
        client.disconnect()

    return best

if __name__ == "__main__":
    find_max_rate()
```

#### Recipe 30: Latency Monitoring Script

```python
from trex.stl.trex_stl_lib.api import *
import time, json

def monitor_latency(server="127.0.0.1", profile="stl/bench.py",
                    rate="1gbps", duration=60, interval=5):
    """Collect latency measurements over time."""
    client = STLClient(server=server)
    client.connect()
    client.acquire(ports=[0, 1])
    client.reset(ports=[0, 1])

    results = []

    try:
        prof = STLProfile.load(profile)
        client.add_streams(prof.get_streams(), ports=[0])
        client.start(ports=[0], mult=rate, duration=duration)

        elapsed = 0
        while elapsed < duration:
            time.sleep(interval)
            elapsed += interval
            stats = client.get_stats()
            sample = {
                "time": elapsed,
                "tx_pps": stats[0].get("tx_pps", 0),
                "rx_pps": stats[1].get("rx_pps", 0),
                "tx_bps": stats[0].get("tx_bps", 0),
            }
            results.append(sample)
            print(json.dumps(sample))

        client.wait_on_traffic(ports=[0])
    finally:
        client.disconnect()

    return results

if __name__ == "__main__":
    monitor_latency()
```

#### Recipe 31: Multi-Profile Sequential Test

```python
from trex.stl.trex_stl_lib.api import *

profiles = [
    ("stl/bench.py", {"size": 64}, "1gbps", 30),
    ("stl/bench.py", {"size": 256}, "1gbps", 30),
    ("stl/bench.py", {"size": 512}, "1gbps", 30),
    ("stl/bench.py", {"size": 1518}, "1gbps", 30),
    ("stl/bench.py", {"size": "imix"}, "1gbps", 30),
]

client = STLClient()
client.connect()
client.acquire(ports=[0, 1])

for profile_path, tunables, rate, duration in profiles:
    print(f"\n--- Testing {profile_path} with {tunables} at {rate} ---")
    client.reset(ports=[0, 1])
    client.clear_stats()

    prof = STLProfile.load(profile_path, **tunables)
    client.add_streams(prof.get_streams(), ports=[0])
    client.start(ports=[0], mult=rate, duration=duration)
    client.wait_on_traffic(ports=[0])

    stats = client.get_stats()
    tx = stats[0]["opackets"]
    rx = stats[1]["ipackets"]
    loss = ((tx - rx) / tx * 100) if tx > 0 else 0
    print(f"TX: {tx:,} | RX: {rx:,} | Loss: {loss:.4f}%")

client.disconnect()
```

### 5.6 IPv6 Workflows

#### Recipe 32: IPv6 Traffic Generation

```bash
# Start TRex with IPv6 mode
sudo ./t-rex-64 -i --sl --ipv6 --cfg /etc/trex_cfg.yaml
```

In the console, use the `scan6` command to discover IPv6 neighbors:

```bash
trex> scan6 -p 0
trex> start -f stl/bench.py -m 1gbps -p 0
```

#### Recipe 33: IPv6 Neighbor Discovery via Plugin

```bash
trex> plugins load IPv6ND
trex> ipv6nd start -p 0
trex> ipv6nd show -p 0
```

### 5.7 PCAP-Based Workflows

#### Recipe 34: Replay a Pcap at Line Rate

```bash
trex> push -f /path/to/traffic.pcap --port 0 -m 100% -d 60
```

#### Recipe 35: Convert Pcap to STL Profile

```python
from trex.stl.trex_stl_lib.api import *

class STLPcap(object):
    def get_streams(self, tunables, **kwargs):
        pcap_file = tunables.get('pcap_file', 'input.pcap')
        return STLProfile.load_pcap(pcap_file, ipg_usec=10.0)

def register():
    return STLPcap()
```

### 5.8 Port Configuration Workflows

#### Recipe 36: Set Port to Promiscuous Mode

```python
client.set_port_attr(ports=[0], promiscuous=True)
```

Or in the console:

```bash
trex> portattr --port 0 --prom on
```

#### Recipe 37: ARP Resolution Before Traffic

```bash
trex> resolve -p 0 1
```

This sends ARP requests from TRex ports to discover the destination MAC address based on the `default_gw` in your config.

#### Recipe 38: Viewing Port Info

```bash
trex> portattr -a
```

### 5.9 Statistics & Monitoring

#### Recipe 39: Real-Time TUI Dashboard

```bash
trex> tui
```

The TUI provides a live-updating dashboard showing:
- Per-port TX/RX rates (pps, bps)
- Total transmitted/received packets
- Drop counters and error counters
- Active stream information

Press `q` to exit the TUI.

#### Recipe 40: Export Statistics to JSON

```python
import json

client = STLClient()
client.connect()
# ... start traffic ...

stats = client.get_stats()
with open("trex_stats.json", "w") as f:
    json.dump(stats, f, indent=2, default=str)
```

#### Recipe 41: ZMQ Statistics Subscription

If `enable_zmq_pub: true` in your config, external tools can subscribe to the stats stream:

```python
import zmq, json

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:4500")
socket.setsockopt_string(zmq.SUBSCRIBE, "")

while True:
    msg = socket.recv_json()
    print(json.dumps(msg, indent=2))
```

### 5.10 Network Equipment Testing Workflows

#### Recipe 42: Firewall Rule Testing

Generate traffic matching specific firewall rules to verify pass/block behavior:

```python
from trex.stl.trex_stl_lib.api import *

class FirewallTest(object):
    def get_streams(self, tunables, **kwargs):
        # HTTP traffic (should be allowed)
        http = STLStream(
            packet=STLPktBuilder(
                pkt=Ether() / IP(src="10.0.0.1", dst="192.168.1.1") /
                    TCP(dport=80) / Raw(b"GET / HTTP/1.1\r\n\r\n")
            ),
            mode=STLTXCont(pps=100),
            name="http_allowed"
        )

        # SSH traffic (should be blocked)
        ssh = STLStream(
            packet=STLPktBuilder(
                pkt=Ether() / IP(src="10.0.0.1", dst="192.168.1.1") /
                    TCP(dport=22) / Raw(b"\x00" * 20)
            ),
            mode=STLTXCont(pps=100),
            name="ssh_blocked"
        )

        return [http, ssh]

def register():
    return FirewallTest()
```

#### Recipe 43: Load Balancer Distribution Testing

Generate traffic from many source IPs to verify load balancer hash distribution:

```python
from trex.stl.trex_stl_lib.api import *

class LBTest(object):
    def get_streams(self, tunables, **kwargs):
        pkt = Ether() / IP(src="10.0.0.1", dst="192.168.1.100") / \
              TCP(dport=80) / Raw(b"\x00" * 100)

        vm = STLScVmRaw([
            STLVmFlowVar(name="src", min_value="10.0.0.1",
                         max_value="10.0.255.254", size=4, op="random"),
            STLVmWrFlowVar(fv_name="src", pkt_offset="IP.src"),
            STLVmFixIpv4(offset="IP"),
        ])

        return [STLStream(
            packet=STLPktBuilder(pkt=pkt, vm=vm),
            mode=STLTXCont(pps=10000)
        )]

def register():
    return LBTest()
```

#### Recipe 44: Router Throughput Testing (RFC 2544 Style)

```python
from trex.stl.trex_stl_lib.api import *

FRAME_SIZES = [64, 128, 256, 512, 1024, 1280, 1518]

client = STLClient()
client.connect()
client.acquire(ports=[0, 1])

for size in FRAME_SIZES:
    client.reset(ports=[0, 1])
    client.clear_stats()

    prof = STLProfile.load("stl/bench.py", size=size)
    client.add_streams(prof.get_streams(), ports=[0])
    client.start(ports=[0], mult="100%", duration=60)
    client.wait_on_traffic(ports=[0])

    stats = client.get_stats()
    tx = stats[0]["opackets"]
    rx = stats[1]["ipackets"]
    throughput_pps = stats[0].get("tx_pps", 0)
    loss = ((tx - rx) / tx * 100) if tx > 0 else 0

    print(f"Frame: {size}B | PPS: {throughput_pps:,.0f} | Loss: {loss:.4f}%")

client.disconnect()
```

#### Recipe 45: IDS/IPS Evasion Testing with Mixed Traffic

```bash
# Generate a mix of traffic types
trex> start -f stl/imix.py -m 5gbps -p 0
```

The IMIX profile generates a realistic distribution of packet sizes that mirrors real-world traffic patterns, exercising DPI engines across multiple packet boundaries.

### 5.11 Advanced Profile Techniques

#### Recipe 46: Chained Streams (Sequential Traffic Patterns)

```python
from trex.stl.trex_stl_lib.api import *

class STLChained(object):
    def get_streams(self, tunables, **kwargs):
        # Stream 1: burst of 1000 small packets, then trigger stream 2
        s1 = STLStream(
            packet=STLPktBuilder(
                pkt=Ether() / IP() / UDP() / Raw(b'\x00' * 22)
            ),
            mode=STLTXSingleBurst(total_pkts=1000, pps=10000),
            name="burst_small",
            next="burst_large"
        )

        # Stream 2: burst of 500 large packets, then trigger stream 3
        s2 = STLStream(
            packet=STLPktBuilder(
                pkt=Ether() / IP() / UDP() / Raw(b'\x00' * 1400)
            ),
            mode=STLTXSingleBurst(total_pkts=500, pps=5000),
            name="burst_large",
            next="burst_small",
            self_start=False
        )

        return [s1, s2]

def register():
    return STLChained()
```

#### Recipe 47: YAML-Based Profile (Alternative to Python)

For simple stream definitions, YAML profiles can be used:

```yaml
# simple_udp.yaml
- name: udp_stream
  stream:
    self_start: true
    packet:
      binary: stl/udp_64B_no_crc.pcap
    mode:
      type: continuous
      pps: 1000
```

```bash
trex> start -f simple_udp.yaml -m 1gbps -p 0
```

### 5.12 Packet Capture & Analysis

#### Recipe 48: Capture and Save Packets

```bash
# Start monitoring RX on port 1
trex> capture monitor start --rx 1 --limit 500

# Generate traffic
trex> start -f stl/bench.py -m 10kpps -p 0 -d 10

# View capture status
trex> capture monitor show

# Save to pcap
trex> capture monitor save /tmp/rx_capture.pcap

# Stop capture
trex> capture monitor stop
```

#### Recipe 49: Record and Replay

```bash
# Step 1: Capture live traffic with tcpdump on another machine
tcpdump -i eth0 -w recorded.pcap -c 10000

# Step 2: Replay on TRex at high rate
trex> push -f recorded.pcap --port 0 -m 10gbps -d 300
```

#### Recipe 50: Dry Run Profile Validation

```bash
# Validate profile syntax and packet construction without sending
sudo ./t-rex-64 -f stl/my_profile.py --dry -v 2
```

---

## 6. Troubleshooting

### 6.1 Installation & Startup Issues

---

**Error:** `Could not find requested interface` during DPDK setup

**Cause:** The PCI address in your config does not match any NIC on the system, or the NIC is not bound to a DPDK-compatible driver.

**Solution:**
1. List all NICs and their current driver bindings:
   ```bash
   sudo ./dpdk_setup_ports.py -s
   ```
2. Verify PCI addresses match your config file.
3. Rebind the interfaces:
   ```bash
   sudo ./dpdk_setup_ports.py -i
   ```

**Prevention:** Always run `dpdk_setup_ports.py -s` after system changes (kernel updates, NIC changes) to confirm bindings. (Related: GitHub issue [#919](https://github.com/cisco-system-traffic-generator/trex-core/issues/919))

---

**Error:** `failure in ixgbe_dev_start()` or NIC driver start failure

**Cause:** The DPDK driver cannot initialize the NIC. Common with ixgbe NICs when the kernel driver is still loaded or IOMMU settings conflict.

**Solution:**
1. Unload the kernel driver:
   ```bash
   sudo modprobe -r ixgbe
   ```
2. Check IOMMU settings in BIOS and GRUB:
   ```bash
   cat /proc/cmdline | grep iommu
   ```
3. Try `vfio-pci` instead of `igb_uio`:
   ```bash
   sudo modprobe vfio-pci
   sudo ./dpdk_setup_ports.py -i
   ```

**Prevention:** Use `vfio-pci` as the default DPDK driver on modern kernels (4.x+). It has better IOMMU support. (Related: GitHub issue [#934](https://github.com/cisco-system-traffic-generator/trex-core/issues/934))

---

**Error:** `no Driver_str in dpdk_setup_ports.py`

**Cause:** The NIC's driver string cannot be detected, usually because the interface is in an unexpected state or uses an unsupported chipset.

**Solution:**
1. Check `lspci -v` for the NIC's kernel driver.
2. Manually specify the driver in `dpdk_setup_ports.py`.
3. Ensure the DPDK-compatible driver module is loaded:
   ```bash
   sudo modprobe igb_uio
   # or
   sudo modprobe vfio-pci
   ```

**Prevention:** Stick to well-tested NIC families: Intel X710, XL710, E810, 82599 (ixgbe), i40e. (Related: GitHub issue [#1135](https://github.com/cisco-system-traffic-generator/trex-core/issues/1135))

---

**Error:** `X710 NIC card can't bind driver` on Ubuntu 20.04

**Cause:** Kernel version and DPDK driver compatibility issue with i40e NICs.

**Solution:**
1. Update to the latest TRex release.
2. Use `vfio-pci` driver with IOMMU enabled:
   ```bash
   # Add to GRUB_CMDLINE_LINUX:
   intel_iommu=on iommu=pt
   ```
3. Rebuild GRUB and reboot:
   ```bash
   sudo update-grub && sudo reboot
   ```

**Prevention:** Always enable IOMMU when using `vfio-pci`. (Related: GitHub issue [#1127](https://github.com/cisco-system-traffic-generator/trex-core/issues/1127))

---

**Error:** Unable to start `./t-rex-64` in ASTF mode

**Cause:** Missing ASTF configuration, wrong flags, or port setup issues.

**Solution:**
1. Ensure you are using `--tcp_cfg` flag (not `--sl`):
   ```bash
   sudo ./t-rex-64 -i --tcp_cfg --cfg /etc/trex_cfg.yaml
   ```
2. Verify port_info uses IP-based config (ASTF needs IP, not just MAC):
   ```yaml
   port_info:
     - ip: 10.0.0.1
       default_gw: 10.0.0.2
     - ip: 10.1.0.1
       default_gw: 10.1.0.2
   ```
3. Check that ARP resolution succeeds:
   ```bash
   trex> resolve
   ```

**Prevention:** ASTF mode requires IP-based port configuration so that the TCP/IP stack can function. MAC-only config works for STL but not ASTF. (Related: GitHub issue [#1183](https://github.com/cisco-system-traffic-generator/trex-core/issues/1183))

---

### 6.2 Traffic Generation Issues

---

**Error:** Packets truncated to 64 bytes with Intel E810 NICs using STL VM

**Cause:** Known issue with E810 NICs and the field engine (VM) in STL mode where packets get truncated.

**Solution:**
1. Update to the latest TRex release.
2. If the issue persists, try disabling hardware offloads:
   ```bash
   trex> portattr --port 0 --prom on
   ```
3. As a workaround, use a cache-based approach instead of VM for packet variation.

**Prevention:** Test with the `--dry` flag first to verify expected packet sizes. (Related: GitHub issue [#1205](https://github.com/cisco-system-traffic-generator/trex-core/issues/1205))

---

**Error:** TRex attempting to set MTU to 65518

**Cause:** A profile or configuration is requesting jumbo frames beyond the NIC's capability.

**Solution:**
1. Verify your profile does not request packets larger than 9000 bytes.
2. Check NIC firmware supports the requested MTU.
3. Set explicit MTU in port attributes:
   ```bash
   trex> portattr --port 0 --mtu 1500
   ```

**Prevention:** Set explicit MTU values in port configuration rather than relying on auto-detection. (Related: GitHub issue [#1204](https://github.com/cisco-system-traffic-generator/trex-core/issues/1204))

---

**Error:** Stop traffic causes TRex abnormalities

**Cause:** Stopping traffic abruptly during certain profile types can leave internal state in an inconsistent condition.

**Solution:**
1. Use `stop -a` to stop all ports cleanly.
2. Follow with `reset` to clear all state:
   ```bash
   trex> stop -a
   trex> reset
   ```
3. If the server becomes unresponsive, restart:
   ```bash
   # Kill and restart
   sudo killall t-rex-64
   sudo ./t-rex-64 -i --sl --cfg /etc/trex_cfg.yaml
   ```

**Prevention:** Avoid stopping mid-burst on complex chained stream profiles. Use duration-limited runs (`-d`) when possible. (Related: GitHub issue [#1207](https://github.com/cisco-system-traffic-generator/trex-core/issues/1207))

---

**Error:** `Method not found 'ctx_stop_autotriggers'`

**Cause:** Version mismatch between the TRex console/client and the TRex server. The client is calling an API method that does not exist on the server version.

**Solution:**
1. Ensure console and server are the same TRex version.
2. Update both to the latest release.
3. If using a remote console, verify the connection is to the correct server.

**Prevention:** Always update console and server together. (Related: GitHub issue [#1208](https://github.com/cisco-system-traffic-generator/trex-core/issues/1208))

---

### 6.3 Performance Issues

---

**Error:** Cannot reach expected throughput (e.g., only 10 Gbps on a 40G NIC)

**Cause:** Insufficient CPU cores assigned, NUMA mismatch, or missing hugepages.

**Solution:**
1. Add more datapath cores:
   ```yaml
   c: 4   # or more
   ```
2. Pin threads to the correct NUMA node (see Section 3.3).
3. Verify hugepages are allocated:
   ```bash
   grep Huge /proc/meminfo
   ```
4. Set `port_bandwidth_gb` to match your NIC:
   ```yaml
   port_bandwidth_gb: 40  # for XL710
   ```

**Prevention:** Always configure CPU pinning for production deployments. One core is rarely sufficient for rates above 10 Gbps.

---

**Error:** High CPU usage but low packet rate

**Cause:** Small packets (64B) require more CPU cycles per unit of bandwidth. The field engine (VM) also adds CPU overhead per packet.

**Solution:**
1. Use `cache_size` to pre-compute packets:
   ```bash
   trex> start -f stl/bench.py -m 100% -t cache_size=4096
   ```
2. Reduce field engine complexity.
3. Add more datapath cores.

**Prevention:** For maximum PPS with small packets, minimize field engine usage and maximize cache size.

---

**Error:** TRex cannot effectively use 400G network cards

**Cause:** Current DPDK support and TRex architecture have limitations at 400G speeds.

**Solution:**
1. Use multiple datapath cores (maximum 8):
   ```yaml
   c: 8
   platform:
     master_thread_id: 0
     latency_thread_id: 1
     dual_if:
       - socket: 0
         threads: [2, 3, 4, 5, 6, 7, 8, 9]
   ```
2. Use pre-cached streams to minimize per-packet CPU work.
3. Monitor for TRex updates with improved 400G support.

**Prevention:** For 400G testing, plan for potential rate limitations and test incrementally. (Related: GitHub issue [#1185](https://github.com/cisco-system-traffic-generator/trex-core/issues/1185))

---

### 6.4 ARP & Connectivity Issues

---

**Error:** Gratuitous ARP broken

**Cause:** Known regression in certain TRex versions affecting gratuitous ARP handling.

**Solution:**
1. Update to the latest TRex release.
2. Manually set destination MACs in config as a workaround:
   ```yaml
   port_info:
     - dest_mac: "aa:bb:cc:dd:ee:ff"
       src_mac: "11:22:33:44:55:66"
   ```
3. Use the `resolve` console command to manually trigger ARP.

**Prevention:** Test ARP behavior after every TRex upgrade. (Related: GitHub issue [#1162](https://github.com/cisco-system-traffic-generator/trex-core/issues/1162))

---

**Error:** ARP resolution fails — no response from DUT

**Cause:** The DUT (device under test) is not responding to ARP requests from TRex ports.

**Solution:**
1. Verify IP configuration in port_info matches the DUT's expected subnet.
2. Check that the DUT has an ARP entry or responds to ARP:
   ```bash
   trex> ping -p 0 -d 10.0.0.2
   ```
3. Set promiscuous mode:
   ```bash
   trex> portattr --port 0 --prom on
   ```
4. Fall back to static MAC addresses in config.

**Prevention:** Always verify L2/L3 connectivity before running traffic tests.

---

### 6.5 Console & API Issues

---

**Error:** Cannot connect to TRex server from console

**Cause:** Server not running, wrong port, or firewall blocking ZMQ.

**Solution:**
1. Verify the server is running:
   ```bash
   ps aux | grep t-rex-64
   ```
2. Check the ZMQ RPC port (default 4501):
   ```bash
   ss -tlnp | grep 4501
   ```
3. Connect with explicit server/port:
   ```bash
   ./trex-console -s 127.0.0.1 -p 4501
   ```

**Prevention:** Use `zmq_rpc_port` in config to set a known port. Ensure no firewall blocks the ZMQ ports.

---

**Error:** Port ownership conflict — `port is already owned by another user`

**Cause:** Another console or API session has acquired the ports.

**Solution:**
1. Use `--force` to take ownership:
   ```bash
   trex> acquire --force
   ```
2. Or from the API:
   ```python
   client.acquire(ports=[0, 1], force=True)
   ```

**Prevention:** Always `release` ports and `disconnect` cleanly when done.

---

### 6.6 Profile Issues

---

**Error:** Profile fails to load — Python import error

**Cause:** The profile script has a Python syntax error or missing import.

**Solution:**
1. Test the profile with dry run:
   ```bash
   sudo ./t-rex-64 -f stl/my_profile.py --dry -v 2
   ```
2. Validate Python syntax:
   ```bash
   python3 -c "import py_compile; py_compile.compile('stl/my_profile.py', doraise=True)"
   ```
3. Ensure the profile has a `register()` function returning the profile class.

**Prevention:** Always test profiles with `--dry` before using in production.

---

**Error:** Profile tunables not taking effect

**Cause:** The tunable names in `-t key=value` do not match the profile's `get_tunables()` definition.

**Solution:**
1. Check the profile source for available tunables.
2. Use exact tunable names as defined in the profile:
   ```bash
   trex> start -f stl/bench.py -m 1gbps -t size=128 vm=var1
   ```

**Prevention:** Document tunables in profile source comments.

---

### 6.7 Build Issues

---

**Error:** Build fails with `waf` errors

**Cause:** Missing build dependencies or wrong compiler version.

**Solution:**
1. Install required packages:
   ```bash
   sudo apt install build-essential g++ python3 zlib1g-dev libpcap-dev
   ```
2. Set the compiler:
   ```bash
   export CXX=g++-9
   ```
3. Clean and rebuild:
   ```bash
   cd linux_dpdk
   ./b clean
   ./b configure
   ./b build
   ```

**Prevention:** Use the pre-built binaries unless you specifically need a custom build.

---

### 6.8 Memory Issues

---

**Error:** `EAL: No free hugepages reported` or `Cannot allocate memory`

**Cause:** Hugepages not configured or exhausted by other DPDK applications.

**Solution:**
1. Check hugepage availability:
   ```bash
   grep Huge /proc/meminfo
   ```
2. Allocate more hugepages:
   ```bash
   echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
   ```
3. Use `limit_memory` in config to cap TRex memory:
   ```yaml
   limit_memory: 2048  # MB
   ```

**Prevention:** Set hugepages at boot via kernel command line for reliability.

---

**Error:** Out of datapath flow entries

**Cause:** The `memory.dp_flows` value is too low for the traffic profile.

**Solution:**
Increase `dp_flows` in config:
```yaml
memory:
  dp_flows: 4194304  # 4M flows
```

**Prevention:** Size `dp_flows` based on your expected concurrent flow count. Rule of thumb: 2x your expected peak.

---

## 7. FAQ

### General

**Q: What operating systems does TRex support?**
TRex runs on Linux x86_64 only. It requires kernel 2.6.32 or newer. Ubuntu 18.04+, CentOS 7+, and RHEL 7+ are commonly used.

**Q: Does TRex require special hardware?**
TRex requires DPDK-compatible NICs (Intel X710, XL710, E810, 82599 ixgbe, i40e are best supported). Hugepage-capable RAM is required. Any modern x86_64 CPU with SSE 4.2 works.

**Q: Can TRex run in a virtual machine?**
Yes, with virtual NICs (virtio, vmxnet3). Performance is limited — expect 1-5 Gbps depending on the hypervisor. Set `port_bandwidth_gb: 1` in config.

**Q: What is the maximum throughput TRex can achieve?**
TRex can generate 200+ Gbps on appropriate hardware. Actual throughput depends on NIC count, CPU cores, packet size, and field engine complexity.

**Q: Is TRex free?**
Yes, TRex is fully open-source under the Apache 2.0 license.

### Modes

**Q: What is the difference between STL, ASTF, and STF modes?**
STL (Stateless) generates raw packet streams — maximum PPS, no connection state. ASTF (Advanced Stateful) creates full TCP/UDP connections with application-layer data exchange. STF (Stateful) is the legacy stateful mode using YAML/pcap templates (use ASTF for new work).

**Q: When should I use STL vs. ASTF?**
Use STL for throughput/PPS testing, packet-level testing, and when you need maximum rate. Use ASTF for testing firewalls, proxies, load balancers, and any device that tracks connections or inspects application-layer content.

**Q: What is EMU mode?**
EMU (Emulation) simulates protocol-level clients — DHCP, DNS, ARP, IGMP, etc. Use it when you need devices to respond to protocol interactions, not just forward packets.

**Q: Can I run STL and ASTF simultaneously?**
No, you must start the TRex server in one mode (`--sl` for STL or `--tcp_cfg` for ASTF). To switch, stop the server and restart in the other mode.

### Configuration

**Q: Where should I put the config file?**
The default location is `/etc/trex_cfg.yaml`. Use `--cfg /path/to/file.yaml` to specify an alternate location.

**Q: How do I find my NIC PCI addresses?**
Run `lspci | grep -i ethernet` or `sudo ./dpdk_setup_ports.py -s`.

**Q: Can I use more than 4 ports?**
The `port_limit` config supports 2 or 4 ports. For more ports, run multiple TRex instances with different `prefix` values.

**Q: What is the `prefix` config option for?**
It enables running multiple TRex instances on the same machine. Each instance needs a unique prefix, unique ports (zmq_pub_port, zmq_rpc_port, telnet_port), and unique NIC interfaces.

**Q: What does `port_bandwidth_gb` do?**
It tells TRex the link speed so it can correctly calculate percentage-based rates. Set to 1 for VMs, 10 for 10G NICs, 40 for XL710 40G NICs.

**Q: How do I configure CPU pinning?**
Use the `platform` section with `master_thread_id`, `latency_thread_id`, and `dual_if[].threads`. Pin all threads to the same NUMA node as the NICs for optimal performance. See Section 3.3.

### Traffic Profiles

**Q: What profile formats does TRex support?**
Python (recommended) and YAML. Python profiles are more flexible with support for tunables, field engines, and complex logic.

**Q: What are tunables?**
Tunables are runtime parameters passed to profiles via `-t key=value`. They let you customize packet size, flow count, protocol options, etc., without editing the profile source.

**Q: What is the field engine (VM)?**
The field engine varies packet fields (IP addresses, ports, etc.) at line rate without CPU overhead per-packet. It is configured via `STLScVmRaw` with flow variables and write instructions.

**Q: What is IMIX?**
Internet Mix — a standardized distribution of packet sizes (64B, 594B, 1518B) that approximates real-world internet traffic. Use `-t size=imix` with the bench profile.

**Q: What is the `cache_size` tunable?**
It pre-computes N packets in memory and replays them, bypassing per-packet construction. Dramatically increases PPS for profiles with field engines. Higher values use more memory but allow more packet variation.

**Q: How do chained streams work?**
Streams can reference a `next` stream by name. When a burst stream finishes, TRex starts the next stream. This creates sequential traffic patterns (e.g., handshake then data).

**Q: Can I replay pcap files?**
Yes. Use `push -f file.pcap` in the console or `STLProfile.load_pcap()` in the API. TRex replays packets from the pcap at the specified rate.

### Console & API

**Q: Can I connect to a remote TRex server?**
Yes. Use `./trex-console -s <server_ip>` or `STLClient(server="<server_ip>")` in the API.

**Q: What is read-only mode?**
Connect with `--readonly` to view stats without the ability to start/stop traffic. Useful for monitoring while someone else controls traffic.

**Q: How do I take over ports from another session?**
Use `acquire --force` in the console or `client.acquire(ports=[0], force=True)` in the API.

**Q: What happens if the console disconnects during a test?**
Traffic continues running on the server. Reconnect and re-acquire ports to regain control.

**Q: Can I automate TRex from CI/CD?**
Yes. Use the Python API in test scripts. Start the TRex server before the test suite, run traffic, collect stats, and assert thresholds.

### Performance

**Q: How many CPU cores does TRex need?**
Minimum: 1 master + 1 latency + 1 datapath = 3 cores. For high throughput, allocate 4-8 datapath cores plus the master and latency threads.

**Q: How do I maximize packets per second?**
Use small packets (64B), maximize datapath cores, use `cache_size` to pre-compute packets, disable field engine if not needed, and pin threads to the correct NUMA node.

**Q: Why is my latency high?**
Common causes: NUMA mismatch (threads and NICs on different sockets), CPU contention from other processes, insufficient hugepages causing TLB misses, or a misconfigured DUT.

**Q: How much memory does TRex need?**
Base: 1-2 GB for the DPDK memory pool. Add more for large flow tables (`memory.dp_flows`), large cache sizes, and ASTF connection state. Use `limit_memory` to cap usage.

### Troubleshooting

**Q: TRex starts but no packets appear on the wire. What should I check?**
1. Verify NICs are bound to DPDK (`dpdk_setup_ports.py -s`). 2. Check that dest_mac or default_gw is correct. 3. Run `resolve` to ARP. 4. Try `portattr --prom on` on the receiving side. 5. Check for cable/fiber issues.

**Q: How do I check if TRex is running?**
`ps aux | grep t-rex-64` or try connecting the console.

**Q: How do I cleanly shut down TRex?**
From the console: `stop -a` then `quit`. Then kill the server process: `sudo killall t-rex-64`.

**Q: My NIC is supported by DPDK but TRex does not start. What now?**
Check TRex release notes for NIC-specific requirements. Some NICs need firmware updates. Try both `igb_uio` and `vfio-pci` drivers. Enable IOMMU if using vfio-pci.

**Q: How do I debug a profile that produces unexpected traffic?**
Use `--dry -v 2` to see the packet structure without sending. Use packet capture (`capture monitor`) to examine actual packets on the wire. Check field engine variables with `-v 3`.

**Q: The TUI shows TX but zero RX. Is that normal?**
It depends on your setup. If you have a loopback cable (port 0 to port 1), you should see RX on port 1. If traffic goes through a DUT, the DUT may be dropping packets. Check DUT logs and counters.

**Q: How do I report a bug?**
File an issue at `https://github.com/cisco-system-traffic-generator/trex-core/issues` with your TRex version, OS, NIC model, config file, profile, and exact error message.

> [!CAUTION]
> When reporting issues, never include sensitive network information (real production IPs, credentials, internal hostnames) in public GitHub issues. Sanitize configs before posting.

---

_Generated May 2026 from cisco-system-traffic-generator/trex-core at 27e0153. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._