Network Traffic Generator

Network Traffic Generator — Installation Manual

This guide explains how to setup and use commands in our Network Traffic Generator(NTG) to generate traffic flows in intervals with specific parameters. In this manual, we demonstrate things as below :

• Network Traffic Generator — Installation Manual
  • Required environment and libraries for NTG
  • Files Overview
  • Network Traffic Generator Configuration
  • Flow Generation Configuration
    • flow Command Configuration
    • dist Command Configuration
  • Support Commands in NTG
    • Commands

Required environment and libraries for NTG

• Linux system is needed for all machines.
• Python 3.8+ recommended.

Install Python + pip:

Ubuntu/Debian:

sudo apt update
sudo apt install -y python3 python3-pip
python3 --version
pip3 --version

Libraries used by NTG controller-side scripts (network_traffic_generator.py + Utilis/*.py):

• loguru
• prompt_toolkit
• nornir, nornir-utils
• pyyaml
• pydantic
• numpy, pandas
• paramiko
• requests

Install them:

pip install --upgrade pip
pip install loguru prompt_toolkit nornir nornir-utils pyyaml numpy pandas paramiko requests pydantic

Clone our NTG:

git clone https://github.com/ndtwin-lab/Network-Traffic-Generator.git

If your environment uses Conda/virtualenv, activate it before installing packages.

If you use NTG with hardware testbed, you need to install below libraries for all machines that used to generate flows.

• fastapi
• uvicorn (used to start the API server)
• pydantic
• loguru
• orjson (required by ORJSONResponse)

pip install --upgrade pip
pip install fastapi "uvicorn[standard]" pydantic loguru orjson

Files Overview

├── development_guide
│   └── README.md
├── installation_guide
│   └── README.md
├── dist_template.json
├── flow_template.json
├── LICENSE
├── network_traffic_generator.py
├── network_traffic_generator_worker_node.py
├── testbed_topo.py
├── NTG.yaml
├── README.md
├── setting
│   ├── API_server_startup.yaml
│   ├── Hardware.yaml
│   └── Mininet.yaml
└── Utilis
    ├── command_utils.py
    ├── communicator.py
    └── distance_seperate.py

• network_traffic_generator.py: Interactive CLI that accepts commands.
• network_traffic_generator_worker_node.py: The worker node which receive flow generation requests from network_traffic_generator.py and manage iperf3 processes. Only used in Hardware testbed.
• testbed_topo.py: The example Mininet topology code.
• Utilis/: Contains some related util functions.
• setting/Hardware.yaml: NTG setting file for Hardware testbed. Contains IPs of NDTwin Kernel, worker nodes and on-site host IPs.
• setting/API_server_startup.yaml: Nornir group file (startup/shutdown commands for worker-node API servers).
• setting/Mininet.yaml: NTG setting file for Mininet. Contains runtime mode and local ndtwin kernel.
• NTG.yaml: Nornir configuration pointing to the inventory files above.
• flow_template.json & dist_template.json: Example configuration for intervals and flow generation parameters.

Network Traffic Generator Configuration

Ensure the following files exist and match your environment.

NTG.yaml

inventory:
  plugin: SimpleInventory
  options:
    host_file: "./setting/Mininet.yaml"
    group_file: "./setting/API_server_startup.yaml"

runner:
  plugin: threaded
  options:
    num_workers: 5

logging:
  enabled: false

Notes:

• If you want to use Mininet, please change the path of host_file to ./setting/Mininet.yaml

setting/API_server_startup.yaml

worker_node_servers:
  data:
    startup_commands:
      - "source ~/ntg/bin/activate && cd Desktop/NTG && pwd && nohup uvicorn network_traffic_generator_worker_node:app --host 0.0.0.0 --port 8000 > uvi.log 2>&1 &"
    shutdown_commands:
      - "killall -9 uvicorn"

Notes:

• startup_commands should start your FastAPI server (e.g., uvicorn network_traffic_generator_worker_node:app ...) on each worker node.
• Adjust virtualenv path and ports as needed.

setting/Hardware.yaml (abbreviated; tailor to your hosts)

Hardware_Testbed:
  hostname: "hardware_testbed"
  data:
    ndtwin_kernel: "http://10.10.xx.xx:8000"
    recycle_interval: 10
    sleep_time:
      min: 0.4
      max: 1.3
    ports_limitation:
      min_port: 5204
      max_port: 10204
      exclude_ports:
        - 8000
    log_level: "DEBUG"
    tcp_socket_write_length: "10K"  # in bytes, not including header, only payload, can use K for 1024 bytes, M for 1024*1024 bytes, etc.
    udp_packet_length: "1460"  # in bytes, not including header, only payload, can use K for 1024 bytes, M for 1024*1024 bytes, etc.


worker_node1:
  hostname: 10.10.xx.xx
  username: "server1"
  password: "xxooxx"
  port: 22
  groups:
    - worker_node_servers
  data:
    worker_node_server: "http://10.10.xx.xx:8000"
    retries: 5
    backoff_min_ms: 250.0
    backoff_max_ms: 500.0
    thread_count: 500
    on_site_hosts:
      h1: "192.168.yy.yy"
      # ... continue for h2–hzz

# Repeat similarly for worker_node2–worker_node4 with their ranges

Notes:

• ndtwin_kernel is the location of our NDTwin Kernel.
• recycle_interval is the interval for asking whether worker_node_server have finished some iperf3 and recycle the used ports.
• sleep_time defines the random waiting interval (in seconds) between starting iperf server and client to avoid “Connection Refused”. min and max set the lower and upper bounds.
• ports_limitation controls the port range used for flow generation:
  • min_port: The minimum port number available for iperf flows.
  • max_port: The maximum port number available for iperf flows.
  • exclude_ports: A list of ports to exclude from the available range (e.g., ports used by other services like the API server).
• log_level sets the logging verbosity. Supported values: TRACE, DEBUG, INFO, WARNING, ERROR.
• tcp_socket_write_length sets the TCP socket write buffer length for each write operation (payload only, excluding headers).
• udp_packet_length set the payloads length of UDP packets.(excluding headers).
• groups must be a YAML list (e.g., - worker_node_servers).
• data.worker_node_server must be reachable from the machine running network_traffic_generator.py.
• retries specifies the number of retry attempts for worker nodes to restart iperf client when some connection errors occur.
• backoff_min_ms and backoff_max_ms define the minimum and maximum backoff time (in milliseconds) between retry attempts.
• thread_count sets the maximum number of concurrent threads for starting iperf client/server on each worker node.
• data.on_site_hosts maps logical host names (e.g., h1) to IP addresses. If you define lots of hosts in one worker node, please list all of them.
• Our NTG will distribute the role of client/server in iperf using the logical host names from the NDTwin Kernel. Thus, please make sure the logical host name is the same in NDTwin Kernel and NTG.

setting/Mininet.yaml

Mininet_Testbed:
  hostname: "mininet_testbed"
  data:
    ndtwin_kernel: "http://127.0.0.1:8000"
    mode: "cli"
    log_level: "DEBUG"
    sleep_time:
      min: 0.4
      max: 1.3
    tcp_socket_write_length: "10K"  
    udp_packet_length: "1460"

Notes:

• mode can be cli (Mininet’s CLI) or custom_command (our generator’s cli)
• When using it, please make sure that Mininet is on you’re machine.

Flow Generation Configuration

flow Command Configuration

flow_template.json defines intervals and, for each interval, the flow mix and parameters. Key sections:

• traffic_generator(iperf_or_iperf3): iperf3 or iperf (tcp/udp supported)
• intervals: Array of interval objects. Each interval may include varied_traffic or fixed_traffic or both.

Common fields inside varied_traffic:

• flow_arrival_rate(flow/sec): Average flow arrivals per second during the interval.
• flow_distance_probability: Probability distribution over near/middle/far path selection.
• flow_type_probability: Probability distribution over flow types (tcp/udp, limited/unlimited size/rate/duration).
• flow_parameters: Per-type parameter map defining size(bytes), rate(bits), and/or duration(sec) when applicable.

Common fields inside fixed_traffic:

• fixed_flow_number: Exact number of flows to generate in the interval.
• flow_distance_probability, flow_type_probability, flow_parameters: Same semantics as above.

Parameter format rules:

• size(bytes): integer or suffixed with K, M, G (e.g., 10K, 2M)   - Defining how many bytes you want to send.
• rate(bits): integer or suffixed with K, M, G (e.g., 37.4M allowed where implementation supports decimals)   - Defining the sending rate of flows.
• duration(sec): positive integer seconds.    - Defining how long will the flow keep alive.
• packet_payload_size(bytes): integer or suffixed with K, M, G (e.g., 1024, 2K)
  • Defining the payload size of each packet (optional parameter for fine-tuning packet size).
  • Can be added to any flow type.
• Provide only the parameters that make sense for the chosen type:   - limited size → requires size(bytes)   - limited rate → requires rate(bits)   - limited duration → requires duration(sec)   - udp types include -u internally; tcp types do not.

Supported flow types for varied_traffic:

• unlimited_size_unlimited_rate_unlimited_duration_tcp
• unlimited_size_unlimited_rate_limited_duration_tcp
• limited_size_unlimited_rate_tcp
• unlimited_size_limited_rate_unlimited_duration_tcp
• unlimited_size_limited_rate_limited_duration_tcp
• limited_size_limited_rate_tcp
• unlimited_size_unlimited_rate_unlimited_duration_udp
• unlimited_size_unlimited_rate_limited_duration_udp
• limited_size_unlimited_rate_udp
• unlimited_size_limited_rate_unlimited_duration_udp
• unlimited_size_limited_rate_limited_duration_udp
• limited_size_limited_rate_udp

Supported flow types for fixed_traffic :

• limited_size_unlimited_rate_tcp
• unlimited_size_limited_rate_limited_duration_tcp
• unlimited_size_limited_rate_unlimited_duration_tcp
• limited_size_limited_rate_tcp
• unlimited_size_limited_rate_unlimited_duration_udp
• unlimited_size_limited_rate_limited_duration_udp
• limited_size_limited_rate_udp

Note: for each section (varied_traffic or fixed_traffic), the keys in flow_type_probability and flow_parameters must be chosen from the corresponding list above.

Example snippet (already present in flow_template.json):

{
  "traffic_generator(iperf_or_iperf3)": "iperf3",
  "intervals": [
    {
      "interval_duration(d/h/m/s)": "3s",
      "varied_traffic": {
        "flow_arrival_rate(flow/sec)": 5,
        "flow_distance_probability": {"near": 0.0, "middle": 0.5, "far": 0.5},
        "flow_type_probability": {
          "limited_size_limited_rate_tcp": 0.1,
          "limited_size_unlimited_rate_udp": 0.9
        },
        "flow_parameters": {
          "limited_size_limited_rate_tcp": {"size(bytes)": "2M", "rate(bits)": "30M"},
          "limited_size_unlimited_rate_udp": {"size(bytes)": "4K"}
        }
      },
      "fixed_traffic":{
        "fixed_flow_number": 8,
        "flow_distance_probability": {"near": 0.0, "middle": 0.3, "far": 0.7},
        "flow_type_probability": {
          "limited_size_limited_rate_tcp": 0.2,
          "limited_size_unlimited_rate_udp": 0.8
        },
        "flow_parameters": {
          "limited_size_limited_rate_tcp": {"size(bytes)": "5K", "rate(bits)": "5M"},
          "limited_size_unlimited_rate_udp": {"size(bytes)": "4K"}
        }
      }
    }
  ]
}

Notice that the limited_duration in fixed_traffic will use interval_duration(d/h/m/s) as the default duration setting. If you would like to use the interval_duration(d/h/m/s) as the flow duration, then just make it empty as below

{
    "interval_duration(d/h/m/s)": "10s",
    "fixed_traffic": {
        "fixed_flow_number": 20,
        "flow_distance_probability": {
            "near": 0.0,
            "middle": 0.5,
            "far": 0.5
        },
        "flow_type_probability": {
            "unlimited_size_limited_rate_limited_duration_tcp": 1.0
        },
        "flow_parameters": {
            "unlimited_size_limited_rate_limited_duration_tcp": {
                "rate(bits)": "37.4M"
            }
        }
    }
}

dist Command Configuration

dist_template.json is the same as flow_template.json but the parameter of flows are come from distribution files.

The distribution file is the measured network flow properties from data centers or ISP. For using dist command, you need to generate 3 kinds of distribution files:

1. distribution_flow_duration.csv : the distribution of flow’s life span.
2. distribution_flow_sending_rate.csv : the distribution of flow’s sending rate
3. distribution_flow_size.csv : the distribution of flow’s sent file size.

Each of the distribution files are used to replace the duration(sec), rate(bits), size(bytes) parameter of flow_template.json.

The distribution files is csv formate and the content likes below:

bin_midpoint,probability
1024,0.3
2048,0.5
4096,0.2

• bin_midpoint: The representative value for that bin (e.g., average flow size in bytes).
• probability: The probability of a flow having the characteristic of this bin. The sum of probabilities should be 1.0.

Example snippet (already present in dist_template.json):

{
    "traffic_generator(iperf_or_iperf3)": "iperf3",
    "flow_size_csv": "./distribution_flow_size.csv",
    "flow_duration_csv": "./distribution_flow_duration.csv",
    "flow_sending_rate_csv": "./distribution_flow_sending_rate.csv",
    "intervals": [
        {
            "interval_duration(d/h/m/s)": "2s",
            "fixed_traffic": {
                "fixed_flow_number": 5,
                "flow_distance_probability": {
                    "near": 0.0,
                    "middle": 0.5,
                    "far": 0.5
                },
                "flow_type_probability": {
                    "unlimited_size_limited_rate_limited_duration_tcp": 1.0
                }
            }
        },
        {
            "interval_duration(d/h/m/s)": "3s",
            "varied_traffic": {
                "flow_arrival_rate(flow/sec)": 2,
                "flow_distance_probability": {
                    "near": 0.0,
                    "middle": 0.3,
                    "far": 0.7
                },
                "flow_type_probability": {
                    "unlimited_size_unlimited_rate_limited_duration_tcp": 0.3,
                    "limited_size_unlimited_rate_tcp": 0.3,
                    "unlimited_size_limited_rate_limited_duration_tcp": 0.4
                }
            }
        }
    ]
}

Notice: we do not support setting packet_payload_size(bytes) parameter in any flow type when using dist command.

Support Commands in NTG

Commands

Our NTG support commands as below :

• flow : generate flows with defined parameters
• dist : flows’ parameters come from distribution files.
• exit : exit the NTG.

Notice that when using flow and dist command, you must add --config <json files> to specify which flow configuration files to use in one experiment.