README.md 18.1 KB
Newer Older
Tom Henderson's avatar
Tom Henderson committed
1 2 3 4 5 6 7 8
# ns-3 tests for tsvwg scenarios

This repository contains scripts and results using
[ns-3](https://www.nsnam.org) to investigate potential
[issues](https://trac.ietf.org/trac/tsvwg/report/1?sort=ticket&asc=1&page=1)
raised by [IETF Transport Area Working Group](https://datatracker.ietf.org/group/tsvwg/about/)
participants about the L4S architecture.  

9 10 11 12
This repository currently focuses on [Issue #16 - Interaction w/ 3168-only ECN AQMs.](https://trac.ietf.org/trac/tsvwg/ticket/16) and [Issue #17 - Interaction w/ FQ AQMs.](https://trac.ietf.org/trac/tsvwg/ticket/17)   

Current simulation results for [issue 17, one flow](https://l4s.cablelabs.com/results.html) are posted on a CableLabs server.

13 14 15
The most recent changes to the code can be viewed from the 
[commit log](https://gitlab.com/tomhend/modules/l4s-evaluation/commits/master).

16 17 18 19 20 21 22 23
This simulation code can be parameterized to simulate and compare with selected results of 

* [Scenario 2 (single fq-codel)](https://github.com/heistp/sce-l4s-bakeoff#scenario-2),
* [Scenario 3 (single codel)](https://github.com/heistp/sce-l4s-bakeoff#scenario-3),
* [Scenario 5 (consecutive bottleneck](https://github.com/heistp/sce-l4s-bakeoff#scenario-5), and
* [Scenario 6 (consecutive bottleneck)](https://github.com/heistp/sce-l4s-bakeoff#scenario-6)

in the [testbed results](https://github.com/heistp/sce-l4s-bakeoff) posted by 
Tom Henderson's avatar
Tom Henderson committed
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Pete Heist, which investigates a similar configuration, as well as related
[testbed results](https://l4s.cablelabs.com/l4s-testing/README.html) from Olivier Tilmans.

## Table of Contents

1. [Introduction](#toc_2)
2. [Simulation Setup](#toc_3)
3. [Simulation Outputs](#toc_4)
4. [Scenarios and Results](#toc_5)
5. [Installation](#toc_6)
6. [Usage](#toc_7)
7. [Future Work](#toc_8)

## Introduction

39
ns-3 provides an accessible framework for simulation-based experiments and allows for completely reproducible results, to complement testbed results.  ns-3 also provides a limited capability for inserting Linux kernel TCP code, which we have used for validation of results using native ns-3 models.
Tom Henderson's avatar
Tom Henderson committed
40 41

ns-3 models exist for some of the configurations under study in testbeds.
42
This repository extends the ns-3.31 release with additional models:
43 44 45 46 47 48 49

- TCP Cubic model derived from [RFC 8312](https://tools.ietf.org/html/rfc8312)
- scripts to run scenarios of interest to tsvwg

This repository also contains scripts to automate the running and processing
of simulation scenarios proposed in the IETF transport area working group.

Tom Henderson's avatar
Tom Henderson committed
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69
## Simulation Setup

The general simulation setup for issues 16 and 17 can be implemented in
ns-3 with the following generic topology:

```
// ---> downstream (primary data transfer from servers to clients)
// <--- upstream (return acks and ICMP echo response)
//
//  links:  (1)      (2)      (3)      (4)      (5)      (6)
//              ----     ----     ----     ----     ----
//  servers ---| WR |---| M1 |---| M2 |---| M3 |---| LR |--- clients
//              ----     ----     ----     ----     ----
//  ns-3 node IDs:
//  nodes 0-2    3        4        5        6         7       8-10
//
// The use of 'server' and 'client' terminology is consistent with RFC 1983
// terminology in that home clients are requesting data from Internet servers
//
// - The box WR is a WAN router, aggregating all server links
70
// - The box M1 is notionally an access network headend such as a CMTS or BRAS
Tom Henderson's avatar
Tom Henderson committed
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141
// - The box M2 is notionally an access network CPE device such as a cable or DSL modem
// - The box M3 is notionally a home router (HR) running cake or FQ-CoDel
// - The box LR is another LAN router, aggregating all client links (to home devices)
// - Three servers are connected to WR, three clients are connected to LR
//
// clients and servers are configured for ICMP measurements and TCP throughput
// and latency measurements in the downstream direction
//
// Depending on the 'scenario', the middleboxes and endpoints will be
// configured differently.
```

The file `l4s-evaluation/examples/tsvwg-scenarios.cc` implements the
above topology.  The node depicted as M3 implements the rate shaping
on the notional home gateway, and the node depicted as M1 implements
rate shaping at a slightly higher rate.

In addition, the simulation configures one client-server pair for a
ping process (default 100ms interval) and provides two client-server
pairs for long running TCP-based file transfers.

A number of command line options are available for key configuration details,
including the ability to disable the M1 bottleneck (this is called a 
`control scenario`), to change the bottleneck link rates, to change 
the queueing discipline on M3, and to configure an IAQM response and
its delay threshold on CoDel.

## Simulation Outputs

The simulation is instrumented to output several traces into raw data files
(suffixed with `.dat`):

* ping RTT latency trace
* TCP cwnd and estimated RTT traces
* Time series of the file transfer throughput, measured over 200ms sampling intervals
* Length of the M1 and M3 queues, expressed in terms of milliseconds of queueing delay at the bottleneck link rate
* All drops and marks from queues M1 and M3
* Time series of the frequency of drops and marks over 100ms intervals

### Simulation Plots

Using Python matplotlib and pdfjam utilities, several Python programs and
a shell script (`plot.sh`) convert the raw data into a single-page PDF
with a number of subplots, an example of which is shown 
[here](https://l4s.cablelabs.com/results/6-dctcp-20191018-214743/6-dctcp_29_50Mbps_0.95_80ms.pdf).  This example shows single flow DCTCP performance across
the Reading across first by row and then by column, the
plots show (for single flow TCP experiments) plots of the ICMP RTT latency,
the M1 queue occupancy (in ms), a zoom of the M1 queue occupancy over a
shorter timescale (simulation time 5-10 seconds), the TCP RTT estimate,
the M3 queue occupancy (in ms), the number of CE marks per 100 ms, the
TCP throughput (measured at the application layer on the receiver), and
the TCP congestion window (in units of segments).

These plotting scripts are available in an `experiments` directory in 
the module, making it easier to configure, run, and post-process an 
experiment compared with running each simulation by hand and manually 
gathering and processing output data.

### Larger scale experiments

Additional Bash scripts are available to control simulation experiments
that sweep through multiple parameter value combinations.  The Bash
script `run-single.sh` will allow the user to set some initial variables
and automate the generation of the plot of a single run of the simulator.
The Bash scripts `sweep.sh` and `run-all.sh` build on this capability
to control the parametric sweep through multiple parameter values.

In the experiment directory
`l4s-evaluation/experiments/tsvwg-issue-17-one-flow`, the existing script
`run-all.sh` is configured to run 180 simulations and generate the HTML
page to render the results shown [here](https://l4s.cablelabs.com/results.html).
142
A `run-single.sh` is available to run one parameter combination.
Tom Henderson's avatar
Tom Henderson committed
143 144 145 146 147 148 149 150

The simulations are run with all combinations of the following parameter values:

* congestion controller: {reno, cubic, dctcp}
* FIFO rate (link3rate): {10Mbps, 20Mbps, 50Mbps, 100Mbps, 200Mbps}
* fq_codel shaper rate ratio (link5rateRatio):  {0.95, 0.9}
* base RTT: {0ms, 10ms, 20ms, 50ms, 80ms, 100ms}

151 152 153 154 155 156 157 158 159 160 161 162 163
In the experiment directory
`l4s-evaluation/experiments/tsvwg-issue-17-two-flows`, the existing script
`run-all.sh` is configured to run multiple simulations corresponding to the
two flow cases.  Results are not yet posted.
A `run-single.sh` is available to run one parameter combination.

The simulations are run with all combinations of the following parameter values:

* congestion controller: {'cubic-cubic', 'dctcp-dctcp', 'cubic-dctcp', 'dctcp-cubic'}
* FIFO rate (link3rate): {10Mbps, 20Mbps, 50Mbps, 100Mbps}
* fq_codel shaper rate ratio (link5rateRatio):  {0.95}
* base RTT: {0ms, 10ms, 20ms, 50ms, 80ms}

Tom Henderson's avatar
Tom Henderson committed
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205
## Scenarios and Results

### Observations on tsvwg issue 17

The resulting 180 scenarios are plotted [here](https://l4s.cablelabs.com/results.html).
In all tested conditions, we see that the impact on the sparse ICMP flow from the presence of the TCP flow is largely the same regardless of which congestion controller is used.  We do not see a 4 second peak in the ICMP latency caused by the presence of DCTCP.  

We do see, as expected, that in low RTT cases the slow linear ramp-up in time of the CoDel marking rate results in several seconds of DCTCP latency that is greater than the CoDel threshold of 5ms. 

Additionally we see that, even in a situation in which the AQM algorithm is not ideal for the DCTCP congestion controller, DCTCP generally provides better throughput at lower latency than either cubic or reno.

### Comparison to "Scenario 6" by Pete Heist
The description of Pete Heist's Scenario 6 aligns with the Issue 17 experiment described by Jonathan Morton.

> This is similar to Scenario 5, but narrowed down to just the FIFO and CoDel
> combination.  Correct behaviour would show a brief latency peak caused by the
> interaction of slow-start with the FIFO in the subject topology, or no peak at
> all for the control topology; you should see this for whichever
> [RFC 3168](https://tools.ietf.org/html/rfc3168) flow is chosen as the control.
> Expected results with L4S in the subject topology, however, are a peak
> extending about 4 seconds before returning to baseline.

Our simulation results are most closely related to the ["L4S one-flow"](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/) results from Pete Heist. The topology in that case was labelled 'L4S' but is really about Prague or Cubic operating over FIFO and FQ-CoDel middleboxes (i.e. there is no L4S dualq-coupled-aqm nor any immediate AQM).  The topology in that experiment was:

> L4S: Sender → Delay → FIFO middlebox (bottleneck #1, 52.5Mbit) → FQ-AQM middlebox (bottleneck #2, 50Mbit) → L4S receiver

Pete's experiment used the following RTTs: ~0ms, 10ms, 80ms (implemented in the "Delay" element).

So our 50Mbps, 95% cases with RTTs of (0ms, 10ms, 80ms) are the most comparable to Pete's cases (which were 52.5Mbps instead of 50Mbps).

| Congestion Control | Bottleneck Rate / fq-codel Rate  (Mbps) | RTT (ms) | PH Experiment | NS3 Experiment |
|:-------------:|:-------------:|:-----:|:-----:|:-----:|
| cubic | 50/47.5 | 0 |  [batch-l4s-s6-1-cubic-50Mbit-0ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-cubic-50Mbit-0ms_var.png)  |  [6-cubic_25_50Mbps_0.95_0ms.pdf](https://l4s.cablelabs.com/results/6-cubic-20191018-205613/6-cubic_25_50Mbps_0.95_0ms.pdf)  |
| cubic | 50/47.5  | 10 |  [batch-l4s-s6-1-cubic-50Mbit-10ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-cubic-50Mbit-10ms_var.png)  |   [6-cubic_26_50Mbps_0.95_10ms.pdf]( https://l4s.cablelabs.com/results/6-cubic-20191018-205613/6-cubic_26_50Mbps_0.95_10ms.pdf) |
| cubic | 50/47.5  | 80 |  [batch-l4s-s6-1-cubic-50Mbit-80ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-cubic-50Mbit-80ms_var.png)  |    [6-cubic_29_50Mbps_0.95_80ms.pdf](https://l4s.cablelabs.com/results/6-cubic-20191018-205613/6-cubic_29_50Mbps_0.95_80ms.pdf) |
| dctcp | 50/47.5  | 0 | [batch-l4s-s6-1-dctcp-50Mbit-0ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-dctcp-50Mbit-0ms_var.png) |   [6-dctcp_25_50Mbps_0.95_0ms.pdf](https://l4s.cablelabs.com/results/6-dctcp-20191018-214743/6-dctcp_25_50Mbps_0.95_0ms.pdf) |
| dctcp | 50/47.5  | 10 |  [batch-l4s-s6-1-dctcp-50Mbit-10ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-dctcp-50Mbit-10ms_var.png)  | [6-dctcp_26_50Mbps_0.95_10ms.pdf](https://l4s.cablelabs.com/results/6-dctcp-20191018-214743/6-dctcp_26_50Mbps_0.95_10ms.pdf)   |
| dctcp | 50/47.5 | 80 |  [batch-l4s-s6-1-dctcp-50Mbit-80ms_var.png](https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s6-1/batch-l4s-s6-1-dctcp-50Mbit-80ms_var.png)  |    [6-dctcp_29_50Mbps_0.95_80ms.pdf](https://l4s.cablelabs.com/results/6-dctcp-20191018-214743/6-dctcp_29_50Mbps_0.95_80ms.pdf) |


## Installation

206
The current installation relies on using ns-3.31 or development version.
Tom Henderson's avatar
Tom Henderson committed
207 208 209

1) Install prerequisites for ns-3 on your workstation.  The minimal
   requirement for Linux is a C++ (g++ or clang++) compiler, Python 3, and git. 
210 211
   For macOS, install the Xcode command-line tools (or the full Xcode
   environment if you prefer).
Tom Henderson's avatar
Tom Henderson committed
212 213 214 215 216 217
   More details about various prerequisites for installation are provided
   on the [installation wiki](https://www.nsnam.org/wiki/Installation). 

2) To run the plotting scripts, Python Matplotlib and pdfjam are needed.
   Install with your applicable package manager.

218 219 220
3) Download ns-3

3.1) Git clone or fork https://gitlab.com/nsnam/ns-3-dev.git
Tom Henderson's avatar
Tom Henderson committed
221

222
3.2) If you prefer a release version, download the most recent release of ns-3: [ns-3.31](https://www.nsnam.org/releases/ns-allinone-3.31.tar.bz2).
Tom Henderson's avatar
Tom Henderson committed
223

224
4) We will install the `l4s-evaluation` module in the contributed code
Tom Henderson's avatar
Tom Henderson committed
225 226 227 228 229 230 231 232 233
   directory.  This module is on a separate GitLab.com repository.  Cd
into the `contrib` directory and clone from
   [GitLab.com](https://gitlab.com/tomhend/modules/l4s-evaluation).

```
$ cd contrib
$ git clone https://gitlab.com/tomhend/modules/l4s-evaluation
```

234
5) Next, configure and build the program, and confirm that unit tests pass:
Tom Henderson's avatar
Tom Henderson committed
235 236 237 238 239 240 241 242 243 244 245 246 247

```
$ ./waf configure -d optimized --enable-examples --enable-tests
$ ./waf build
$ ./test.py 
```

At this point, you are ready to start using the l4s-evaluation program and
scripts.

## Usage

Users unfamiliar with ns-3 are encouraged to read the 
248
[ns-3 tutorial](https://www.nsnam.org/docs/release/3.31/tutorial/html/index.html) (also available in [PDF format](https://www.nsnam.org/docs/release/3.31/tutorial/ns-3-tutorial.pdf)) to get started.
Tom Henderson's avatar
Tom Henderson committed
249 250 251 252 253 254 255 256 257 258 259

Try running the main `tsvwg-scenarios.cc` program from the
command line, to output its help statement:

```
$ ./waf --run 'tsvwg-scenarios --PrintHelp'
```

You should see a listing of program options:

```
260
ns3.31-tsvwg-scenarios-optimized [Program Options] [General Arguments]
Tom Henderson's avatar
Tom Henderson committed
261 262 263

Program Options:
    --firstTcpType:                 First TCP type (cubic, dctcp, or reno) [cubic]
264
    --secondTcpType:                Second TCP type (cubic, dctcp, or reno) [cubic]
Tom Henderson's avatar
Tom Henderson committed
265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295
    --m3QueueType:                  M3 queue type (fq or codel) [fq]
    --baseRtt:                      base RTT [+80000000.0ns]
    --controlScenario:              control scenario (disable M1 bottleneck) [false]
    --useIaqm:                      useIaqm [false]
    --iaqmThreshold:                CoDel IAQM threshold [+1000000.0ns]
    --link3rate:                    data rate of link 3 for FIFO scenarios [50000000bps]
    --link5rateRatio:               ratio of data rate of link 5 to link 3 [0.95]
    --stopTime:                     simulation stop time [+70000000000.0ns]
    --enablePcap:                   enable Pcap [false]
    --pingTraceFile:                filename for ping tracing [tsvwg-scenarios-ping.dat]
    --firstTcpRttTraceFile:         filename for rtt tracing [tsvwg-scenarios-first-tcp-rtt.dat]
    --firstTcpCwndTraceFile:        filename for cwnd tracing [tsvwg-scenarios-first-tcp-cwnd.dat]
    --firstTcpThroughputTraceFile:  filename for throughput tracing [tsvwg-scenarios-first-tcp-throughput.dat]
    --m1DropTraceFile:              filename for m1 drops tracing [tsvwg-scenarios-m1-drops.dat]
    --m1DropsFrequencyTraceFile:    filename for m1 drop frequency tracing [tsvwg-scenarios-m1-drops-frequency.dat]
    --m1LengthTraceFile:            filename for m1 queue length tracing [tsvwg-scenarios-m1-length.dat]
    --m3MarkTraceFile:              filename for m3 mark tracing [tsvwg-scenarios-m3-marks.dat]
    --m3MarksFrequencyTraceFile:    filename for m3 mark frequency tracing [tsvwg-scenarios-m3-marks-frequency.dat]
    --m3DropTraceFile:              filename for m3 drop tracing [tsvwg-scenarios-m3-drops.dat]
    --m3LengthTraceFile:            filename for m3 queue length tracing [tsvwg-scenarios-m3-length.dat]

General Arguments:
    --PrintGlobals:              Print the list of globals.
    --PrintGroups:               Print the list of groups.
    --PrintGroup=[group]:        Print all TypeIds of group.
    --PrintTypeIds:              Print all TypeIds.
    --PrintAttributes=[typeid]:  Print all attributes of typeid.
    --PrintHelp:                 Print this help message.
```

The program can be run from the top-level directory, but some shell scripts
296
are available in the module experiments directory.  cd into the following
Tom Henderson's avatar
Tom Henderson committed
297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315
directory and run the `run-single.sh` script:

```
$ cd contrib/l4s-evaluation/experiments/tsvwg-issue-17-one-flow
$ ./run-single.sh
```

This will show that a single flow experiement using DCTCP, an 80 ms base RTT,
an M1 bottleneck link rate of 50 Mbps, and a link 5 to link 3 ratio of 0.95
(i.e. M3 bottleneck is 47.5 Mbps); this corresponds to Pete Heist's scenario
5 configuration.  Results will be stored in a timestamped `results` directory,
including data files and resulting PDF plots.

The script `run-all.sh` will execute the script `sweep.sh` and generate a
HTML page similar to the one posted on the 
[CableLabs L4S site](https://l4s.cablelabs.com/results.html).

## Future work

316
We plan to add example programs and experiments as needed.  We plan to add ns-3
Tom Henderson's avatar
Tom Henderson committed
317 318 319 320 321 322 323 324 325 326
models of dual queue coupled PI2 and TCP Prague, and to do further work
on CoDel's IAQM support. 

TCP Cubic and DCTCP models will be added to mainline ns-3 when ready.  We 
are performing further validation and writing unit tests before proposing
them to mainline.  The congestion window response to an ECE signal requires
further validation (although response to a loss will invoke PRR and
fast recovery, the current response to an ECE signal is to simply perform a
window halving (reno, dctcp) or beta reduction (cubic)).

327 328
For questions about the model, please use the
[ns-3-users Google Groups](https://groups.google.com/forum/?fromgroups#!forum/ns-3-users)
Tom Henderson's avatar
Tom Henderson committed
329 330 331 332
forum, and feel free to generate GitLab.com merge requests for proposed
patches.   All code contributions should follow 
[ns-3 guidelines](https://www.nsnam.org/develop/contributing-code/)
for contributed code.