Why your maximum throughput is less than your bandwidth.
This posting is more technical than most on this blog. It was motivated by a recent customer that has a 6Mbps MPLS network connecting Boston to China. The network was been performing very well. But then they recently started using FTP to transfer files and they could not attain anywhere close to 6Mbps of throughput. This entry goes into the details.
TCP provides reliable transport of data between devices. The sender sends data in a stream in which bytes are identified by sequence numbers. The receiver acknowledges that it has received the data by sending the sequence number (acknowledgment number) of the next byte of data that it expects to receive. The receiver also advertises its TCP Window Size to the sender to advertise the amount of data that it can accept.
This “TCP Window” is the maximum number of bytes that can be transmitted by the sender without being acknowledged as having been successfully received by the other end. This means the receiver can safely expect that no more than window-size bytes of data need to be buffered for that connection, and that any additional data beyond that in sequence order can legitimately be discarded (because it is so defined by the TCP protocol).
When you have a high latency connection, such as satellite or a very long distance, the latency has a big effect on thru-put since TCP specifies that acknowledgements must be sent and received as the data is streaming.
How can you determine your maximum transfer rate? The formula below will do the trick. Just remember that this formula does not account for packet loss, so reduce the number to account for this:
Maximum Possible Transfer Rate = TCP Window Size/RTT
Where RTT is Ping Response in Millseconds/1000.
An example:
- Ping time from NY to Los Angeles: 70ms. Therefore, RTT= 70/1000=.07
- Maximum Possible Transfer Rate = 32/.07 = 457 KiloBytes/sec or 3.657 Mbps (Remember: 1 byte = 8 bits) This tells you that for a single data stream, such as FTP, a circuit larger than 3.657Mbps will not allow you to transfer a file any faster than 3.657Mbps.
- It doesn’t matter how big your circuit is.
Here is another example:
- Ping time from NY to Shanghai is 280ms.Therefore, RTT = 280/1000=.28
- Maximum Possible Transfer Rate = 32/.28 = 114 KiloBytes/sec or 914Kbps.
- If you have an E1/T1, you will have full access to 1540 Kbps, but with a single data flow, such as FTP, you will be limited to 914Kbps.
Now what if you make configuration changes to your TCP Window size to 64K:
For the NY to LA example, 64/.07=914 KiloBytes/sec
For NY to Shanghai: 64/.28=228 KiloBytes/sec
So what do we learn from this?
There are only two things you can do to affect your data thru-put on a wide area network:
- Increase your TCP window size, or
- Reduce latency.
So a low latency connection makes a big difference on your thru-put. If latency is 1 second round-trip, the peak data rate can never exceed 65KB/second, which is 524Kbps, using a TCP Window of 65,535 bytes.
What size TCP window is needed to fill a 10Mbps circuit that has a 70ms latency between two locations? Here is the calculation:
.07 seconds x 10Mbps x 1byte/8bits = 87,500 bytes required window size to use entire bandwidth with one data stream.
Standard TCP allows a maximum window size of 64,000 bytes. Cisco’s WINScale TCP option allows you to configure a larger window size. You should experiment with this, since larger window sizes also mean larger retransmits when packets are lost, so a point of diminshing returns will be approached.
If you need more thru-put for FTP, run multiple sessions so that you can take advantage of the limitation that apply to a single data stream.
Here are some numbers to save you the calculations. If you configure a 64KB TCP Window Size, your maximum thoughtput, without packet loss, is the following:
| RTT | Max Throughput | Max Throughput | Max Throughput |
| Latency | 64K Window | 64K Window | Windows XP |
| In Kbps | In KBps | In Kbps | |
| 0.1 mS | 803,755 Kbps | 100,469.4 | 565,965 Kbps |
| 50 mS | 10,371 Kbps | 1,296.4 | 2,795 Kbps |
| 60 mS | 8,658 Kbps | 1,082.3 | 2,330 Kbps |
| 70 mS | 7,431 Kbps | 928.9 | 1,998 Kbps |
| 80 mS | 6,509 Kbps | 813.6 | 1,749 Kbps |
| 90 mS | 5,790 Kbps | 723.8 | 1,555 Kbps |
| 100 mS | 5,214 Kbps | 651.8 | 1,400 Kbps |
| 110 mS | 4,742 Kbps | 592.8 | 1,272 Kbps |
| 120 mS | 4,349 Kbps | 543.6 | 1,167 Kbps |
| 130 mS | 4,016 Kbps | 502.0 | 1,077 Kbps |
| 140 mS | 3,730 Kbps | 466.3 | 1,000 Kbps |
| 150 mS | 3,482 Kbps | 435.3 | 933 Kbps |
| 200 mS | 2,614 Kbps | 326.8 | 700 Kbps |
| 250 mS | 2,093 Kbps | 261.6 | 560 Kbps |
| 260 mS | 2,012 Kbps | 251.5 | 539 Kbps |
| 270 mS | 1,938 Kbps | 247.9 | 519 Kbps |
| 280 mS | 1,869 Kbps | 233.6 | 500 Kbps |
| 290 mS | 1,804 Kbps | 225.5 | 483 Kbps |
| 300 mS | 1,744 Kbps | 218.0 | 467 Kbps |
| 350 mS | 1,496 Kbps | 187.0 | 400 Kbps |
| 400 mS | 1,309 Kbps | 163.6 | 350 Kbps |
| 450 mS | 1,164 Kbps | 145.5 | 311 Kbps |
| 500 mS | 1,047 Kbps | 130.9 | 280 Kbps |
I hope that this post adds some clarity to an often misunderstood concept.
