How to Calculate TCP throughput for long distance WAN/MAN links

So you’ve just turned up your new high-speed link between Data Centers but are unpleasantly surprised to see relatively slow file transfers across this high speed, long distance link. Before you call Cisco TAC and start trouble shooting your network, do a quick calculation of what you should realistically expect in terms of TCP throughput from a one host to another over this long distance link.

Note: This post is specifically for colleagues located in Asia who continue to complain about a 2 mbit glass window in their uploads to the US, 300 milliseconds away.

When using TCP to transfer data the two most important factors are the TCP window size and the round trip latency. If you know the TCP window size and the round trip latency you can calculate the maximum possible throughput of a data transfer between two hosts, regardless of how much bandwidth you have.

Formula to Calculate TCP throughput:

TCP-Window-Size-in-bits / Latency-in-seconds = Bits-per-second-throughput

So lets work through a simple example. I have a 1Gig Ethernet link from Chicago to New York with a round trip latency of 30 milliseconds. If I try to transfer a large file from a server in Chicago to a server in New York using FTP, what is the best throughput I can expect?

First lets convert the TCP window size from bytes to bits.  In this case we are using the standard 64KB TCP window size of a Windows machine.

64KB = 65536 Bytes.   65536 * 8 = 524288 bits

Next, lets take the TCP window in bits and divide it by the round trip latency of our link in seconds.  So if our latency is 30 milliseconds we will use 0.030 in our calculation.

524288 bits / 0.030 seconds = 17476266 bits per second throughput = 17.4 Mbps maximum possible throughput

So, although I may have a 1GE link between these Data Centers I should not expect any more than 17Mbps when transferring a file between two servers, given the TCP window size and latency.

What can you do to make it faster?   Increase the TCP window size and/or reduce latency.

To increase the TCP window size you can make manual adjustments on each individual server to negotiate a larger window size.  This leads to the obvious question:  What size TCP window should you use?  We can use the reverse of the calculation above to determine optimal TCP window size.

Formula to calculate the optimal TCP window size:

Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes

So in our example of a 1GE link between Chicago and New York with 30 milliseconds round trip latency we would work the numbers like this…

1,000,000,000 bps * 0.030 seconds = 30,000,000 bits / 8 = 3,750,000 Bytes

Therefore if we configured our servers for a 3750KB TCP Window size our FTP connection would be able to fill the pipe and achieve 1Gbps throughput.

One downside to increasing the TCP window size on your servers is that it requires more memory for buffering on the server, because all outstanding unacknowledged data must be held in memory should it need to be retransmitted again.  Another potential pitfall is performance (ironically) where there is packet loss, because any lost packets within a window requires that the entire window be retransmitted – unless your TCP/IP stack on the server employs a TCP enhancement called “selective acknowledgements”, which most do not.

Another option is to place a WAN accelerator at each end that uses a larger TCP window and other TCP optimizations such as TCP selective acknowledgements just between the accelerators on each end of the link, and does not require any special tuning or extra memory on the servers.  The accelerators may also be able to employ Layer 7 application specific optimizations to reduce round trips required by the application.

Reduce latency?  How is that possible?  Unless you can figure out how to overcome the speed of light there is nothing you can do to reduce the real latency between sites.  One option is, again, placing a WAN accelerator at each end that locally acknowledges the TCP segments to the local server, thereby fooling the servers into seeing very low LAN like latency for the TCP data transfers.  Because the local server is seeing very fast local acknowledgments, rather than waiting for the far end server to acknowledge, is the very reason why we do not need to adjust the TCP window size on the servers.

In this example the perfect WAN accelerator would be the Cisco 7371 WAAS Appliance, as it is rated for 1GE of optimized throughput.

WAAS stands for:  Wide Area Application Services

The two WAAS appliances on each end would use TCP optimizations over the link such as large TCP windows and selective acknowledgements.  Additionally, the WAAS appliances would also remove redundant data from the TCP stream resulting in potentially very high levels of compression.  Each appliance remembers previously seen data, and if that same chunk of data is seen again, that data will be removed and replaced with a tiny 2 Byte label.  That tiny label is recognized by the remote WAAS appliance and it replaces the tiny label with the original data before sending the traffic to the local server.

The result of all this optimization would be higher LAN like throughput between the server in Chicago and New York without any special TCP tuning on the servers.

Formula to calculate Maximum Latency for a desired throughput:

You might want to achieve 10 Gbps FTP throughput between two servers using standard 64KB TCP window sizes.  What is the maximum latency you can have between these two servers to achieve 10 Gbps?

TCP-window-size-bits / Desired-throughput-in-bits-per-second = Maximum RTT Latency

524288 bits / 10,000,000,000 bits per second = 52.4 microseconds

Comments?