Tuning at compile time

/LINUX-SOURCE-DIR/include/linux/skbuff.h


 /* These are just the default values. This is run time configurable.
  * FIXME: Probably the config option should go away. - erics
  */
#ifdef CONFIG_SKB_LARGE
#define SK_WMEM_MAX     65535
#define SK_RMEM_MAX      65535
#else
#define SK_WMEM_MAX     32767
#define SK_RMEM_MAX     32767
#endif

/LINUX-SOURCE-DIR/include/net/tcp.h

/*
 * Never offer a window over 32767 without using window scaling. Some
 * poor stacks do signed 16bit maths!
 */
#define MAX_WINDOW      32767
#define MIN_WINDOW      2048

This last item is what limits you to using only 15 bits of the window field in the TCP packet header. Suppose you wish to use a window of 40 kB. If you simply set the rmem_default to 40 kB, the stack will recognize that this is less than 64 kB and therefore will not negotiate a winshift. However, because of this second check, you will only get 32 kB. Therefore, you need to set the rmem_default to something larger than 64 kB in order to force a winshift=1, which then lets you express the desired 40 kB in only 15 bits (and in fact you'll probably then end up with 64 kB whether you want it or not).

I imagine that a better idea is to simply change this value for MAX_WINDOW to 65535 if you need windows larger than 32 kB. I haven't tested this out to see how well it works. Alas, this part of the code is somewhat hard to follow. I'd appreciate any comments on how well this works.

User testimonial: With the tuned TCP stacks it was possible to get a maximum throughput between 1.5 - 1.8 Mbit/s via a 2Mbit/s satellite link, measured with netperf.

Information about tuning for MacOS

I don't have detailed information, however, someone pointed me to a good website with useful information. The URL is http://www.sustworks.com/products/prod_ottuner.html. I don't endorse the product they are selling (since I've never tried it). However, it is available for a free trial, and they appear to do an excellent job of describing perf-tune issues for Macs.

Procedure for raising network limits under Microsoft Windows 98

New: Some folks at NLANR/MOAT in SDSC have written a tool to do guide you through some of this stuff. It can be found at http://moat.nlanr.net/Software/TCPtune/.

Even newer: I've updated some sending window information which was inaccurate. See below.

Several folks have recently helped me to figure out how to accomplish the necessary tuning under Windows98, and the features do appear to exist and work. Thanks to everyone for the assistance! The new description below should be useful to even the complete Windows novice (such as me :-).

Windows98 includes implementation of RFC1323 and RFC2018. Both are on by default. (However, with a default buffer size of only about 8kB, window scaling doesn't do much).

Windows stores the tuning parameters in the Windows Registry. In the registry are settings to toggle on/off Large Windows, Timestamps, and SACK. In addition, default socket buffer sizes can be specified in the registry.

In order to modify registry variables, do the following steps:

1. Click on Start -> Run and then type in "regedit". This will fire up the Registry Editor.
2. In the Registry Editor, double click on the appropriate folders to walk the tree to the parameter you wish to modify. For the parameters below, this means clicking on HKEY_LOCAL_MACHINE -> System -> CurrentControlSet -> Services -> VxD -> MSTCP.
3. Once there, you should see a list of parameters in the right half of your screen, and MSTCP should be highlighted in the left half. The parameters you wish to modify will probably not appear in the right half of your screen; this is OK.
4. In the menu bar, Click on "Edit -> New -> String Value". It is important to create the parameter with the correct type. All of the parameters listed below are strings.
5. A box will appear with "New Value #1"; change the name to the name listed below, exactly as shown. Hit return.
6. On the menu, click on "Edit -> Modify" (your new entry should still be selected). Then type in the value you wish to assign to the parameter.
7. Exit the registry editor, and reboot windows. (The rebooting is important, *sigh*.)
8. When your system comes back up, you should have access to the features you have just turned on. The only real way to verify this is through packet traces (or by noticing a significant performance improvement).

TCP/IP Stack Variables

Support for TCP Large Windows (TCPLW)

Win98 TCP/IP supports TCP large windows as documented in RFC 1323. TCP large windows can be used for networks that have large bandwidth delay products such as high-speed trans-continental connections or satellite links. Large windows support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP

The registry key Tcp1323Opts is a string value type. The values for Tcp1323Opt are

The default value for Tcp1323Opts is 3: Window Scaling and Time stamp options. Large window support is enabled if an application requests a Winsock socket to use buffer sizes greater than 64K. The current default value for TCP receive window size in Memphis TCP is 8196 bytes. In previous implementations the TCP window size was limited to 64K, this limit is raised to 2**30 through the use of TCP large window support.

Support for Selective Acknowledgements (SACK)

Win98 TCP supports Selective Acknowledgements as documented in RFC 2018. Selective acknowledgements allow TCP to recover from IP packet loss without resending packets that were already received by the receiver. Selective Acknowledgements is most useful when employed with TCP large windows. SACK support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP

The registry key SackOpts is a string value type. The values for SackOpts are

Support for Fast Retransmission and Fast Recovery

Win98 TCP/IP supports Fast Retransmission and Fast Recovery of TCP connections that are encountering IP packet loss in the network. These mechanisms allow a TCP sender to quickly infer a single packet loss by reception of duplicate acknowledgements for a previously sent and acknowledged TCP/IP packet. This mechanism is useful when the network is intermittently congested. The reception of 3 (default value) successive duplicate acknowledgements indicates to the TCP sender that it can resend the last unacknowledged TCP/IP packet (fast retransmit) and not go into TCP slow start due to a single packet loss (fast recovery). Fast Retransmission and Recovery support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP\Parameters

The registry key MaxDupAcks is DWORD taking integer values from 2 to N. If MaxDupAcks is not defined, the default value is 3.

Update: If you wish to set the default receiver window for applications, you should set the following key:

DefaultRcvWindow

HKLM\system\currentcontrolset\services\VXD\MSTCP

DefaultRcvWindow is a string type and the value describes the default receive windowsize for the TCP stack. Otherwise the windowsize has to be programmed in apps with setsockopt.

Misc Info about Windows NT

Editor's note: See Windows 98 above for a detailed description of how this all works. In NT land, the Registry Editor is called regedt32.

Any Registry Values listed appear in:
	HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Receive Window
	maximum value = 64kB, since window scaling is not supported
	default value = min( max( 4 x MSS, 
				  8kB rounded up to nearest multiple of MSS),
			     64kB) 
	Registry Value: 
		TcpWindowSize

Path MTU Discovery Variables:
		EnablePMTUDiscovery	(default = enabled)
			turn on/off path MTU discovery
		EnablePMTUBHDetect	(default = disabled)
			turn on/off Black Hole detection

Using Path MTU Discovery:
        EnablePMTUDiscovery     REG_DWORD
	Range: 0 (false) or 1 (true)
	Default: 1

    Determines whether TCP uses a fixed, default maximum transmission unit
    (MTU) or attempts to find the actual MTU. If the value of this entry is
    0, TCP uses an MTU of 576 bytes for all connections to computers outside
    of the local subnet. If the value of this entry is 1, TCP attempts to
    discover the MTU (largest packet size) over the path to a remote host.

Using Path MTU Discovery's "Blackhole Detection" algorithm:
        EnablePMTUBHDetect     REG_DWORD
	Range: 0 (false) or 1 (true)
	Default: 0 

    If the value of this entry is 1, TCP tries to detect black hole routers
    while doing Path MTU Discovery. TCP will try to send segments without
    the Don't Fragment bit set if several retransmissions of a segment go
    unacknowledged. If the segment is acknowledged as a result, the MSS will
    be decreased and the Don't Fragment bit will be set in future packets on
    the connection.

PMTU Discovery. If PMTU is turned on, NT 3.1 cannot cope with routers that have the BSD 4.2 bug (see RFC 1191, section 5). It loops resending the same packet. Only confirmed on NT 3.1.

Misc Info about Windows 95

Editor's note: See Windows 98 above for more detailed descriptions of how this all works. I haven't personally tested the Win95 info below.

New: A Patch is available for Win95 at the following URL: http://support.microsoft.com/support/kb/articles/q182/1/08.asp. This patch includes support for TCPLW and SACK. I haven't tried it out, but I assume that the info above on tuning Win98 will be useful.

Any Registry Values listed appear in:
	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VxD\MSTCP

Receive Window
	maximum value = 64kB, since window scaling is not supported
	default value = min( max( 4 x MSS, 
				  8kB rounded up to nearest multiple of MSS),
			     64kB) 

	Registry Value: 
		DefaultRcvWindow


Path MTU Discovery Variables:
	present

	Registry Values:
		PMTUDiscovery		(default = 1 (enabled))
			turn on/off path MTU discovery
		PMTUBlackHoleDetect	(default = 0 (disabled))
			turn on/off Black Hole detection

(See NT for more info on using PMTU discovery and black hole
detection).

TCP retries. Not strictly performance related but a common cause of TN3270 emulators dropping their sessions if the mainframe is busy for a second or two. Instead of retrying up to 240 seconds (RFC 1122, section 4.2.3.1), Windows 3.11 and 95 default to 5 retries without a time limit. Even with RTO doubling, on a fast link 5 retries gives up after less than a second of no response. Hkey_Local_Machine\System\CurrentControlSet\Services\VxD\MSTCP, add variable MaxDataRetries. I normally set it to 64.

Procedure for raising network limits under NetBSD

      sysctl net.inet.tcp.rfc1323

The maximum socket buffer size can be modified by changing SB_MAX in /usr/src/sys/sys/socketvar.h.

The default socket buffer sizes can be modified by changing TCP_SENDSPACE and TCP_RECVSPACE in /usr/src/sys/netinet/tcp_usrreq.c.

It may also be necessary to increase the number of mbufs, NMBCLUSTERS in /usr/src/sys/arch/*/include/param.h.

Update: It is also possible to set these parameters in the kernel configuration file.

options		SB_MAX=1048576		# maximum socket buffer size
options		TCP_SENDSPACE=65536	# default send socket buffer size
options		TCP_RECVSPACE=65536	# default recv socket buffer size
options		NMBCLUSTERS=1024	# maximum number of mbuf clusters

Procedure for raising network limits under SGI systems under IRIX 5.3 or 6.1

     /var/sysgen/master.d/bsd

• tcp_mtudisc: To enable MTU Discovery, set the variable tcp_mtudisc = 1. [Note: I have received reports of poor performance with MTU discovery on the Iconwest/Phobos G100 fast ethernet card. I would like to hear more about this problem if anyone else is experiencing it.]

• tcp_sendspace, tcp_recvspace: To increase the default socket buffer size for TCP, set the variables tcp_sendspace and tcp_recvspace to the desired value (in byes). Under IRIX 5.3, the maximum socket buffer size allowed is 512 kB. Under IRIX 6.x this limit has been increased to 1 MB, and under future releases it is rumored that there will be (effectively) no limit on these TCP socket buffer size. [Note: Under IRIX 6.2, the comment in the bsd file says there is still a limit of 512 kB. We are running 1 MB on some of our systems here, and I am searching out the answer on what the real max is.]

• tcp_winscale controls the use of RFC1323 winshift. It is turned on by default and need not be modified.

Only slightly related to this page, SGI Hippi performance info.

Procedure for raising network limits under SGI systems under IRIX 6.5

Under this version, there are two locations where configuration is done. Although I list the BSD information first, SGI recommends using systune which is described below.

The BSD values are now stored in /var/sysgen/mtune/bsd.

For instance from the file:

* name                  default         minimum   maximum
*
* TCP window sizes/socket space reservation; limited to 1Gbyte by RFC
1323
*
tcp_sendspace                   61440   2048    1073741824
tcp_recvspace                   61440   2048    1073741824

These variables are used similarly to earlier IRIX 5 and 6 versions.

There is now a systune command. This command allows you to configure other networking variables. systune keeps strack of the chances you make in a file called stune so that you can see them all in one place. Also note that changes made using systune are permanent. Here is a sample of things which can be tuned using systune:

/usr/sbin/systune (which is like sysctl for BSD) is what you use for
tuneable values.

 group: net_stp (statically changeable)
        stp_ttl = 60 (0x3c)
        stp_ipsupport = 0 (0x0)
        stp_oldapi = 0 (0x0)

 group: net_udp (dynamically changeable)
        soreceive_alt = 1 (0x1)
        arpreq_alias = 0 (0x0)
        udp_recvgrams = 2 (0x2)
        udp_sendspace = 61440 (0xf000)
        udp_ttl = 60 (0x3c)

 group: net_tcp (dynamically changeable)
        tcp_gofast = 0 (0x0)
        tcp_recvspace = 61440 (0xf000)
        tcp_sendspace = 61440 (0xf000)
        tcprexmtthresh = 3 (0x3)
        tcp_2msl = 60 (0x3c)
        tcp_mtudisc = 1 (0x1)
        tcp_maxpersistidle = 7200 (0x1c20)
        tcp_keepintvl = 75 (0x4b)
        tcp_keepidle = 7200 (0x1c20)
        tcp_ttl = 60 (0x3c)

 group: net_rsvp (statically changeable)
        ps_num_batch_pkts = 0 (0x0)
        ps_rsvp_bandwidth = 50 (0x32)
        ps_enabled = 1 (0x1)

 group: net_mbuf (statically changeable)
        mbretain = 20 (0x14)
        mbmaxpages = 16383 (0x3fff)

 group: net_ip (dynamically changeable)
        tcpiss_md5 = 0 (0x0)
        subnetsarelocal = 1 (0x1)
        allow_brdaddr_srcaddr = 0 (0x0)
        ipdirected_broadcast = 0 (0x0)
        ipsendredirects = 1 (0x1)
        ipforwarding = 1 (0x1)
        ipfilterd_inactive_behavior = 1 (0x1)
        icmp_dropredirects = 0 (0x0)

 group: network (statically changeable)
        netthread_float = 0 (0x0)

 group: inpcb (statically changeable)
        udp_hashtablesz = 2048 (0x800)
        tcp_hashtablesz = 8184 (0x1ff8)

Changes made using systune may or may not require a reboot. This can be easily determined by looking at the 'group' heading for each section of tunables. If the group heading says dynamic, changes can be made on the fly. Group headings labelled static require a reboot.

Finally, the tcp_sendspace and tcp_recvspace can be tuned on a per-interface basis using the rspace and sspace options to ifconfig.

SACK: As of 6.5.7, SACK is included in the IRIX operating system and is on by default.

Procedure for raising network limits under SunOS 4.1.4

Procedure for raising network limits under Solaris 2.5

The ndd variable tcp_max_buf specifies the maximum socket buffer size.

To change them, use:

	ndd -set /dev/tcp tcp_max_buf xxx
	ndd -set /dev/tcp tcp_xmit_hiwat xxx
	ndd -set /dev/tcp tcp_recv_hiwat xxx

(Note: I believe xxx should be specified in bytes)

The ndd variable ip_path_mtu_discovery controls the use of path MTU discovery. The default value is 1, which means on.

Note that ndd can also be used to increase the volume of TCP connections available to a machine.

	ndd -set /dev/tcp tcp_conn_req_max 

  tcp_conn_req_max_q 
    default value = 128
    number of connections in ESTABLISHED state
    (3-way handshake completed; not yet accepted)

  tcp_conn_req_max_q0 
    default value =  1024
    number of connections in SYN_RCVD state