Automatic TCP Send and Receive Socket Buffer Sizing
Some time ago i have found very interesting patch for FreeBSD kernel. Patch was originally posted by Andre Oppermann in freebsd-current maillist. That patch allows dynamically change TCP send and receive buffer sizes. However Andre posed version only for FreeBSD CURRENT, so i have make note to try patch as soon as i have chance to install CURRENT. But today, my colleague told me that he found version for 6.x, and i have spent last few hours for patching & testing. As i wrote before, patch allows dynamically change TCP buffer size. Using that patch you can save a lot of kernel memory, because you don’t need to set big values for socket buffers. TCP session will start with a small buffer size and grow to match network conditions. Andre showed about 1000% better performance in his tests.
Below you can find my test results.
There is two versions of patch available for download:
Test system was a Dell PowerEdge 850 with FreeBSD amd64 RELENG_6_2.
I have create 100mb test file, and download it from 2 locations. Here is results before changes:
Location A: 10:41:21 (353.39 KB/s) - `100mb.bin’ saved [104857600/104857600]
Location B: 07:41:22 (197.88 KB/s) - `100mb.bin’ saved [104857600/104857600]
Then i have download/apply patch/rebuild kernel. There is few new sysctl values added with this patch:
- net.inet.tcp.sendbuf_auto=1 (enabled by default)
- net.inet.tcp.sendbuf_inc=8192 (8K, step size)
- net.inet.tcp.sendbuf_max=262144 (256K, growth limit)
- net.inet.tcp.recvbuf_auto=1 (enabled by default)
- net.inet.tcp.recvbuf_inc=16384 (16K, step size)
- net.inet.tcp.recvbuf_max=262144 (256K, growth limit)
I have made 2 tests, one with defaults, and one with some changes. However there is no big difference between tests, because a lot of things are depended on receiver side. Here is tests results with automatic buffers enabled:
Location A : 12:11:32 (694.67 KB/s) - `100mb.bin.1′ saved [104857600/104857600]
Location B: 09:10:37 (386.47 KB/s) - `100mb.bin.1′ saved [104857600/104857600]
As we can see there is no 1000% growth, but double growth is also _very good_. I will look how this work in next few weeks, and if everything goes OK i will try patch some high-loaded systems, to see how patch saves kernel memory.