Speeding up slow zfs resilver on FreeNas

A couple of months ago I began receiving constant e-mail alerts stating that my FreeNas box was 80% full. My 2-year-old setup had 4 2TB Seagate drives in a Raidz1 pool. After some research and test with new firmware builds I found out that this was not optimal since the Raidz1 should follow the 2*n+1 formula [with n>0] (3, 5, 7 or 9 … drives).

I cannot afford to rebuild the pool at this time and one of the original drives failed and was replaced by a newer 4TB unit.

My approach was to replace every remaining 2TB drive on the pool by a 4TB one and this proved to be very time consuming. My box was taking too long to resilver the pool.

After some more research I came across Allan Jude’s “ZFS Advanced Topics” chapter proposed to the FreeBSD documentation project.

sudo sysctl vfs.zfs.resilver_delay=0

sudo sysctl vfs.zfs.scrub_delay=0

These tunables reduce the wait time between each resilver and scrub IO operation. Client performance was somewhat degraded but getting my pool back into pristine condition was more important.

Improving network performance of a new FreeBSD server

Thanks to Calomel.org for these tips. I was having network performance issues and my throughput more than doubled now!

Editing /etc/sysctl.conf:

# Default is fine for most networks. You may want to increase to 4MB if the
# upload bandwidth is greater the 30Mbit. For 10GE hosts set to at least 16MB
# as well as to increase the TCP window size to 65535 and window scale to 9.
# For 10GE hosts with RTT over 100ms you will need to set a buffer of 150MB and
# a wscale of 12.  Default of "2097152 = 2*1024*1024" is fine for 1Gbit, FIOS
# or slower.
# network:   1 Gbit   maxsockbuf:    2MB   wsize:  6    2^6*65KB =    4MB (default)
# network:   1 Gbit   maxsockbuf:    4MB   wsize:  7    2^7*65KB =    8MB (FIOS 150/65)
# network:  10 Gbit   maxsockbuf:   16MB   wsize:  9    2^9*65KB =   32MB
# network:  40 Gbit   maxsockbuf:  150MB   wsize: 12   2^12*65KB =  260MB
# network: 100 Gbit   maxsockbuf:  600MB   wsize: 14   2^14*65KB = 1064MB
kern.ipc.maxsockbuf=4194304  # (default 2097152)

# set auto tuning maximums to the same value as the kern.ipc.maxsockbuf above.
# Use at least 16MB for 10GE hosts with RTT of less then 100ms. For 10GE hosts
# with RTT of greater then 100ms set buf_max to 150MB. The default of
# "2097152" is fine for most networks.  
net.inet.tcp.sendbuf_max=4194304  # (default 2097152)
net.inet.tcp.recvbuf_max=4194304  # (default 2097152)