Speeding up slow zfs resilver on FreeNas

A couple of months ago I began receiving constant e-mail alerts stating that my FreeNas box was 80% full. My 2-year-old setup had 4 2TB Seagate drives in a Raidz1 pool. After some research and test with new firmware builds I found out that this was not optimal since the Raidz1 should follow the 2*n+1 formula [with n>0] (3, 5, 7 or 9 … drives).

I cannot afford to rebuild the pool at this time and one of the original drives failed and was replaced by a newer 4TB unit.

My approach was to replace every remaining 2TB drive on the pool by a 4TB one and this proved to be very time consuming. My box was taking too long to resilver the pool.

After some more research I came across Allan Jude’s “ZFS Advanced Topics” chapter proposed to the FreeBSD documentation project.

sudo sysctl vfs.zfs.resilver_delay=0

sudo sysctl vfs.zfs.scrub_delay=0

These tunables reduce the wait time between each resilver and scrub IO operation. Client performance was somewhat degraded but getting my pool back into pristine condition was more important.

Automatically shutdown FreeNAS box when all clients are offline

I’ve been struggling with trying to keep the power consumption of my FreeNAS box to a minimum for quite a while now.

On the past months I also started to play with the Raspberry Pi, running it as a media center device primarely using OpenElec at first and then Raspbmc later on.

I intend to talk about those experiences on a separate posting. At first what concerned me is that for my media center to work I would need to keep my file server running all the time, something that I would not like to do.

My solution comes in two parts:

First to build a device to monitor my lan, pinging known addresses (statically issued by my local DHCP server) and sending a Wake-On-Lan packet when one of them come online.

Second, to keep monitoring my lan, checking every minute if such devices are still present, and if they are not, shut down my file server after 30 minutes.

Now imagine the following scenario: I arrive at home with my phone and laptop. My phone upon seeing my home wifi network will automatically connect. The first raspberry pi will be pinging every minute a list of statically defined addresses looking for a phone or laptop, mine or my wife’s.

Once it receives a response, it sends a WOL packet that will turn my file server on. Now that it is on, the first raspberry pi will only keep pinging the file server, as long as it is on, there is no need for further checks.

On the file server, it would have a script running, pinging another list of addresses. As long as one of them answers, it will do nothing. When all addresses on the list fail to answer on the past 30 minutes it will initiate a shutdown.

And the cycle repeats.

When there is someone at home, the file server will be on, when everyone leaves, it turn itself gracefully off.

Perfect!

Since I have only ONE raspberry pi to do this, I will be turning my file server manually on for the time being and focusing this post on the second part.

This is the script I am using on my FreeNAS box:

#!/bin/sh

CHECK_EVERY=60
MAX_FAIL_COUNT=30

keep_on() {
  for p in htpc.home raphael-pc.home sala-tv.home teste.home;
  do
    if ping -c 1 $p >/dev/null 2>&1; then
      return 0
    fi
  done
  return 1
}

# Client must be up before starting main loop
while sleep 5
do
  if keep_on; then
    break
  fi
done

FAIL_COUNT=0

# main script
while sleep ${CHECK_EVERY}
do
  if keep_on; then
    FAIL_COUNT=0
  else
    FAIL_COUNT=$((FAIL_COUNT+1))
    echo $FAIL_COUNT
  fi
  if [ $FAIL_COUNT == $MAX_FAIL_COUNT ]; then
    shutdown -p now
    exit
  fi
done 2>&1

Not much complex stuff. The variable CHECK_EVERY state that the checks should be every 60 seconds and MAX_FAIL_COUNT that after 30 fail attempts it will shut itself down.

There is one failsafe: The script will only act when it receives an answer from at least one device on the list. This is to prevent the box to be turning off if something goes wrong with my internal DNS or if I plug it on someone else’s network. You never know…

To allow this to persist between boots, I first made the root writable with

su
mount -uw /

Then, I saved this script on /conf/base/etc/autoshutdown.sh and added a line calling it on /conf/base/etc/rc.local:

#!/bin/sh

/conf/base/etc/autoshutdown.sh

Also, made both scripts executable.

And that is it!

When I get a hold of a second raspberry pi I’ll post the other scripts here as well.

Letting your pool sleep…

Some very good points on an article I just stumbled upon…

Mount all your filesystems/pools with noatime

This way you won’t generate writes every time a file is accessed. I had this suggested by an episode of TechSnap where one of the hosts mentioned that they do this to avoid writes while doing reads but never came back to actually implement it.

I don’t have other filesystems on my FreeNAS box and ZFS has a property for this. Just run:

zfs set atime=off POOLNAME

Find files modified in the last day or so

A good snippet to try to get to these files is:

find / -mtime -1

Relocate directories and files to non-rotating media

Also another great suggestion on the original article:

Get a cheap USB drive (does not need to be big) and format it as ext4 (technically, you could set up another ZFS pool there too). Then, set it to be mounted in `/var/volatile` on your fstab. You can now move directories that contain frequently modified files there. After you’re done moving those directories, you can symlink them from their original location. So, for example, you would move `/var/log` to `/var/volatile/log`, then creating a symbolic link to `/var/volatile/log` named `/var/log`. At this point, it would be wise to make a cron job to nightly back the contents of this USB drive up (think `rsync -a`) to a backups directory somewhere in your pool. OK. If you’ve moved the most frequently modified files to `/var/volatile`, your disks will be idle unless you are actually using your file server. Now it’s time to take advantage of that idleness.

“Warning: /var/lib/mlocate/daily.lock present”

I am still fixing small issues as they appear in my home setup. Right now I have a file server running FreeNAS named capella.home and a virtualization box running Proxmox named andromeda.home.

I configured andromeda to map a NFS share from capella as a repository for images and templates and for backups to be saved as well. Everyday andromeda performs full backups of all my VMs but to conserve power and to preserve the hardware I turn capella off whenever I’m travelling and every single time I was getting multiple warning e-mails with the message:

/etc/cron.daily/mlocate:
Warning: /var/lib/mlocate/daily.lock present, not running updatedb.
run-parts: /etc/cron.daily/mlocate exited with return code 1

This message was being sent both from andromeda and multiple VMs hosted inside it.

At first I thought that this was somehow due to auto-upgrade issues, even scheduled my VM host to auto reboot every couple days to see if it would avoid it (terribly bad practice, I know) with no success.

It turns out this was happening because the NFS server was offline and mlocate was trying to index it, so I adapted my puppet base recipe to include the following:

#
# locate, mlocate and updatedb
#

# Limit where updatedb scans
file {'/etc/updatedb.conf':
  ensure  => 'present',
  owner   => 'root',
  group   => 'root',
  source  => 'puppet:///etc/base/updatedb.conf'
}

and borrowed the contents of updatedb.conf from :

PRUNE_BIND_MOUNTS="yes"
PRUNENAMES=".git .bzr .hg .svn"
PRUNEPATHS="/tmp /var/spool /media"
PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf fuse.glusterfs fuse.sshfs ecryptfs fusesmb devtmpfs"

And that is it!

Struggling with FreeNAS power consumption optmization

DisksView

I’ve been using FreeNAS 8.3.0 for several months now and it became such a work/entertainment hub that I simply started take it for granted. Yesterday I needed to create a new dataset and just out of curiosity I looked at the Reporting tab. My box has been running for over a month now, with all four disks spinning the entire time.

Ok, spinning disks evolved a long way but this is just a waste of energy. If it is in the middle of the night or during the day when no one is at home it should be in stand-by mode. I don’t want to manually turn it on and off because I will forget it and I don’t want to schedule power on and add a cron job to shut it down because I might be working late some day or just finishing a extra long movie and have to wait until it boots up again.Movie Fifty Shades Darker (2017)

It has to be automatic and seamless

Thankfully FreeNAS offers settings on the web ui to manage this. After a lot of experimentation and frustation I figured it out:

First of all, these settings seem to come into effect only after a reboot. A little note somewhere would have been helpful.

Second, on the View Disks page, the HDD Standby and the Advanced Power Management settings seem to be mutually exclusive. The first one just a “dumb” timer (that I don’t know if is in seconds of minutes, it’s behaviour is erratic) and the second one is a setting that is applied by the HD firmware. APM values lower than 128 allow the drive to spin down when idle.

After reading How to find out if a drive is spinning down properly and HDD standby times not as expected on the FreeNAS forum I did a couple more experiments.

All my drives have been set with HDD Standby = Always On and Advanced Power Management = 64.

Reboot and a improved check-spinning.sh:

#!/bin/sh

while [ 1 ]
do
    echo -n `date`    
    camcontrol devlist | awk -F\( '{print $2'} | awk -F\, '{print $1}' |while read LINE
    do
        CM=$(camcontrol cmd $LINE -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | awk '{print $10}')
        if [ "$CM" = "FF" ] ; then
            echo -n " SPINNING  "
        elif [ "$CM" = "00" ] ; then
            echo -n "   IDLE    "
        else 
            echo -n "  UNKNOWN  "
        fi
    done    
    echo
    sleep 5
done

I let it run for a few hours and the output was something like:

Sat Mar 16 21:41:23 BRT 2013   IDLE       IDLE       IDLE       IDLE    
Sat Mar 16 21:41:28 BRT 2013   IDLE       IDLE       IDLE       IDLE    
Sat Mar 16 21:41:33 BRT 2013   IDLE       IDLE       IDLE       IDLE    
Sat Mar 16 21:41:38 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:41:48 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:41:53 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:41:59 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:04 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:10 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:15 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:20 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:26 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:31 BRT 2013 SPINNING   SPINNING   SPINNING   SPINNING  
Sat Mar 16 21:42:36 BRT 2013 SPINNING     IDLE       IDLE     SPINNING  
Sat Mar 16 21:42:41 BRT 2013 SPINNING     IDLE       IDLE     SPINNING

The second script I wrote was to log all processes touching the disk at the time that they were being used.Watch Full Movie Online Streaming Online and Download

Following some suggestions from StackExchange, check-top.sh:

#!/bin/sh

while [ 1 ]
do
    echo -n `date`
    top -I -mio -b -n -a 2
    sleep 1
done

With the long output of this one I cross-referenced all times and found out that the culprit were three python scripts scheduled to run every minute: /usr/local/bin/graph.py, /usr/local/www/freenasUI/tools/alert.py and /usr/local/www/freenasUI/tools/autosnap.py.

They are responsible for updating the reports on the gui, for updating alerts and for generating automatic snapshots and replicating them to offsite FreeNAS boxes.

Don’t get me wrong, I use and love these three features, but I don’t need to have these scripts running every minute. Basically, if I am not connected to my NAS reading or writing files to it, I don’t need automatic snapshots or replication (nothing is changing right?). If there is no one connected, there is also no need to updating reports that no one is seeing.

How did I fixed it?

I wrote a bash script to check the state of my disks before invoking the passed command. Here is /conf/base/etc/runifspinning.sh:

#!/bin/bash

while read LINE; do
    CM=$(camcontrol cmd $LINE -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | awk '{print $10}')
    if [ "$CM" = "00" ] ; then
        exit 0
    fi
done < <(camcontrol devlist | awk -F\( '{print $2'} | awk -F\, '{print $1}')

$*

Then, I changed the base crontab at /conf/base/etc/crontab from:

*/5 *   *   *   *   root    /usr/local/bin/python /usr/local/bin/graph.py
*/5 *   *   *   *   root    /usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1
*   *   *   *   *   root    /usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1

To:

*/5 *   *   *   *   root    /conf/base/etc/runifspinning.sh /usr/local/bin/python /usr/local/bin/graph.py
*/5 *   *   *   *   root    /conf/base/etc/runifspinning.sh /usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1
*   *   *   *   *   root    /conf/base/etc/runifspinning.sh /usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1

Rebooted and bingo! Now, I still have all the functionality but only when the disks are already spinning. No more waking from idle to perform repetitive tasks.

A few last notes

  • To change the base configuration you will have to ssh into your box and be able to run commands as root.
  • All these changes will probably be lost when upgrading since they are not persisted on the config database.
  • To make /conf writable you can use mount -uw /.

Enabling sudo and persisting the sudoer’s list on FreeNAS

Fiddling with FreeNAS for a while now it was bothering me to have to su before running a command that requires root.

This sound simple but there is one caveat: If you just edit /usr/local/etc/sudoers, your changes will be lost when your box reboots.

FreeNAS stores some of its base files in /conf/base. To modify it you have to make it writable first, ssh into your box and run:

su
mount -uw /

Now you can nano /conf/base/etc/local/sudoers and add the following line:

%wheel ALL=(ALL) ALL

Make sure that the users that require sudo will be on the wheel group (you can change this from the web gui).

Reboot your FreeNAS and the system will apply this setting. That’s it!

More info on DistroGeeks, Karl Keppner and Sudo.ws.

Update 2013-10-26: Trying to apply this to FreeNAS 9.1.1 I noticed that the correct path to the persistent sudoers file is /conf/base/etc/local/sudoers.