ZFS snapshotting done better

zfs, say cheese!

Problem statement

I’ve been using rather simple scripts to periodically snapshot my ZFS storages since forever.

And they kind of work well, if you want to snapshot nearly everything in the tank.

Recently I started using zfs_autobackup to shuffle backups around, and it’s breaking the snapshotting due to a race condition.

This is a quick note on how I changed my setup, and what’s oh so much better about it.

Background

The zfs_autobackup script works in a rather predictable way – takes a snapshot, then invokes (roughly) zfs send | ssh remote zfs recv¹.

You can also split the functionality. In other words, you can run it in any of these configurations:

• snapshot, don’t backup remotely
• backup, don’t snapshot
• snapshot, then backup

It’s actually a rather lovely script², with some of its concepts truly worth stealing. :-D

But, since it’s flexible enough, I can avoid reinventing the wheel, and just use it.

Solution

The trouble I’ve run into is that I’m snapshotting and syncing data from one host, and then sending it over to another host, which does similar snapshotting.

My way was (copied from the previous article):

#!/bin/sh
if [ $# -ne 1 ]; then
  echo "Usage: $0 <name>"
  exit 1
fi
name="$1"
now=$(date +%s)
code=0
for pool in $(zpool list -Ho name); do
  zfs snapshot -r ${pool}@${name}_${now} || code=1
  # special case -- murder image snapshots
  zfs list -t snapshot ${pool}/containers/.images@${name}_${now} >/dev/null 2>&1 &&
    zfs destroy -r ${pool}/containers/.images@${name}_${now}
done
exit $code

In other words, snapshot all (using -r), then delete a subset of them.

If the incoming backup snapshot from zfs_autobackup interleaves just the right way, zfs recv fails (because the new snapshot makes the storage incompatible)³.

So, a better approach is needed.

Turns out, zfs_autobackup implements exactly that better approach. Watch:

$  zfs-autobackup -v --allow-empty --no-send --dry-run --debug snaps
  zfs-autobackup v3.3 - (c)2022 E.H.Eefting (edwin@datux.nl)
[...]
  #### Snapshotting
# [Source] nvmetank: Dataset should exist
# [Source] nvmetank: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name nvmetank)
# [Source] nvmetank/ROOT: Dataset should exist
# [Source] nvmetank/ROOT: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name nvmetank/ROOT)
[...]
  [Source] Creating snapshots snaps-20260523164014 in pool nvmetank
# [Source] CMDSKIP> (zfs snapshot nvmetank@snaps-20260523164014 nvmetank/ROOT@snaps-20260523164014 nvmetank/ROOT/alpine@snaps-20260523164014)
  [Source] Creating snapshots snaps-20260523164014 in pool ssdtank
# [Source] CMDSKIP> (zfs snapshot ssdtank@snaps-20260523164014 ssdtank/containers@snaps-20260523164014 [...])
[...]

This truly is worth emulating…

Because you select the stuff you (don’t) want acted on using user properties on the individual datasets (which get auto-inherited by zfs downstream):

zfs set autobackup:snaps=true nvmetank
zfs set autobackup:snaps=true ssdtank
zfs set autobackup:snaps=false ssdtank/containers/.images
zfs set autobackup:snaps=false ssdtank/backups

and good guy zfs_autobackup does the snapshot using an explicit list of datasets, no “snap all, purge later”.

So the setup above snapshots all of nvmetank, and all of ssdtank except what is under ssdtank/containers/.images and ssdtank/backups.

Neat!

In other words, I have shrunk my snapshotting (and more importantly, snapshot purging) script to a rather simple⁴:

#!/bin/sh
exec >> /var/log/zfs-snapshots.log 2>&1
 
code=0
 
zfs-autobackup \
  --allow-empty \
  --keep-source=25,1h25h,1d8d,1w5w,1m13m,1y10y \
  --no-send \
  snaps 
 
code=$?
 
curl -fsS -m 10 --retry 5 -o /dev/null \
  https://hc-ping.com/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/$code
 
exit $code

Obviously this comes with the knowledge that there’s some 4.3k lines of Python powering this bad boy. At 87% test coverage? I’ll take that. ;)

And no, I don’t think that cobbling together something like:

zfs snapshot $(zfs get -t volume,filesystem -o name,value,source \
    -H autobackup:snaps | \
  awk "\$2 ~ /true/ { printf \"%s@snap-$(date +%s)\n\", \$1 }")`

would have been particularly hard⁵. But then you’re left writing the purging logic, etc. And that doesn’t make sense, does it?

Closing words

I like simple solutions. Like replacing a few lines of bash with a few thousand lines of Python. ;)

In this case, I’d argue it’s the right tradeoff.