Moving Nexenta from HDD to USB stick

I wanted this because the 7200 RPM disk that shipped with HP Microserver makes an annoying metallic noise when spinning. USB key is quieter, greener and frees an additional disk bay. Also, Microserver has an extra USB slot inside the chasis just for this purpose.

However, the operation was not as simple as I thought1 — because you can’t shrink zpools (that is, shrink from the 250GB internal drive to the 16GB USB drive).

So I had two options: reinstall from scratch and choose USB stick as the target, or move the existing system. I chose the latter to avoid the reconfiguration. However, if you haven’t installed it yet and are thinking about this as a future upgrade path, I recommend you skip the HDD entirely and go directly to USB stick; it will save you a lot of trouble.

In the end, this process worked:

  • Install Nexenta (fresh install) on USB key (I did this using VMWare and USB passthrough). This creates suitable partition table, installs grub and saves many other steps as well.
  • Boot the old system (disable usb boot) and plug in the USB stick.
  • Import new zpool on the usb stick as newsyspool (to distinguish from old syspool).
  • Delete all filesystems in newsyspool using zfs destroy newsyspool/dump (also: newsyspool/swap, newsyspool/nmu… everything); this deletes all data on the USB drive — we just need partitions (i.e., ZFS slices).
  • Make a recursive snapshot of the old (internal HDD) syspool: zfs snapshot -r syspool@20111112
  • Copy entire internal HDD (i.e., syspool) to the USB stick: zfs send -R syspool@20111112 | zfs recv -vFd newsyspool
  • Set boot property of the USB drive in the same way than that of your internal HDD. Get the latter with zpool get bootfs syspool (this shows old bootfs property, e.g., syspool/rootfs-nmu-000). Then set the former (newsyspool on USB drive) with zpool set bootfs=newsyspool/rootfs-nmu-000 newsyspool
  • Set noatime=off to disable writing a timestamp on every read: zfs set atime=off newsyspool
  • Disconnect your internal HDD, set bios to boot from USB drive and reboot.
  • If the system doesn’t boot, just reconnect the old drive and you haven’t lost anything.
  • If the system boots, delete the old syspool which is now faulted (since there’s no HDD): zpool destroy syspool (warning, this asks for no confirmation! be sure you’ve booted from USB!)

I got most of it from these guides:

 

Update 2011-12-20:

About 1 month after the upgrade the USB key failed; the server wouldn’t boot (stuck at grub loading stage2), so I plugged back the old HDD, scrubbed USB key zpool and it found approx. 1000 errors, of which 500 were unrecoverable. It was not the cheapest USB key (PQI), but the constant swapping or whatever Nexenta’s doing when nobody’s looking must have killed it.

Update 2012-05-15

Fixed some errors pointed out in the comment.

  1. What I thought was this: plug in the USB stick, create a mirrored syspool and remove the HDD [↩]