Moving Nexenta from HDD to USB stick
I wanted this because the 7200 RPM disk that shipped with HP Microserver makes an annoying metallic noise when spinning. USB key is quieter, greener and frees an additional disk bay. Also, Microserver has an extra USB slot inside the chasis just for this purpose.
However, the operation was not as simple as I thought1 — because you can’t shrink zpools (that is, shrink from the 250GB internal drive to the 16GB USB drive).
So I had two options: reinstall from scratch and choose USB stick as the target, or move the existing system. I chose the latter to avoid the reconfiguration. However, if you haven’t installed it yet and are thinking about this as a future upgrade path, I recommend you skip the HDD entirely and go directly to USB stick; it will save you a lot of trouble.
In the end, this process worked:
- Install Nexenta (fresh install) on USB key (I did this using VMWare and USB passthrough). This creates suitable partition table, installs grub and saves many other steps as well.
- Boot the old system (disable usb boot) and plug in the USB stick.
- Import new zpool on the usb stick as newsyspool (to distinguish from old syspool).
- Delete all filesystems in newsyspool using zfs destroy newsyspool/dump (also: newsyspool/swap, newsyspool/nmu… everything); this deletes all data on the USB drive — we just need partitions (i.e., ZFS slices).
- Make a recursive snapshot of the old (internal HDD) syspool: zfs snapshot -r syspool@20111112
- Copy entire internal HDD (i.e., syspool) to the USB stick: zfs send -R syspool@20111112 | zfs recv -vFd newsyspool
- Set boot property of the USB drive in the same way than that of your internal HDD. Get the latter with zpool get bootfs syspool (this shows old bootfs property, e.g., syspool/rootfs-nmu-000). Then set the former (newsyspool on USB drive) with zpool set bootfs=newsyspool/rootfs-nmu-000 newsyspool
- Set noatime=off to disable writing a timestamp on every read: zfs set atime=off newsyspool
- Disconnect your internal HDD, set bios to boot from USB drive and reboot.
- If the system doesn’t boot, just reconnect the old drive and you haven’t lost anything.
- If the system boots, delete the old syspool which is now faulted (since there’s no HDD): zpool destroy syspool (warning, this asks for no confirmation! be sure you’ve booted from USB!)
I got most of it from these guides:
- http://waddles.org/content/replicating-zfs-root-disks#Create_recursive_snapshot
- http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express
- http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/28298
Update 2011-12-20:
About 1 month after the upgrade the USB key failed; the server wouldn’t boot (stuck at grub loading stage2), so I plugged back the old HDD, scrubbed USB key zpool and it found approx. 1000 errors, of which 500 were unrecoverable. It was not the cheapest USB key (PQI), but the constant swapping or whatever Nexenta’s doing when nobody’s looking must have killed it.
Update 2012-05-15
Fixed some errors pointed out in the comment.
- What I thought was this: plug in the USB stick, create a mirrored syspool and remove the HDD [↩]
araj, 
Damian Wojsław, and 1 other are discussing.
Roland Hordos 20:30 on 15 May. 2012 Permalink
Thank you! This worked for me with a couple of tweaks, then some additional (resolved) drama.
NexentaStor 3.1.2 – SunOS 5.11 NexentaOS_134f 64bit
Tweak 1 – The snapshot also needs to be recursive or the send/recv will miss most everything# zfs snapshot -r syspool@20100514-2Tweak 2 – The atime statement for me needed to be:# zfs set atime=off newsyspool I subsequently decided to revert the atime back to “on” which is what the original syspool has, just in case there’s some algorithm foiled by this and inadvertenly causing _more_ writes (perhaps causing your early device failure?)Problem with Licensing – the mucking with the syspool causes the licensing system to get a headache on reboot and wants you to reenter your registration key. You do but License bully doesn’t like that you’ve changed the syspool name and errors, forcing you to take a bash shell or return to the licensing screen.Solution:* Booted into recovery console* Imported newsyspool as syspool (needs -f)* Rebooted into appliance, licensing bully faked outI imagine a cleaner way to do this but exporting the in use newsyspool first did not seem viable. Cleanup: so there’s also still a “newsyspool” because it was never exported and now the metadata is a mess from what I did in recovery console with -f. Newsyspool is ONLINE and healthy whereas “syspool” is degraded. Despite degraded it is actually in use as running df in bash shows a syspool originating root mount. So ..* Exported newsyspool so it’s no longer claiming the same device* Cleared errors on syspool as it has it’s device back* Reboot auto repaired a minor checksum error (declared as such)
Thank you, this freed the initial 1TB drive I used for my syspool on install. I then went on to mirror the syspool to further protect my blazing fast NexentaStor !
Urban 21:48 on 15 May. 2012 Permalink
Thanks for the comment, your tweaks are actually the right way to do it.. I’ve updated the post to fix the errors 🙂
Roland Hordos 16:33 on 16 May. 2012 Permalink
Cool! Sorry for the de-formatted mess I made in your post <>
Damian Wojsław 09:30 on 15 Jun. 2012 Permalink
I think there is lots of logging going on during normal operation, so you’d get lots of write cycles. If your appliance is swapping, I’d say that you have too little RAM for normal operation. 🙂
araj 17:21 on 13 Dec. 2012 Permalink
were you able to exactly figure out what went wrong while using the USB as boot drive. I am trying to setup a nexentastor based NAS with boot from USB 2x16GB-mirrored, and 8GB RAM
Any input is much appreciated.The area I am most uncertain about is on safely using the USB flash drive as boot disk.
Thanks
Urban 01:55 on 14 Dec. 2012 Permalink
Unfortunately, no.. I didn’t have the slightest idea how to approach such post-mortem diagnostics. USB flash drives use wear leveling, which makes the entire drive wear out equally (below the level of FS). This makes it next to impossible to isolate the reason after the fact..
In hindsight, my best guess is swap (as another commenter noted), which should’ve been disabled (duh). I guess I didn’t think swap would get used that much, as the machine had 8 GB of RAM (which should’ve been plenty).
Also note that I used Nexenta Core, which is basically “an ordinary linux distro with solaris kernel”. I would think this is not optimized in any way, out-of-the-box. On the other hand, NexentaStor is a ready-made appliance with all kinds of things preconfigured, so it’s very likely a better starting point and might already be tweaked to play better with flash storage.
All that said, any flash drive you use will still have a limited number of writes and it will most definitely not be sustainable in the long run if there’s stuff being written to it. If you want a bulletproof solution, you need to mount it in readonly mode and move all writable config files, temp dirs and logs to another drive (you can replace originals with symlinks to the new location). This could very easily prove to be more trouble than is’t worth..
Alternatively, you could use a NAS distro that’s already optimized to run in read-only mode from a flash drive, such as FreeNAS.
Or just use two HDDs (maybe internal + usb). Or even only 1 HDD with copies=2 to protect against soft errors.. depends on what you want to optimize for.
araj 15:24 on 14 Dec. 2012 Permalink
Thanks a lot for the reply and the suggestions will definitely help me.