Quick answer: If a ZFS pool will not import, the first rule is do not damage the disks. Take a read only sector image of every member first (ddrescue), then attempt recovery on the image copies. ZFS binds every write to a transaction group (txg) and keeps a 128 entry uberblock ring in each vdev label, so even if the last few txgs are corrupt you can roll back to an earlier consistent point with zpool import -F or -T <txg>. If the pool still will not come up, you inspect it deeply with zpool import -o readonly=on and zdb. A RAIDZ array is genuinely lost when the number of missing disks exceeds the parity level (two for RAIDZ1, three for RAIDZ2) or a wrong resilver has overwritten the recoverable state.

ZFS is one of the strongest file systems for integrity. Its copy on write design, end to end checksums and transaction group model normally keep it consistent even after a crash. But those same properties make recovery different from classic file systems when a pool really does break. This guide explains, at an expert level, why a ZFS pool on a TrueNAS, Proxmox or Linux server fails to import, which internal structures make recovery possible, and which actions make the situation irreversibly worse.

Stop first: every action can change the pool

The most expensive mistakes in ZFS recovery happen in the first hours. zpool import -f, zpool clear, disk replacement and resilver all write to the pool. Writing to a broken pool can overwrite an older recoverable txg state with new metadata and close the rollback window.

The correct order is this. Mount all pool members read only, or physically remove them to a recovery bench. Take a full sector image of each disk; failing or clicking disks need a hardware imager, following the method in our imaging a failing disk guide and our raw sector copy article. All recovery attempts run on those image copies while the originals stay in the safe. This matters both for a possible chain of custody and so you can start over after a failed attempt.

How ZFS sits on disk

To understand recovery decisions you need four structures.

Vdev labels. Each physical device carries four label copies, two at the start and two at the end. A label holds the pool name, GUID values, the vdev tree and the uberblock ring. Four copies mean the configuration can still be read even if a disk's head or tail region is damaged.

The uberblock ring. Each label stores 128 uberblocks in a circular buffer. An uberblock holds the current root block pointer of the pool (the MOS, the meta object set root) and the transaction group number (txg). When each txg completes, ZFS writes a new uberblock into the next slot. That leaves a history of up to the last 128 consistent states on disk.

Transaction groups (txg). ZFS commits writes not one by one but as groups that accumulate over a few seconds, atomically. A txg is either written fully or not at all. This model prevents landing in a half written state on power loss and provides natural points to roll back to during recovery.

The copy on write tree. No block is modified in place. Every changed block is written to a new location and the pointers above it are updated upward to new blocks. That is why old metadata often lingers on disk for a while and rollback works.

Why the pool will not import: a symptom map

Symptom Likely cause First approach
cannot import: no such pool labels unreadable, disk order or path changed scan with zpool import -d /dev/by-id, serve images by id
one or more devices is currently unavailable member disk missing or failed serve the missing disk from an image, try a degraded import if RAIDZ
pool was previously in use from another system different hostid, no clean export try read only before zpool import -f
import hangs or SUSPENDED last txg corrupt, metadata error roll back the last txg with zpool import -F
insufficient replicas lost disks exceed parity level image every member, recover weak disks and retry
I/O error or panic after mount deep metadata corruption import with readonly=on, analyze with zdb

txg rollback: the gradual, safe path

If the last few transaction groups are corrupt (typically after sudden power loss, a controller fault or a half write) the solution is to roll back to the last consistent txg. Proceed in steps.

First, a harmless attempt: a read only import. If this makes the pool and data visible without writing anything, you can immediately copy the data to another target.

zpool import -o readonly=on -d /dev/by-id <pool>

If that fails, the next step is an import in recovery mode. -F asks ZFS to discard the last few txg values and come up with the most recent consistent uberblock it can find. Use -n for a dry run first to see how much data would be lost.

zpool import -F -n <pool>      # dry run, reports only
zpool import -F <pool>          # actual rollback

If you need to be more aggressive, -X enables an extreme rewind (more txgs may be discarded), and if you know a specific target txg you go there with -T <txg>. You see which txg values are on disk with zdb.

zdb -l /dev/by-id/<member-disk>     # vdev labels and uberblock txg range
zdb -ul /dev/by-id/<member-disk>    # all entries in the uberblock ring and their txg
zpool import -FX -T <txg> <pool>

Important warning: -F, -X and -T write to the pool. So run these commands only on image copies, with the original disks safe.

RAIDZ logic and rebuilding

RAIDZ uses the same parity idea as hardware RAID 5 and 6, but implemented in software inside ZFS, with variable width stripes and a per block checksum. RAIDZ1 survives one disk, RAIDZ2 two disks, RAIDZ3 three disks.

When a RAIDZ array fails, the decision depends on how many members are genuinely missing or unreadable. If you are within the parity level, the missing data can be recomputed from parity. If you exceed the parity level by one disk (such as two failed disks in a RAIDZ1) the pool cannot fully reconstruct the data. Then the only chance is to recover the weak disks as far as possible with a hardware imager (reducing bad sectors) and then try to import the pool degraded. Thanks to parity and checksums, ZFS can bring back a surprising amount of correct data even from partially readable disks. We also covered the general approach on the hardware RAID side in our RAID 5 and 6 array rebuild article.

The most dangerous mistake is to replace disks and start a resilver once the parity level is exceeded. A resilver writes to the remaining disks and can destroy the recoverable old state. On a failed array the rule image first, diagnose second, write last is absolute.

Deep inspection and data extraction with zdb

zdb is the ZFS debugging tool, used in recovery to analyze a pool without writing to it. It can read the pool structure, datasets, spacemap consistency and specific block pointers.

zdb -e -p /image/dir <pool>        # inspect an exported pool from images
zdb -e -bcsvL <pool>               # block count and consistency scan
zdb -e -d <pool>                   # list datasets and object counts

As soon as the pool imports read only, the single goal is to copy the data to a completely separate and healthy store. If snapshots exist, taking a consistent stream with zfs send is the safest route. If the pool cannot mount but zdb can see objects, object level extraction is done in an expert lab environment. In enterprise settings, backup and disaster recovery plans should be reviewed right after this recovery; we examined this in our business continuity and disaster recovery article and our restoring from enterprise backups article.

When a pool is genuinely lost

To be honest: ZFS cannot recover everything. Data is permanently lost in these cases. If more disks than the parity level are physically unreadable and hardware recovery also fails, there is not enough data to reconstruct. If new data was written to the pool after a zpool destroy, it may have overwritten the old metadata. If a wrong resilver or format erased the recoverable txg history, there is no point left to roll back to. On an SSD based pool, if TRIM has physically freed the deleted blocks, those blocks do not come back; we detailed that mechanism in our SSD TRIM and FTL article. Accepting these limits up front avoids wasted effort and false hope.

The DSET approach

At DSET we run ZFS and RAIDZ recoveries in our lab at Ankara Hacettepe Teknokent on an image first principle. We take a read only sector image of every pool member, keep the original disks under evidence grade safety, and run all rollback and zdb analysis on the image copies. For weak or clicking disks we use a hardware imager; we touched on this in our PC-3000 lab hardware article. We see whether recovery is possible through dry runs up front, and we do not make inflated promises. If a TrueNAS, Proxmox or Linux ZFS pool will not import, contact DSET before running any write command against the pool. For Linux server file systems our ext4, Btrfs and LVM recovery article and for enterprise NAS our QNAP NAS recovery article also help.

Frequently asked questions

What is the difference between zpool import -f and -F? -f only force imports a pool that comes from a different system or was not cleanly exported; it does not change the data structure but writes host information to the pool. -F is recovery mode: it discards the last few corrupt txg values and rolls back to the most recent consistent state it can find, which may mean a small data loss. It is right to see what would be lost first with an -F -n dry run.

Two disks failed in my RAIDZ2 array, is the data gone? No, RAIDZ2 keeps two parity to survive two disk losses, so with two disks the pool should still come up. The problem begins if a third disk also fails or the remaining disks have many unreadable sectors. At that point the right move is not a resilver but to image the weak disks with a hardware imager and attempt a degraded import.

Can you recover the data without touching the pool at all? In most cases yes. Our goal is to write to the pool as little as possible, ideally not at all. Through read only imports and working on image copies, we move the data to a separate target without altering the original disks. This is the correct method both for data safety and for evidence integrity in a possible lawsuit.

Sources

  • OpenZFS documentation, pool and vdev structure, import and recovery options.
  • OpenZFS on disk format specification, uberblock, txg and block pointer structure.
  • illumos and FreeBSD zpool and zdb manual pages, recovery command behavior.
  • DSET blog: RAID 5 and 6 array rebuild and Linux ext4, Btrfs, LVM recovery.