ZFS Deep Dive

ZFS is not a traditional filesystem. It is a combined volume manager and filesystem built from three Distinct layers:

graph TD
    A[Application / POSIX Interface] --> B[ZPL - ZFS POSIX Layer]
    B --> C[DMU - Data Management Unit]
    C --> D[SPA - Storage Pool Allocator]
    D --> E[Physical Storage - vdevs]

1. ZPL (ZFS POSIX Layer): The filesystem layer that provides POSIX-compliant semantics. Files, directories, permissions, extended attributes, and ACLs. It translates file operations into block operations.
2. DMU (Data Management Unit): The transactional layer that manages objects, blocks, and snapshots. All writes are handled as atomic transactions. The DMU also manages the ARC (Adaptive Replacement Cache).
3. SPA (Storage Pool Allocator): The lowest layer that manages physical storage. It handles vdev topology, I/O scheduling, checksumming, compression, and self-healing.

ZFS never overwrites data in place. Every write creates a new copy of the data block, and only after The new block is written and its checksum verified does ZFS update the metadata to point to the new Block. This has several consequences:

• Snapshots are instantaneous and free (initially). A snapshot is a marker in the transaction history that prevents old blocks from being freed.
• No write hole. Unlike hardware RAID, a power loss during a write cannot leave data and parity in an inconsistent state. Either the old data or the new data is referenced, never a partial update.
• Fragmentation is inevitable. Over time, as blocks are updated and freed, the pool becomes fragmented. This is the primary trade-off of copy-on-write.

Every block in ZFS is identified by its content hash (SHA-256 by default), forming a Merkle tree. The root of the Merkle tree (the block pointer) stores the checksum of its child blocks, and so on Recursively.

When ZFS reads a block, it:

1. Reads the block and its checksum from disk.
2. Recomputes the checksum of the read data.
3. Compares the computed checksum with the stored checksum.
4. If they match, the data is returned.
5. If they do not match (silent corruption), ZFS uses redundancy (mirror or parity) to reconstruct the correct data and repair the corrupted copy.

:::info Set the checksum algorithm at pool creation time with -O checksum=sha256. It cannot be Changed after pool creation. edonr is the fastest on hardware with SSE4.2+ support and provides Excellent collision resistance. :::

A ZFS pool (zpool) is constructed from one or more vdevs (virtual devices). The vdev is the Fundamental unit of redundancy and performance. Data is striped across vdevs for performance.

graph TD
    A[zpool] --> B[vdev 0: mirror-0]
    A --> C[vdev 1: raidz2-0]
    B --> D[disk1]
    B --> E[disk2]
    C --> F[disk3]
    C --> G[disk4]
    C --> H[disk5]
    C --> I[disk6]

RAIDZ uses dynamic stripe width. Unlike traditional RAID 5/6 where the stripe width is fixed (e.g., 4+1 for RAID 5 with 5 drives), RAIDZ varies the stripe width based on the size of the incoming Write:

• Small writes (less than one sector per data disk) are written as “full stripe” writes with variable-width padding.
• Large writes that fill the stripe exactly avoid any padding overhead.
• Medium writes may leave some sectors unused (wasted space).

This is why RAIDZ capacity is slightly less than the theoretical (N-P)/N formula, where P is the Parity count. The actual usable capacity depends on the recordsize and write patterns.

The ashift property controls the physical sector size that ZFS assumes for the drives. It must be Set at pool creation time and cannot be changed afterward.

:::caution Always set ashift=12 (4 KB) at minimum for modern drives. Setting ashift=9 on a drive With 4 KB physical sectors causes read-modify-write amplification, devastating performance. On TrueNAS, the default ashift is 12, which is correct for virtually all modern drives. :::

# Mirror pool (2-way mirror)
zpool create tank mirror /dev/sda /dev/sdb

# RAIDZ2 pool with ashift=12
zpool create -o ashift=12 tank raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd

# Hybrid pool: SSD mirror for special vdev + HDD RAIDZ2 for data
zpool create -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1 \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

A dataset (zfs filesystem) is a logical namespace within a pool. Each dataset has its own Properties, mount point, and can have its own snapshots, quotas, and compression settings.

# Create a dataset
zfs create tank/data

# Set properties
zfs set compression=lz4 tank/data
zfs set atime=off tank/data
zfs set recordsize=128K tank/data
zfs set quota=500G tank/data

# List all properties
zfs get all tank/data

The recordsize property determines the maximum block size ZFS uses for a file. ZFS uses Variable-size blocks up to this maximum. The optimal recordsize depends on the workload:

:::caution Changing recordsize on an existing dataset only affects new writes. Existing files Retain their original block sizes. To benefit from a recordsize change, you must rewrite the data (e.g., copy files to a new dataset with the desired recordsize). :::

Because ZFS is copy-on-write, a snapshot is a point-in-time marker in the transaction History. Creating a snapshot is instantaneous and consumes no space initially. Space is consumed Only when blocks referenced by the snapshot are modified or deleted in the live filesystem.

The space used by a snapshot is the total size of blocks that have been modified or deleted in the Live filesystem since the snapshot was taken. This is called “written” space:

# List snapshots and their space usage
zfs list -t snapshot -o name,used,refer,written

# Check space used by a specific snapshot
zfs list -o name,used,refer tank/data@daily.2024-01-01

# Create a snapshot
zfs snapshot tank/data@daily.2024-01-01

# List snapshots
zfs list -t snapshot

# Destroy a snapshot
zfs destroy tank/data@daily.2024-01-01

# Destroy snapshots matching a pattern
zfs destroy tank/data@daily.2023-*

# Clone a snapshot (creates a writable copy)
zfs clone tank/data@daily.2024-01-01 tank/data-restore

# Promote a clone (make it independent of the snapshot)
zfs promote tank/data-restore

The ARC is ZFS”s primary read cache, stored in system RAM. It uses the Adaptive Replacement Cache Algorithm, which maintains two lists:

• MRU (Most Recently Used): Recently accessed data.
• MFU (Most Frequently Used): Frequently accessed data.

The ARC dynamically balances between these two lists, evicting from the list with lower hit rates. This performs better than a simple LRU cache for mixed workloads with both sequential and random Access patterns.

ARC sizing: The default ARC maximum is 50% of system RAM on TrueNAS. For dedicated NAS Workloads, increasing the ARC to 70–80% of RAM can significantly improve read performance for hot Datasets.

# Check ARC stats
zfs get arcstats 2>/dev/null || cat /proc/spl/kstat/zfs/arcstats

# Key metrics:
# arc_hits    — Cache hits
# arc_misses  — Cache misses
# arc_hit_ratio — Percentage of reads served from cache

The L2ARC is a secondary read cache stored on a dedicated SSD (or partition of an SSD). When the ARC Evicts data, it can write it to the L2ARC before discarding it entirely. On subsequent accesses, if The data is not in the ARC but is in the L2ARC, it can be read from the L2ARC rather than from the Slower pool disks.

L2ARC considerations:

• L2ARC is read-through, not write-through. Data is written to L2ARC only when evicted from ARC.
• The L2ARC does not speed up writes — only reads.
• L2ARC requires significant ARC space to be effective. The ARC metadata for tracking L2ARC entries consumes RAM.
• L2ARC is most effective when the working set is larger than ARC but smaller than ARC + L2ARC.

The SLOG (Separate Log) is an accelerator for synchronous writes. When a synchronous write request Arrives (from NFS, SMB sync, or a database), ZFS must ensure the data is on stable storage before Acknowledging the write. Without a SLOG, this means writing directly to the pool, which is slow for HDD-based pools.

A dedicated SLOG device ( a low-latency NVMe SSD or Intel Optane) absorbs synchronous Writes at SSD speed, then asynchronously flushes them to the pool. This dramatically improves NFS And database write performance on HDD-based pools.

:::caution The SLOG must have power-loss protection (PLP). Without PLP, a power loss during a Synchronous write can lose acknowledged data, violating the sync guarantee. Intel Optane DC Persistent memory is the gold standard for SLOG devices. Enterprise NVMe SSDs with PLP are also Acceptable. Consumer NVMe SSDs without PLP should not be used as SLOG devices. :::

A scrub reads all data in the pool and verifies checksums. If a checksum mismatch is detected (the Block is corrupted), ZFS automatically repairs it from a redundant copy (mirror or parity).

# Start a scrub
zpool scrub tank

# Check scrub status
zpool status tank

# Scrub scheduling on TrueNAS:
# Configure under Data Protection → Scrub Tasks
# Recommended: Monthly scrubs for HDD pools, Weekly for SSD pools

Scrub best practices:

• Run scrubs at off-peak hours. Scrubbing a large HDD pool can take days and significantly impacts pool performance.
• SSD pools scrub much faster (hours instead of days) due to higher throughput.
• Monitor scrub progress with zpool status. The scrub will report any errors found and repaired.
• If a scrub finds uncorrectable errors, immediately back up critical data and replace the failing drive.

A resilver rebuilds the data on a replaced drive. Unlike traditional RAID rebuilds, ZFS resilvers Only copy the actual data (not the entire disk), and they prioritize data based on its metadata Importance.

# Replace a failed drive
zpool replace tank /dev/sda /dev/sdb

# Monitor resilver progress
zpool status tank

:::caution During a resilver, the pool is vulnerable. If a second drive fails during resilver of a RAIDZ1 pool, all data is lost. For RAIDZ2, you can tolerate a second failure. Always monitor Resilver progress and ensure the pool is healthy before and after. :::

ZFS send/receive is the native mechanism for replicating datasets between pools or systems. It works At the block level, sending only the changed blocks between two snapshots.

# Full replication (initial)
zfs send tank/data@snapshot1 | zfs recv backup/data

# Incremental replication (send only changes since snapshot1)
zfs send -i tank/data@snapshot1 tank/data@snapshot2 | zfs recv backup/data

# Replication with compression over SSH
zfs send -Rcv tank/data@snapshot1 | ssh nas2 zfs recv backup/data

# Raw send (preserves encryption and compression)
zfs send -w tank/data@snapshot1 | zfs recv backup/data

zpool status -v tank

Key fields to understand:

With modern drives (8 TB+), the probability of encountering an unrecoverable read error (URE) during A resilver approaches certainty. A RAIDZ1 pool with 8 TB drives has a resilver time of 12–24 hours. During that time, reading every block on every remaining drive means the chance of hitting a URE (and losing the pool) is non-trivial. Use RAIDZ2 (or RAIDZ3) for any pool with drives larger than 4 TB.

Deduplication maintains an in-memory hash table of every unique block. This table requires Approximately 320 bytes per unique block. A 10 TB pool with 4 TB of unique data can require 100+ GB Of RAM for the dedup table. If the system runs out of RAM and must swap, performance collapses. Only Enable dedup if your data is highly redundant (VM templates, ISO images) and you have sufficient RAM. In most cases, compression (lz4) provides better space savings with no memory cost.

While ZFS allows mixing drive sizes in a RAIDZ vdev, the pool capacity is determined by the smallest Drive in the vdev. A RAIDZ2 vdev with three 12 TB drives and one 4 TB drive will have the capacity Of four 4 TB drives. Always use identical drives within a vdev.

Once a pool is created, ashift cannot be changed. Creating a pool with ashift=9 (512 bytes) on Drives with 4 KB physical sectors causes severe read-modify-write amplification on small writes, Reducing performance by 50–80%. Always use ashift=12 or higher.

ZFS pools become fragmented over time due to the copy-on-write nature. Fragmentation above 70–80% Can significantly reduce performance, especially for random read workloads. Monitor fragmentation With zpool list -v. There is no native defragmentation tool for ZFS — the only way to defragment Is to copy the data to a new pool. Regular snapshot pruning and avoiding small random writes on HDD Pools help keep fragmentation manageable.

Mirrors are the gold standard for performance and redundancy:

# 2-way mirror (most common)
zpool create -o ashift=12 -O compression=lz4 -O atime=off tank \
  mirror /dev/sda /dev/sdb \
  mirror /dev/sdc /dev/sdd \
  mirror /dev/sde /dev/sdf

# Performance characteristics:
# Read: N × single-disk IOPS (any disk in a mirror can serve the read)
# Write: N × single-disk IOPS (writes go to all mirrors simultaneously)
# Capacity: 50% of total raw
# Fault tolerance: 1 disk per mirror vdev

Mirror pools provide the best random I/O performance because every vdev can serve reads Independently. A 6-disk mirror pool (3 mirror vdevs) can serve 3× the random IOPS of a single disk.

RAIDZ2 provides dual-parity protection at better capacity efficiency:

# RAIDZ2 with 8 drives per vdev
zpool create -o ashift=12 -O compression=lz4 -O atime=off tank \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh \
  raidz2 /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp

# Performance characteristics:
# Read: Good (reads span all data disks)
# Write: Moderate (parity calculation overhead)
# Capacity: (N-2)/N of total raw per vdev
# Fault tolerance: 2 disks per vdev

DRAID is a ZFS feature that distributes spare capacity across all drives in the pool, rather than Dedicating entire drives as hot spares:

# dRAID2 with distributed spares
zpool create -o ashift=12 tank \
  draid2:2d:8c:2s /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh \
  /dev/sdi /dev/sdj

# Parameters:
# 2d     = 2 data drives per stripe
# 8c     = 8 children (drives) per redundancy group
# 2s     = 2 distributed spares

DRAID provides faster resilvering than traditional RAIDZ because all drives participate in Rebuilding simultaneously.

For workloads with mixed metadata and data requirements:

# NVMe metadata vdev + HDD data vdev
zpool create -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1 \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

# Assign special small blocks to NVMe
zfs create -o special_small_blocks=32K tank/data

This stores metadata (directories, file attributes) on the fast NVMe vdev while data resides on the HDD vdev, dramatically improving directory listing performance.

On modern CPUs (8+ cores), lz4 compression overhead is negligible for most workloads. The I/O time Saved by writing less data to disk exceeds the CPU time spent compressing.

Disable compression only for data that is already compressed or encrypted:

# Disable compression for an existing dataset
zfs set compression=off tank/media/movies
zfs set compression=off tank/backups/encrypted
zfs set compression=off tank/software/isos

The ARC maintains five lists:

1. MRU (Most Recently Used): Ghost + active MRU lists.
2. MFU (Most Frequently Used): Ghost + active MFU lists.
3. Metadata (ARC meta): Separate cache for metadata (dnode structures, directory entries).

The replacement algorithm:

• New data enters the MRU list.
• On a cache hit, data is promoted from MRU to MFU (if accessed multiple times).
• On eviction, data moves from active to ghost list. Ghost entries remember the data’s identity but not its content.
• If a ghost entry is accessed again (cache miss → hit in ghost list), the data is fetched from disk and placed at the head of the appropriate active list.
• The ARC size is bounded by arc_max (primary cache) and arc_meta_limit (metadata cache).

Metadata (directory entries, file attributes, indirect blocks) can consume a significant portion of The ARC. The arc_meta_limit parameter controls the maximum fraction of ARC dedicated to metadata:

# Default: 1/4 of ARC
# Recommended for metadata-heavy workloads: 1/2 to 3/4

# Check current metadata usage
kstat -p zfs:0:arcstats:arc_meta_used
kstat -p zfs:0:arcstats:arc_meta_max

# The metadata-to-data ratio indicates workload characteristics:
# High metadata/data ratio → Many small files (mail server, source code, home directories)
# Low metadata/data ratio → Few large files (media, VM images, backups)

# Recommended naming convention:
tank/data@daily-2024-01-15
tank/data@weekly-2024-W03
tank/data@monthly-2024-01
tank/data@pre-upgrade-2024-01-15T10-30-00
tank/data@manual-description

# List snapshots with sorting
zfs list -t snapshot -s creation -o name,creation,used,refer

# Show differences between two snapshots
zfs diff tank/data@snap1 tank/data@snap2

# Output format:
# M       +       path      # Modified file
# M       -       path      # Deleted file
# +       +       path      # New file
# R       +       old -> new # Renamed file

# Rollback a dataset to a specific snapshot (DESTRUCTIVE: destroys all snapshots taken after)
zfs rollback tank/data@daily-2024-01-10

# Force rollback (discard changes since snapshot)
zfs rollback -rf tank/data@daily-2024-01-10

:::caution zfs rollback destroys all intermediate snapshots between the current state and the Target snapshot. Use zfs clone instead if you want to preserve the current state. :::

# Raw send preserves encryption without requiring the key on the receiving side
zfs send -Rwv tank/encrypted@snap1 | ssh remote zfs recv -F backup/encrypted

# The receiving system cannot read the data without the encryption key
# This is ideal for offsite backup where the remote system should not have access

# Send with resume token (saved periodically)
zfs send -Rv -t <resume-token> | ssh remote zfs recv -s backup/data

# The resume token is printed when a transfer is interrupted (Ctrl+C)
# Save it and use it to resume the transfer later

# Use pv to limit bandwidth
zfs send -Rcv tank/data@snap1 | pv --rate-limit 50m | ssh remote zfs recv -F backup/data

# 50m = 50 MB/s
# Adjust based on available bandwidth and impact on production workloads

# Replace a smaller disk with a larger one (one at a time)
zpool replace tank /dev/sda /dev/sdb-new

# After all disks in a vdev are replaced with larger disks:
# The vdev automatically expands to use the full capacity
zpool list -v

# Add a new vdev to the pool (stripes across vdevs)
zpool add tank mirror /dev/sdg /dev/sdh

# Add a cache device (L2ARC)
zpool add cache tank /dev/nvme0n1

# Add a log device (SLOG)
zpool add log tank /dev/nvme1n1p1

# Add a spare device
zpool add spare tank /dev/sdi

# Export a pool (unmount all datasets)
zpool export tank

# Import a pool
zpool import tank

# Import a pool from a specific cachefile (after disk replacement)
zpool import -c /path/to/zpool.cache tank

# Import by GUID (more reliable than by name)
zpool import <guid>

# Force import (if pool was not properly exported)
zpool import -f tank

:::caution Always export a pool before disconnecting drives. If a pool is not exported, ZFS may mark It as active on the original system, preventing import on the new system. Use zpool export -f to Force export if necessary.

ZFS batches writes into transaction groups for efficiency:

1. When a write request arrives, ZFS allocates a block and writes the data.
2. The block is checksummed and added to the current TXG.
3. When the TXG fills (every ~5 seconds) or is flushed, the entire group is written to disk atomically.
4. The ZIL (ZFS Intent Log) ensures that synchronous writes survive power loss by writing the intent to the log before the TXG is flushed.

The ZIL records the intent to modify data before the modification is committed to the pool:

• Synchronous writes: Data is written to the ZIL and acknowledged to the client. Only after the ZIL is persisted (or the data reaches the main pool) does ZFS acknowledge the write.
• The ZIL is per-dataset. Some datasets may need synchronous writes (databases), while others do not (media files).
• ZIL is in-memory by default. A dedicated SLOG device (NVMe SSD) accelerates synchronous writes by providing a fast persistent log.

graph TD
    A[Application Write] --> B[ZFS Write]
    B --> C[Allocate Block in DMU]
    C --> D[Compute Checksum]
    D --> E[Add to Current TXG]
    E --> F{Sync Write?}
    F -->|Yes| G[Write to ZIL]
    G --> H[Acknowledge to Client]
    F -->|No| I[Add to TXG Queue]
    I --> J[Write to Pool on TXG Flush]

graph TD
    A[Application Read] --> B[ZFS Read]
    B --> C{ARC Hit?}
    C -->|Yes| D[Return from ARC]
    C -->|No| E{L2ARC Hit?}
    E -->|Yes| F[Return from L2ARC]
    E -->|No| G[Read from Pool]
    G --> H{Checksum Match?}
    H -->|Yes| I[Return Data]
    H -->|No| J[Read Redundant Copy]
    J --> K[Repair Bad Block]
    K --> I

ZFS supports special allocation classes for metadata and small blocks:

# Create a dataset with special small block allocation on NVMe
zfs create -o special_small_blocks=32K tank/data

# This stores metadata and small files on the special vdev (NVMe)
# while regular data goes to the main vdev (HDD)

Bookmarks are point-in-time markers for a dataset, created with zfs bookmark. Unlike snapshots, They don’t hold references to blocks and don’t prevent space from being freed, making them very Lightweight. They are primarily used as send base references for incremental zfs send:

# Create a bookmark from a snapshot
zfs bookmark tank/data@snap1 tank/data#bm_snap1

# Send incremental using a bookmark as the base (snapshot can be destroyed)
zfs send -i tank/data#bm_snap1 tank/data@snap2 | zfs recv backup/data

The dedup DDT (Dedup Table) stores a hash entry for every unique block:

Memory per DDT entry: approximately 320 bytes.

For a pool with 4 TB of unique data at 128K recordsize:

This is why dedup requires careful capacity planning. A 4 TB pool with 4 TB of unique data needs ~10 GB of RAM just for the DDT.

Resilver speed depends on:

1. Pool configuration: Mirror pools resilver faster than RAIDZ pools because data can be read from any mirror copy.
2. Disk speed: Faster disks resilver faster. NVMe resilvers much faster than HDD.
3. Pool load: Resilver shares I/O bandwidth with normal operations. Scheduling during low-usage periods is recommended.
4. Priority: TrueNAS prioritizes resilver I/O over normal I/O, but both compete for disk bandwidth.

# 1. Identify the failed drive
zpool status -v tank
# Look for OFFLINE, FAULTED, or UNAVAIL status

# 2. Prepare the replacement drive
# Ensure the replacement drive is the same size or larger
# Zero the first and last megabytes if the drive was previously used:
dd if=/dev/zero of=/dev/sdx bs=1M count=1
dd if=/dev/zero of=/dev/sdx seek=$((block_count-1)) bs=1M count=1

# 3. Replace the drive
zpool replace tank /dev/sdOLD /dev/sdNEW

# 4. Monitor resilver progress
zpool status tank
# Wait until resilver completes

# 5. Verify pool health
zpool scrub tank
# Wait for scrub to complete
zpool status tank

# Check ARC efficiency
kstat -p zfs:0:arcstats:hits zfs:0:arcstats:misses
# Hit ratio should be >80% for most workloads

# Check pool I/O
zpool iostat -v tank 1

# Check for fragmentation
zdb -r -e tank 2>/dev/null | grep fragmentation

# Check for slow reads/writes
zpool iostat -v tank 5
# Look for high latency (await column)

This topic covers the essential concepts and techniques related to zfs deep dive, including key principles and practical applications.

Key concepts include:

• core concepts and definitions
• key principles and frameworks
• practical applications
• common techniques and methods
• evaluation and critical analysis

A thorough understanding of these concepts, combined with regular practice and review, is essential for mastery of this topic.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.

:::