TrueNAS Performance Tuning

The Adaptive Replacement Cache (ARC) is ZFS’s primary read cache, stored in system RAM. It uses an Algorithm that maintains two lists:

• MRU (Most Recently Used): Tracks recently accessed data.
• MFU (Most Frequently Used): Tracks frequently accessed data.

The ARC dynamically allocates space between these lists based on access patterns. When the ARC is Full, it evicts data from the list with lower hit rates.

On TrueNAS, the default ARC maximum is approximately 50% of physical RAM (5/8 of kernel addressable Memory, minus some overhead). For a dedicated NAS with 64 GB RAM, the ARC will use approximately 32 GB.

# Check current ARC statistics
kstat -p zfs:0:arcstats

# Key metrics:
# arc_size — Current ARC size in bytes
# arc_hits — Number of cache hits
# arc_misses — Number of cache misses
# arc_hit_ratio — Hit percentage (hits / (hits + misses) * 100)
# arc_meta_used — Metadata cache usage

# Set ARC maximum (in bytes, persistent via /etc/system or sysctl)
# For a system with 64 GB RAM, set ARC max to 48 GB:
# vfs.zfs.arc_max=51539607552

# On TrueNAS SCALE (Linux):
echo 51539607552 | sudo tee /sys/module/zfs/parameters/zfs_arc_max

These dataset properties control what data is cached:

For most workloads, primarycache=all is correct. For workloads where the working set is much Larger than RAM (e.g., media streaming), setting primarycache=metadata caches only file metadata (directory structures, file sizes), which can significantly reduce RAM usage while maintaining fast Directory listing performance.

The recordsize property sets the maximum block size for files in a dataset. ZFS uses variable-size Blocks from 512 bytes up to the recordsize. The block size for a specific file is determined at Write time and cannot be changed afterward.

:::caution Changing recordsize on an existing dataset only affects new writes. Existing files keep Their original block size. To benefit from a recordsize change, rewrite the data by copying files to A new dataset. :::

Incorrect recordsize causes read amplification:

• recordsize too large for small random reads: ZFS must read the entire block (e.g., 128 KB) to serve a 4 KB read request, wasting bandwidth. This is devastating for database workloads.
• recordsize too small for large sequential reads: ZFS issues many small reads instead of fewer large reads, increasing per-I/O overhead and reducing throughput.

Always enable compression unless you have a specific reason not to:

1. LZ4 is the default on TrueNAS and adds virtually no CPU overhead (it has a fast path that skips incompressible data in hardware).
2. Compression reduces the amount of data written to disk, which extends SSD lifespan and improves pool performance.
3. Compressed data takes less space in the ARC, effectively increasing cache capacity.
4. The only scenario where compression should be disabled is for data that is already compressed (encrypted files, compressed video/audio archives, ZIP files).

# Enable compression on a dataset
zfs set compression=lz4 tank/data

# Enable zstd for better compression (slightly more CPU)
zfs set compression=zstd tank/data

# Disable compression for already-compressed data
zfs set compression=off tank/media/encrypted

When dedup=onZFS maintains a hash table of every unique block written to the pool. Before Writing a new block, ZFS checks if an identical block already exists. If it does, ZFS stores a Reference instead of a new copy.

The deduplication table (DDT) requires approximately 320 bytes per unique block in RAM:

:::caution In most NAS deployments, deduplication is a net negative. The memory cost of the DDT far Exceeds the space savings from deduplication. Use compression (lz4/zstd) instead — it provides Meaningful space savings with no memory cost. :::

TrueNAS uses Samba’s smbd daemon for SMB file sharing. Tuning options include:

Add these in the SMB share configuration under “Auxiliary Parameters”:

# Performance tuning
aio read size = 1
aio write size = 1
strict locking = no
strict sync = no

# For macOS compatibility
vfs objects = catia fruit streams_xattr
fruit:aapl = yes
fruit:encoding = native

# For large directory support
directory name cache size = 0

SMB3 multichannel allows multiple network connections between client and server, increasing Throughput and providing failover:

1. Ensure the TrueNAS server has multiple network interfaces (or a bonded interface).
2. Enable multichannel in the TrueNAS SMB service settings.
3. The Windows client will automatically detect and use multichannel if it has multiple paths to the server.

# Mount options for optimal performance
mount -t nfs4 -o rw,hard,intr,_netdev,rsize=1048576,wsize=1048576,noatime \
    nas:/mnt/pool/data /mnt/data

# Key options:
# rsize/wsize=1M — Maximum read/write size (1 MB is optimal for 10GbE+)
# hard — Retry on server failure (do not use soft for persistent storage)
# noatime — Do not update access times (reduces metadata writes)
# _netdev — Wait for network before mounting

# Increase NFS server threads (default is 16)
# More threads = more concurrent NFS requests
# Set based on expected client count (e.g., 64 for 20+ clients)

Configure under Sharing → Unix (NFS) Shares → Settings.

For NFS, the default sync behavior depends on the client’s mount options. ZFS’s copy-on-write Ensures data integrity regardless of the NFS sync setting, but async mode can return “success” to The client before the data is actually stable on disk.

Jumbo frames increase the MTU from 1500 bytes to 9000 bytes, reducing per-packet overhead and Increasing throughput for large transfers:

# Set MTU to 9000 on the NAS and all connected devices
ifconfig igb0 mtu 9000

# Verify
ifconfig igb0 | grep mtu

:::caution Jumbo frames must be configured on every device in the network path — NAS, switch, and Clients. A single device with MTU 1500 in the path will cause fragmentation, which is worse than Standard frames. Only enable jumbo frames if you control the entire network path. :::

Link aggregation (LACP, IEEE 802.3ad) bonds multiple network interfaces into a single logical Interface for increased throughput and redundancy:

Important: LACP aggregates bandwidth per-flow, not per-connection. A single SMB or NFS transfer Will use only one link. Aggregation benefits multi-client or multi-stream workloads.

For 10 GbE and faster networks:

1. Use SFP+ or SFP28 direct attach cables (DAC) for distances up to 3 meters.
2. Use fiber optic modules for distances over 3 meters.
3. Ensure the switch supports the full line rate for all connected ports.
4. Enable flow control (802.3x) to prevent packet loss under heavy load.
5. Set TCP buffer sizes appropriately:

# Increase TCP buffer sizes (Linux client)
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 1048576 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 1048576 16777216"

ECC (Error-Correcting Code) RAM detects and corrects single-bit errors and detects multi-bit errors. For ZFS:

• ZFS stores data and checksums in RAM during operations. A bit flip in RAM can cause ZFS to write corrupted data to disk, which the checksum will not catch (the checksum matches the corrupted data).
• ECC RAM is strongly recommended for any ZFS deployment, particularly for large pools where the probability of RAM errors is non-trivial.

A dedicated SLOG device accelerates synchronous writes:

L2ARC extends the ARC to SSD storage:

• L2ARC is most effective when the working set exceeds RAM but fits in RAM + L2ARC.
• The optimal L2ARC size is 5–10x the ARC size. An L2ARC smaller than the ARC provides minimal benefit because the ARC metadata to track L2ARC entries consumes significant RAM.
• Use a dedicated SSD for L2ARC, not a partition of the pool’s SSD vdev.

:::caution Always use an HBA in IT (Initiator Target) mode for ZFS. RAID controllers hide disk Identity and prevent ZFS from performing its error detection, self-healing, and direct disk Management. Flash RAID controllers to IT mode (LSI 9211-8i, LSI 9300-8i) or buy pre-flashed HBAs. :::

Optimal pool layout for general use:

# Mirror vdevs for best random I/O performance
zpool create -o ashift=12 -O compression=lz4 -O atime=off \
    tank \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    mirror /dev/sde /dev/sdf

Optimal pool layout for archival storage:

# RAIDZ2 for capacity efficiency with redundancy
zpool create -o ashift=12 -O compression=zstd -O atime=off \
    tank \
    raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf \
    raidz2 /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl

# Key ARC metrics
kstat -p zfs:0:arcstats:hits
kstat -p zfs:0:arcstats:misses
kstat -p zfs:0:arcstats:arc_size
kstat -p zfs:0:arcstats:arc_meta_used
kstat -p zfs:0:arcstats:l2_hits
kstat -p zfs:0:arcstats:l2_misses

You cannot add a single disk to an existing RAIDZ vdev. RAIDZ vdevs are fixed-size — you can only Add entire new vdevs to the pool (which stripes data across all vdevs). To expand a RAIDZ pool, add A new RAIDZ vdev of the same type.

The exception is that you can replace individual disks in a vdev with larger disks (one at a time), And after all disks in the vdev are replaced, the vdev’s capacity expands to match the largest disk.

Setting the ARC too large (e.g., 90% of RAM) leaves insufficient memory for the OS, applications, And the ZFS prefetch cache. This can cause swapping, which is catastrophic for ZFS performance. Keep The ARC at 50–70% of physical RAM for dedicated NAS systems.

Enabling dedup on a large pool without sufficient RAM (128+ GB) will cause the DDT to overflow to Disk, which destroys performance. The system may become unresponsive. If you must use dedup, ensure You have enough RAM for the entire DDT in memory.

Creating a pool with ashift=9 (512 bytes) on drives with 4 KB physical sectors causes severe Read-modify-write amplification. Every 4 KB write becomes a read-modify-write of a 512-byte-aligned Region, potentially touching multiple 4 KB physical sectors. Always use ashift=12 or higher.

Putting all data in a single dataset with a single recordsize means that all workloads share the Same block size. Database workloads suffer with 128K recordsize, and media streaming suffers with 8K Recordsize. Create separate datasets with appropriate recordsize for different workload types.

The ARC is a slab allocator that manages memory in several categories:

# Detailed ARC statistics
kstat -p zfs:0:arcstats

# Key metrics to calculate hit ratio:
# arc_hits — Cache hits (data served from ARC)
# arc_misses — Cache misses (data read from disk)
# arc_prefetch_data_misses — Prefetch misses (data prefetched but not used)

# Calculate hit ratio:
# hit_ratio = arc_hits / (arc_hits + arc_misses)

# ARC demand data hit ratio (excluding prefetch):
# demand_hit_ratio = arc_hits / (arc_hits + arc_demand_data_misses)

ZFS can prefetch data that it predicts will be needed soon:

• Stream prefetch: Detects sequential access patterns and prefetches ahead.
• Offset prefetch: Detects strided access patterns.

Prefetch can significantly improve performance for sequential workloads but wastes memory for random Workloads (prefetched data evicts useful data from the ARC).

# Disable prefetch for random workloads
zfs set prefetch_disable=1 tank/database

# Enable prefetch for sequential workloads (default)
zfs set prefetch_disable=0 tank/media

When the ARC is full, it evicts data to make room for new data. The eviction policy considers:

1. Ghost lists: Data is moved to a ghost list (metadata only, no data). If the data is accessed again, it is re-read from disk and placed at the head of the active list.
2. MFU vs MRU: Data in the MRU list (accessed once) is evicted before data in the MFU list (accessed multiple times).
3. Metadata vs data: By default, metadata is not evicted to make room for data. This can cause metadata to consume excessive ARC space on metadata-heavy workloads.

Databases issue I/O in page-sized chunks ( 8 KB for PostgreSQL, 16 KB for MySQL InnoDB). Setting recordsize to match the database page size is critical:

# PostgreSQL (8 KB pages)
zfs create -o recordsize=8K -o primarycache=metadata -o logbias=latency \
    tank/databases/postgres

# MySQL InnoDB (16 KB pages)
zfs create -o recordsize=16K -o primarycache=metadata -o logbias=latency \
    tank/databases/mysql

Why primarycache=metadata for databases?

Most databases have their own buffer pool (shared_buffers in PostgreSQL, innodb_buffer_pool_size in MySQL). Caching the same data in both the database buffer pool and the ZFS ARC is redundant. Setting primarycache=metadata caches only ZFS metadata (which the database does not cache), reducing ARC Memory usage and improving overall efficiency.

VM images exhibit mixed I/O patterns: small random reads/writes for metadata and large sequential Reads for boot and application loading.

# VM images
zfs create -o recordsize=64K -o compression=lz4 tank/vms

# If using qcow2 or vmdk (which have internal 64K blocks)
# If using raw images for databases, use 16K instead

# Media files (video, audio, images)
zfs create -o recordsize=128K -o compression=lz4 tank/media
# Or for very large files (4K video):
zfs create -o recordsize=1M -o compression=off tank/media/4k-video

# Key SMB performance parameters (add to auxiliary parameters):
aio read size = 1         # Enable async I/O for reads
aio write size = 1        # Enable async I/O for writes
strict sync = no           # Skip fsync on close (performance)
strict locking = no        # Reduce lock contention
strict rename = no         # Allow rename of open files
kernel share modes = no    # Reduce share mode overhead
use sendfile = yes         # Use sendfile syscall for zero-copy transfers
min receivefile size = 16384  # Minimum size for sendfile
getwd cache = yes          # Cache getwd() results

For SMB multichannel to work:

1. TrueNAS must have multiple network interfaces.
2. The client must be configured for multichannel (Windows 8+ and macOS support it natively).
3. All interfaces must be on the same subnet (or use SMB3 encryption for cross-subnet).
4. No manual bonding is needed. SMB handles it at the protocol level.

# Verify multichannel on TrueNAS
smbstatus

# Check active sessions and channels
# Multiple entries for the same client IP indicate multichannel is active

# Check current NFS thread count
ps aux | grep nfsd | wc -l

# On TrueNAS, configure under Sharing → Unix NFS Shares → Settings
# Recommended values:
# 16 threads for 1-5 clients
# 32 threads for 5-15 clients
# 64 threads for 15+ clients

# Full reference for optimal NFSv4 mounts:
mount -t nfs4 \
    -o rw \
    -o hard \
    -o intr \
    -o _netdev \
    -o noatime \
    -o nodiratime \
    -o rsize=1048576 \
    -o wsize=1048576 \
    -o tcp \
    -o actimeo=60 \
    nas:/mnt/pool/data /mnt/data

# actimeo=60 — Attribute cache timeout (60 seconds)
#   Longer timeout reduces metadata operations for read-mostly workloads
#   Shorter timeout improves consistency for write-heavy workloads

# Set TCP buffer sizes (Linux client)
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 1048576 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 1048576 16777216"
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Enable BBR congestion control for better throughput on high-BDP networks
# BBR (Bottleneck Bandwidth and RTT) outperforms CUBIC on networks with
# high bandwidth-delay product (typical of 10GbE+ links)

# On TrueNAS, configure under Network → Interfaces
# Create a LAGG interface:
# Protocol: LACP
# Ports: igb0, igb1 (or your NIC names)
# Hash policy: L4 (layer 4 — source/destination IP + port)

# LACP requires switch configuration:
# Switch side:
#   interface Port-channel1
#   description LACP to TrueNAS
#   switchport mode trunk
#   channel-group 1 mode active

ZFS stores data and checksums in RAM during read and write operations. If a bit flip occurs in RAM:

1. On write: The corrupted data and its checksum are written to disk. The checksum matches the corrupted data, so ZFS cannot detect the corruption on a subsequent read. The data is permanently corrupted.
2. On read: The corrupted data is returned to the application. The checksum verification may or may not catch it depending on which bits flipped.

ECC RAM detects and corrects single-bit errors in real time, preventing this scenario.

Popular HBAs for ZFS:

Flashing LSI cards to IT mode:

# On a Linux system with the HBA installed:
# 1. Download the sas2flash or sas3flash utility
# 2. Download the IT mode firmware (e.g., mpt2sas.20.00.07.00-fw.bin)
# 3. Flash the firmware:
./sas2flash -o mpt2sas.20.00.07.00-fw.bin -b mptsas2.rom

# Pool I/O statistics
zpool iostat -v tank 5

# ARC statistics
kstat -p zfs:0:arcstats:hits zfs:0:arcstats:misses zfs:0:arcstats:arc_size

# Dataset-level I/O statistics
zfs get all tank/data | grep -E "written|read|used|compressratio"

# Vdev I/O latency
zpool iostat -v tank 1 5

# Disk utilization
iostat -x 5

# NFS statistics
nfsstat -c

# Sequential read test (using zfs send as a benchmark)
zfs send -R tank/data@snapshot > /dev/null

# Random I/O test (using fio)
fio --name=randrw --ioengine=libaio --iodepth=32 --rw=randrw \
    --rwmixread=70 --bs=4k --direct=1 --size=10G \
    --directory=/mnt/tank/data --runtime=300 --group_reporting

# Metadata-heavy workload test (creating/reading many small files)
mkdir /mnt/tank/data/metadata-test
for i in $(seq 1 10000); do dd if=/dev/zero of=/mnt/tank/data/metadata-test/file_$i bs=4k count=1; done

The physical layout of vdevs within a pool has a profound impact on performance, resilience, and Future expandability. Different workloads demand different vdev topologies.

graph TD
    A[Database Workload] --> B{I/O Pattern}
    B -->|Random Write Heavy| C[Mirror vdevs]
    B -->|Sequential Read Heavy| D[RAIDZ2 vdevs]
    C --> E[Higher IOPS / Lower Capacity]
    D --> F[Higher Capacity / Lower IOPS]
    E --> G[ZIL SLOG recommended]
    F --> H[ZIL SLOG optional]

For databases (PostgreSQL, MySQL), mirror vdevs provide significantly higher random IOPS because Each mirror pair can serve reads from both drives simultaneously. A pool of 8 drives configured as 4x mirror vdevs delivers approximately 4x the IOPS of the same 8 drives in a single RAIDZ2 vdev.

ZFS special vdevs store metadata and small blocks on fast storage (NVMe or SSD), leaving the Spinning disk array to handle only large sequential data blocks. This dramatically improves Metadata-heavy operations:

# Create a special vdev with NVMe devices
zpool add -o ashift=12 tank special mirror /dev/nvme0n1p1 /dev/nvme1n1p1

# Create a dedup vdev (only if deduplication is actually needed)
zpool add -o ashift=12 tank dedup mirror /dev/nvme2n1p1 /dev/nvme3n1p1

# Verify special vdev allocation
zpool list -v tank
zdb -bb tank 2>/dev/null | head -30

:::caution Special vdevs cannot be removed after creation. If a special vdev fails, the entire pool Is at risk. Always mirror special vdevs and use high-endurance NVMe drives rated for sustained write Workloads. Check the DWPD (Drive Writes Per Day) rating and ensure it meets your projected metadata Write volume. :::

The recordsize property controls the maximum block size ZFS uses for files. Matching recordsize to Your workload’s typical I/O size is critical:

# Set recordsize for a PostgreSQL dataset
zfs create -o recordsize=16K -o primarycache=metadata -o logbias=latency \
  tank/postgres/data

# Set recordsize for a media library
zfs create -o recordsize=1M -o compression=zstd tank/media/videos

# Verify current settings
zfs get all tank/postgres/data | grep -E "recordsize|primarycache|logbias|compression"

:::tip The recordsize only affects new writes. Existing files retain their original block size Until they are rewritten. To reblock existing data, copy files to a new dataset with the desired Recordsize. :::

The Adaptive Replacement Cache (ARC) is ZFS’s primary caching mechanism. Monitoring ARC hit rates Helps determine whether you need more RAM or an L2ARC device.

# Quick ARC summary
arc_summary

# Individual arc_stats (key fields)
cat /proc/spl/kstat/zfs/arcstats | grep -E \
  "^hits|^misses|^size|^c_max|^deleted|^evict|^mfu|^mru|^prefetch|^demand"

# Calculate ARC hit rate percentage
hits=$(cat /proc/spl/kstat/zfs/arcstats | awk '/^hits/ {print $3}')
misses=$(cat /proc/spl/kstat/zfs/arcstats | awk '/^misses/ {print $3}')
total=$((hits + misses))
hit_rate=$(echo "scale=2; $hits * 100 / $total" | bc)
echo "ARC hit rate: ${hit_rate}%"

# If hit rate is below 80%, consider adding RAM or an L2ARC

By default, ZFS limits ARC to approximately 5/8 of physical RAM. On a dedicated NAS with 128GB RAM, This leaves about 48GB for applications and the kernel. If TrueNAS is the only workload, you can Increase the ARC ceiling:

# Set ARC max to 90% of RAM (128GB system = ~115GB ARC)
echo $(echo '128 * 1024 * 1024 * 1024 * 90 / 100' | bc) > /sys/module/zfs/parameters/zfs_arc_max

# Make persistent across reboots (TrueNAS SCALE)
# Add to /etc/modprobe.d/zfs.conf
echo "options zfs zfs_arc_max=123480309760" > /etc/modprobe.d/zfs.conf

:::caution Setting zfs_arc_max too high leaves insufficient memory for the kernel, applications, And the ZFS prefetch cache. Never set it above 90% of physical RAM, and monitor swap usage after Changes. If the system begins swapping, reduce zfs_arc_max immediately. :::

L2ARC (Level 2 ARC) uses a dedicated SSD as a second-tier cache. It stores ARC evictions and can Dramatically improve read performance for datasets larger than physical RAM.

# Add an L2ARC device
zpool add tank cache /dev/sdb

# Verify L2ARC is active
zpool list -v | grep cache
zdb -C tank | grep -A5 "L2ARC"

L2ARC effectiveness depends on the working set size relative to RAM and L2ARC size. If your working Set is 200GB and you have 64GB RAM plus a 500GB L2ARC SSD, the L2ARC will be effective. If your Working set is 10TB, even a large L2ARC will have a low hit rate because the cache is too small Relative to the dataset.

# Monitor L2ARC effectiveness
arc_summary | grep -A10 "L2 ARC"

# Key fields to watch:
# L2 ARC Hit Ratio: Should be > 30% to justify the SSD cost
# L2 ARC Writes: Should be stable (not constantly writing and evicting)
# L2 ARC Size: Should approach the device capacity over time

This topic covers the essential concepts and techniques related to truenas performance tuning, including key principles and practical applications.

Key concepts include:

• core concepts and definitions
• key principles and frameworks
• practical applications
• common techniques and methods
• evaluation and critical analysis

A thorough understanding of these concepts, combined with regular practice and review, is essential for mastery of this topic.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.