Storage Tuning

The optimal storage strategy depends on your workload profile:

• OS and applications: NVMe SSD (PCIe 4.0+). Fast random I/O makes the system feel responsive.
• Game library: NVMe SSD for frequently played games; HDD for archival storage. Load times are dominated by sequential read speed and random read IOPS.
• Media production (video editing): NVMe SSD with high sustained write endurance. 4K/8K video requires 500 MB/s–2 GB/s sustained write.
• Database workloads: NVMe SSD with high random IOPS and low latency. Optane is ideal but expensive.
• Backup and archival: HDD or high-capacity SATA SSD (QLC). Sequential throughput matters more than latency.
• ZFS SLOG (ZIL): Enterprise NVMe SSD or Optane with power-loss protection (PLP).

NVMe (Non-Volatile Memory Express) is designed from the ground up for PCIe-attached flash storage, Replacing the legacy AHCI protocol that was designed for spinning disks.

Key architectural advantages over AHCI/SATA:

1. Multiple queues: NVMe supports up to 65,535 I/O queues, each with up to 65,535 entries. AHCI has a single command queue with 32 entries. This eliminates the queue bottleneck in multi-threaded workloads.
2. Direct CPU access: NVMe uses MSI-X interrupts and can map completion queues directly into user space, reducing interrupt overhead and enabling kernel bypass.
3. Deep queue depths: The large number of queue entries allows the storage device to optimize its internal command scheduling and garbage collection.
4. Lower latency: NVMe eliminates the SATA protocol overhead (command encoding, FIS framing, spread spectrum clocking), reducing command latency by 2–5 $\mu$S.

A namespace is the NVMe equivalent of a partition — a logical address space exposed to the host. Most consumer NVMe SSDs expose a single namespace (NSID 1) spanning the entire device. Enterprise SSDs may support multiple namespaces for partitioning.

# List NVMe devices
nvme list

# List namespaces on device nvme0
nvme list-ns /dev/nvme0

# Get namespace details
nvme id-ns /dev/nvme0n1

NVMe defines several power states (PS0–PS4) that trade off power consumption against latency:

APST (Autonomous Power State Transition) allows the SSD to transition between power states Automatically. On desktops, this is generally fine. On servers with latency-sensitive workloads, you May want to restrict APST to prevent the SSD from entering deep sleep states.

# Disable APST on Linux
echo 0 | sudo tee /sys/module/nvme_core/parameters/default_ps_max_latency_us

NAND flash stores data in cells, with each cell holding one or more bits. More bits per cell Increases density but reduces endurance and performance.

Modern “3D NAND” stacks memory cells vertically (64, 128, or 176 layers), increasing density without Shrinking the cell size. This improves endurance compared to planar NAND at the same technology Node.

Most TLC and QLC SSDs implement an SLC cache — a portion of the NAND operates in pseudo-SLC mode (one bit per cell) to boost write performance. When the SLC cache is full, write speeds drop Dramatically as data must be folded from SLC into the TLC/QLC area.

:::caution QLC SSDs with full SLC caches can experience catastrophic write speed drops — from 5,000 MB/s to under 200 MB/s. This is a fundamental limitation of QLC NAND, not a defect. Avoid QLC SSDs For write-heavy workloads (video editing, database, OS drive). :::

SSD controllers implement wear leveling to distribute write operations evenly across all NAND Blocks. Two approaches exist:

• Dynamic wear leveling: Only moves data that is actively being updated. Free blocks are preferentially written to the least-worn physical block.
• Static wear leveling: Also moves cold (rarely accessed) data from low-wear blocks to high-wear blocks, ensuring all blocks wear evenly. More effective but higher write amplification.

NAND flash cannot overwrite data in place — a block must be erased before it can be written. Erase Operations happen at the block level ( 4–8 MB), while writes happen at the page level ( 4–16 KB). This mismatch necessitates garbage collection:

1. When a page is invalidated (overwritten or deleted), it is marked as stale.
2. When the number of stale pages in a block exceeds a threshold, the controller copies the valid pages to a new block and erases the old block.
3. This process is called “garbage collection” and causes write amplification. The physical write count exceeds the logical write count.

Write amplification factor (WAF) is the ratio of physical writes to logical writes:

A WAF of 1.0 means no amplification. In practice, WAF is 1.2–3.0 depending on the workload And the amount of over-provisioning.

TRIM is a SATA/NVMe command that tells the SSD which LBAs (Logical Block Addresses) are no longer in Use. Without TRIM, the SSD treats all previously written LBAs as valid data and must copy them During garbage collection, even if the OS has deleted the files. TRIM allows the SSD to skip copying Deleted data, improving garbage collection efficiency and maintaining write performance.

# Check if TRIM is supported
sudo hdparm -I /dev/nvme0 | grep "TRIM supported"

# Verify TRIM is active on ext4/xfs
lsblk -D
# Discard column should show "0B" or the device supports it

# Manually TRIM all mounted filesystems
sudo fstrim -av

# Enable periodic TRIM (weekly) with systemd
sudo systemctl enable fstrim.timer
sudo systemctl start fstrim.timer

:::caution On ZFS, do not use fstrim. ZFS handles discard internally and the autotrim pool Property controls TRIM behavior. :::

Over-provisioning (OP) reserves a portion of the NAND capacity for the SSD controller”s use. This Reserved space is not accessible to the host but provides:

1. More spare blocks for garbage collection, reducing write amplification.
2. Better wear leveling, because more blocks are available to distribute writes across.
3. Sustained write performance, because the SLC cache and garbage collection have more room to work.

Consumer SSDs have 7% OP built in. Enterprise SSDs have 28% or more. Some consumer SSDs (e.g., Samsung 840 EVO) allowed you to manually increase OP by shrinking the user-accessible Partition.

Traditional RAID 5/6 has a “write hole” vulnerability: if power is lost during a stripe write, the Parity may be inconsistent with the data, leading to silent data corruption. Hardware RAID cards With battery-backed write cache (BBWC) or ZFS’s copy-on-write transaction model address this.

ZFS eliminates many traditional RAID problems:

• No write hole (copy-on-write transactions are always consistent)
• No RAID rebuild degradation (resilver prioritizes data, not block order)
• End-to-end checksumming detects silent corruption
• Self-healing repairs corrupted data from parity/mirror copies
• Scrubbing proactively verifies all data integrity

:::caution Never use hardware RAID with ZFS. ZFS needs direct access to individual disks to manage The storage pool. Hardware RAID hides the disks behind a virtual block device, which prevents ZFS From performing its error detection and correction. :::

Modern Linux (kernel 5.0+) uses multi-queue block layer (blk-mq) I/O schedulers:

For NVMe SSDs, none is the best choice because the SSD’s internal controller already has Sophisticated queuing and scheduling logic. Adding a software scheduler on top introduces Unnecessary overhead.

For SATA SSDs, mq-deadline provides good latency guarantees without sacrificing throughput.

For HDDs (or HDD-based arrays), bfq provides the best interactive responsiveness by preventing Large sequential transfers from starving small random I/O.

# View current scheduler
cat /sys/block/nvme0n1/queue/scheduler

# Change scheduler (temporary)
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# Persistent change via udev rule
# /etc/udev/rules.d/60-scheduler.rules
# ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"

The block layer queue depth determines how many I/O requests can be in flight simultaneously:

# View current queue depth
cat /sys/block/nvme0n1/queue/nr_requests

# Set queue depth (temporary)
echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests

For NVMe SSDs, increasing the queue depth can improve throughput for multi-threaded workloads. The Optimal value depends on the SSD’s internal queue depth and the workload’s concurrency. Values of 256–1024 are typical for NVMe.

Read-ahead prefetches data into the page cache before it is requested, improving sequential read Performance but wasting memory for random workloads.

# View read-ahead size (in 512-byte sectors)
cat /sys/block/nvme0n1/queue/read_ahead_kb

# Set read-ahead to 128 KB (good for sequential workloads)
echo 128 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

# Disable read-ahead (for random I/O workloads)
echo 0 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

# ext4 with SSD optimizations
/dev/nvme0n1p2 / ext4 noatime,discard,errors=remount-ro 0 1

# xfs with SSD optimizations
/dev/nvme0n1p2 / xfs noatime,discard 0 0

# Key options:
# noatime  — Don't update file access times (reduces writes)
# discard  — Enable continuous TRIM (or use fstrim.timer)
# nodiratime — Don't update directory access times

For ZFS on SSD, key tunables include:

• ashift=12 or ashift=13 (4K or 8K sector size — always match the SSD’s physical sector size)
• primarycache=all (default — use ARC for caching)
• compression=lz4 (default — reduces writes and improves performance for compressible data)
• atime=off (reduces metadata writes)
• recordsize=128K for media files, recordsize=16K or 8K for databases

SMART (Self-Monitoring, Analysis, and Reporting Technology) provides predictive failure information For storage devices.

# Install smartmontools
sudo apt install smartmontools

# View SMART health
sudo smartctl -a /dev/nvme0n1

# View SMART summary
sudo smartctl -H /dev/nvme0n1

# Run a short self-test
sudo smartctl -t short /dev/nvme0n1

# Run a long self-test
sudo smartctl -t long /dev/nvme0n1

# View test results
sudo smartctl -l selftest /dev/nvme0n1

# Enable smartd daemon
sudo systemctl enable smartd
sudo systemctl start smartd

# Configure smartd (/etc/smartd.conf)
# Monitor all drives and send email on failure
DEVICESCAN -m admin@example.com -M exec /usr/share/smartmontools/smartd-runner

The risk of a second drive failure during rebuild increases with drive capacity and count. With 12 TB+ drives, a RAID 5 rebuild can take 24–72 hours, during which a second drive failure (or Unreadable sectors on another drive) causes complete data loss. Use RAID 6 (dual parity) or RAIDZ2/Z3 for arrays with drives larger than 4 TB.

Without TRIM, SSD performance degrades over time as the garbage collector must process stale data That the OS has already deleted. This can cause write speeds to drop by 50–80% over weeks or months. Enable TRIM either continuously (discard mount option) or periodically (fstrim.timer).

QLC NAND has 10–100x lower write endurance than TLC. A QLC SSD rated for 400 TBW may reach its Endurance limit in months under heavy write workloads (e.g., VM images, database logs, video editing Scratch). Check the TBW (Terabytes Written) rating and compare it against your expected annual write Volume.

NVMe SSDs throttle aggressively when they overheat. Consumer NVMe SSDs throttle at 70–80 °C. Under sustained write workloads (e.g., cloning a drive, large file transfers), the SSD can hit Thermal throttling within seconds. Ensure the M.2 slot has a heatsink and adequate case airflow.

Many modern SSDs have a 512-byte logical sector size but a 4 KB or 8 KB physical sector size. Misalignment between the logical and physical sector boundaries (partition not aligned to 4 KB) Causes read-modify-write amplification. Always use partition tools that align to 1 MB boundaries (parted, gdisk) rather than older tools (fdisk in legacy mode).

NVMe has two command categories:

1. Admin Commands: Sent via the Admin Submission Queue (SQ). Used for controller management:

• Identify Controller (returns controller capabilities and configuration)
• Identify Namespace (returns namespace parameters)
• Get/Set Features (configure power states, interrupt coalescing, etc.)
• Namespace Management (create, delete, attach, detach)
• Firmware Commit (update controller firmware)
• Format NVM (secure erase)

2. I/O Commands: Sent via I/O Submission Queues. Used for data transfer:

• Read, Write (standard data commands)
• Compare (read and compare with host buffer)
• Write Uncorrectable (inject error for testing)
• Dataset Management (hints about data usage: read, write, deallocate)

graph LR
    A[Host Software] --> B[Submission Queue 0]
    A --> C[Submission Queue 1]
    A --> D[Submission Queue N]
    B --> E[Completion Queue 0]
    C --> F[Completion Queue 1]
    D --> G[Completion Queue N]
    E --> H[NVMe Controller]
    F --> H
    G --> H
    H --> I[NAND Flash]

Each Submission Queue (SQ) and Completion Queue (CQ) pair is associated with a processing core. This Eliminates the lock contention that plagues the single-queue AHCI model:

• SQ (Submission Queue): Ring buffer where the host posts commands. The host writes command entries to the tail of the queue and rings the doorbell register to notify the controller.
• CQ (Completion Queue): Ring buffer where the controller posts completions. The host polls or receives interrupts for completed commands.

Queue depth is configurable per queue, with a maximum of 65,535 entries per queue. Deeper queues Allow the SSD controller to reorder and optimize I/O more effectively.

# Detailed namespace information
nvme id-ns /dev/nvme0n1

# Key fields:
# nsze    — Namespace size (total logical blocks)
# ncap    — Namespace capacity (usable blocks)
# nuse    — Namespace utilization (used blocks)
# nlbaf   — Number of LBA formats supported
# flbas   — Current LBA format (data size + metadata size)
# dps     — Data protection (end-to-end protection type)
# nmc     — Namespace multi-path I/O and sharing capabilities

NVMe supports optional end-to-end data protection using protection information (PI) appended to each Logical block:

Type 1 is the most comprehensive and is recommended for enterprise workloads where data integrity is Critical.

SSD firmware updates can fix bugs, improve performance, and extend drive lifespan:

# Check current firmware version
nvme id-ctrl /dev/nvme0n1 | grep fr

# Samsung NVMe firmware update (using samsung_magician or nvme-cli)
# Intel NVMe firmware update (using intelmas or nvme-cli fw-download)
sudo nvme fw-download /dev/nvme0n1 --fw=/path/to/firmware.bin
sudo nvme fw-commit /dev/nvme0n1 --action=1  # 1 = apply immediately

# Check for firmware updates without applying
sudo nvme fw-download /dev/nvme0n1 --fw=/path/to/firmware.bin --save

• When the manufacturer releases a stability fix for your specific drive model
• When you experience unexpected behavior (drops to lower power states, intermittent timeouts)
• Before initial deployment of a new drive
• When a security vulnerability is disclosed in the firmware

:::caution Firmware updates are irreversible on most drives. A failed firmware update can brick the Drive. Ensure the update process is not interrupted (connect the drive to a UPS, close all Applications accessing the drive). :::

The mq-deadline scheduler maintains two sorted queues:

• Read queue: Sorted by request deadline (earliest first).
• Write queue: Sorted by request deadline (earliest first).

Each request is assigned a deadline based on its target sector:

The scheduler always dispatches the request with the earliest deadline. If a batch of reads or Writes accumulates, the scheduler alternates between read and write batches to prevent starvation:

• Maximum number of reads dispatched before switching to writes: 8 (configurable)
• Maximum number of writes dispatched before switching to reads: 8 (configurable)

BFQ (Budget Fair Queueing) assigns each process an I/O budget. A process can issue I/O until its Budget is exhausted, then it must wait for other processes to use their budgets:

• Budget: Measured in sectors served. Default is approximately 128 KB per budget slice.
• Weighting: Higher-priority processes get larger budgets (configurable via cgroups).
• Seek optimization: BFQ accounts for disk seek time when choosing the next request. Requests that are close to the current head position are dispatched first.

BFQ is the best choice for desktop systems where interactive responsiveness matters more than Throughput.

graph TD
    A{What type of storage?} -->|NVMe SSD| B{Workload?}
    A -->|SATA SSD| C{Workload?}
    A -->|HDD| D{Workload?}
    B -->|General| E[none/no-op]
    B -->|Latency-sensitive| F[mq-deadline]
    C -->|General| G[mq-deadline]
    C -->|Desktop/interactive| H[bfq]
    D -->|Desktop/interactive| H
    D -->|Server/sequential| I[mq-deadline or bfq]

The block layer can merge adjacent I/O requests to reduce per-request overhead. However, excessive Merging can increase latency for individual requests:

# View current merge settings
cat /sys/block/nvme0n1/queue/nomerges

# Values:
# 0 — Merge all types
# 1 — Merge only simple adjacent requests
# 2 — Merge all types including cross-queue merges
# 2 — No merging

For low-latency workloads (databases), disabling merges (nomerges=2) can reduce latency at the Cost of higher command overhead.

# Block layer request queue depth
cat /sys/block/nvme0n1/queue/nr_requests
# Default: 128. Increase to 256-1024 for NVMe SSDs.

# Scheduler quantum (number of requests dispatched per round-robin cycle)
cat /sys/block/nvme0n1/queue/scheduler_quantum
# Default: 8. Increase for throughput-oriented workloads.

# Write Same optimization (writes the same data to multiple blocks)
cat /sys/block/nvme0n1/queue/write_same_max_bytes

# Discard zeroes data (does a discard return zeroes?)
cat /sys/block/nvme0n1/queue/discard_zeroes_data

These parameters affect how the kernel handles TRIM/discard commands and block-level write Optimizations.

# Mount options for ext4 on NVMe SSD
/dev/nvme0n1p2 / ext4 noatime,discard,errors=remount-ro,commit=60,barrier=1 0 1

# Key options:
# commit=60    — Flush data to disk every 60 seconds (default is 5)
# barrier=1    — Enable write barriers (safe, slight overhead)
# journal_opts=journal_async_commit — Asynchronous journal commits (faster but slightly less safe)

# Mount options for XFS on NVMe SSD
/dev/nvme0n1p2 / xfs noatime,discard,allocsize=64m,inode64 0 0

# Key options:
# allocsize=64m — Delayed allocation size (larger = better sequential write performance)
# inode64     — Allow inode allocation across the entire filesystem (not just the first 1 TB)
# logbufs=8   — Increase log buffer count (default is 2, useful for metadata-heavy workloads)
# logbsize=256k — Increase log buffer size

# Mount options for BTRFS on SSD
/dev/nvme0n1p2 / btrfs noatime,ssd,discard=async,compress=zstd:1,space_cache=v2 0 0

# Key options:
# ssd           — Enable SSD-specific optimizations (reduced seek cost model)
# discard=async — Background discard (better than continuous discard for SSDs)
# space_cache=v2 — Free space tree (more efficient than v1 for large filesystems)
# compress=zstd:1 — Lightweight compression (fast, saves space without significant CPU cost)

# Database simulation (random read/write, 4K blocks)
fio --name=db-test --ioengine=libaio --iodepth=64 --rw=randrw \
    --rwmixread=70 --bs=4k --direct=1 --size=4G --numjobs=4 \
    --runtime=300 --group_reporting --output-format=json

# Web server simulation (random read, 4K-16K blocks)
fio --name=web-test --ioengine=libaio --iodepth=32 --rw=randread \
    --bs=4k --direct=1 --size=2G --numjobs=8 \
    --runtime=300 --group_reporting

# Media streaming (sequential read, 128K blocks)
fio --name=media-test --ioengine=libaio --iodepth=32 --rw=read \
    --bs=128k --direct=1 --size=16G --numjobs=1 \
    --runtime=300 --group_reporting

# Write endurance test (sequential write, 1M blocks)
fio --name=endurance-test --ioengine=libaio --iodepth=32 --rw=write \
    --bs=1m --direct=1 --size=32G --numjobs=1 \
    --runtime=3600 --group_reporting

Key metrics to analyze from fio JSON output:

Every storage medium has a specified UBER — the probability of an unrecoverable bit error:

While these numbers seem reassuring, they compound in large-scale deployments:

This is why ZFS checksumming is essential — it detects and corrects these errors that would Otherwise cause silent data corruption.

Wear leveling effectiveness determines SSD lifespan:

For a 2 TB TLC SSD with 3,000 P/E cycles and a sustained write rate of 50 GB/day:

In practice, write amplification (WAF 1.2–3.0) and real-world write patterns reduce this Significantly. A more realistic estimate with WAF of 2.0:

Modern TLC SSDs are extremely durable for typical workloads. QLC SSDs (100–1,000 P/E cycles) are the Concern — at 500 P/E cycles and 50 GB/day with WAF 2.0:

Still long for typical desktop use, but write-heavy workloads (video editing, VM images) can Significantly reduce this.

Intel Optane DC P5800X and P4800X drives are the gold standard for ZFS SLOG devices due to their Consistent low latency regardless of workload:

DWPD (Drive Writes Per Day) measures endurance relative to capacity. A 100 DWPD drive can be written To 100 times its capacity every day for 5 years.

The optimal L2ARC size depends on the ARC size and the working set:

• Minimum useful L2ARC size: Equal to the ARC size. Smaller L2ARC devices provide minimal benefit because the metadata overhead consumes too much of the available space.
• Recommended L2ARC size: 3–10x the ARC size. This provides enough capacity for the L2ARC to store a meaningful portion of the working set that overflows from the ARC.
• L2ARC for SSD pools: Generally not recommended. The pool SSDs already provide low-latency access. L2ARC adds cost and complexity without significant benefit.

L2ARC metadata is stored in the ARC, consuming RAM proportional to the number of L2ARC entries:

For a 1 TB L2ARC with 4 KB average block size, this is approximately 17.5 GB of ARC metadata. Ensure You have sufficient RAM to accommodate both the ARC and L2ARC metadata.

graph TD
    A[Client I/O] --> B{Read or Write?}
    B -->|Write| C[Write to NVMe Pool]
    B -->|Read| D{In ARC?}
    D -->|Yes| E[Return from ARC]
    D -->|No| F{In L2ARC?}
    F -->|Yes| G[Return from L2ARC]
    F -->|No| H[Read from Pool]
    C --> I[Snapshot and Replicate]
    I --> J[HDD Archive Pool]

Hot tier (NVMe SSD): Active working data. Databases, VM images, frequently accessed files.

Warm tier (SATA SSD): Recently accessed data. Media libraries, documents, infrequently used VMs.

Cold tier (HDD): Archive data. Backups, long-term storage, rarely accessed files.

ZFS does not have native automatic tiering. You can implement manual tiering with:

1. Separate pools for each tier with different storage devices.
2. Periodic scripts that move data between tiers based on access patterns (using zfs send and zfs recv).
3. L2ARC as a read cache for the warm tier, backed by the hot tier.
4. ZFS special vdevs for metadata, storing metadata on fast storage while data lives on slower storage.

# Check APST status
cat /sys/module/nvme_core/parameters/default_ps_max_latency_us

# Disable APST (set to 0)
echo 0 | sudo tee /sys/module/nvme_core/parameters/default_ps_max_latency_us

# Enable APST with a latency target (in microseconds)
# 200000 us = 200 ms (moderate aggressiveness)
echo 200000 | sudo tee /sys/module/nvme_core/parameters/default_ps_max_latency_us

# Set per-device APST
echo 0 | sudo tee /sys/class/nvme/nvme0/power/pm_qos_latency_tolerance_us

# Set HDD standby timeout (in seconds, 0 = never)
hdparm -S 60 /dev/sda  # Standby after 60 seconds of inactivity
hdparm -y /dev/sda     # Immediately enter standby

# APM (Advanced Power Management) level
hdparm -B 127 /dev/sda  # 1 (aggressive) to 255 (disabled)

:::caution Frequent HDD spin-up/spin-down cycles increase wear. Set standby timeout to a reasonable Value (15–30 minutes) rather than a short interval.

This topic covers the core concepts of storage tuning, including underlying theory, practical implementation, and key applications.

Key concepts include:

• relational databases and SQL
• normalisation (1NF, 2NF, 3NF)
• entity-relationship diagrams
• transaction processing (ACID)
• NoSQL and distributed databases

Understanding these concepts thoroughly is essential for both examinations and practical programming, and requires both theoretical knowledge and hands-on practice.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.

:::