Backup and Replication

The 3-2-1 rule is the minimum standard for data protection:

• 3 copies of your data
• 2 different storage media (e.g., HDD and cloud)
• 1 copy offsite (e.g., cloud, another building, another geographic region)

ZFS makes this rule easy to implement:

1. Primary copy: Your live ZFS pool.
2. Secondary copy: A second ZFS pool (on the same NAS or a second NAS) via snapshot and send/receive.
3. Tertiary copy (offsite): Cloud storage via TrueNAS Cloud Sync or a remote NAS via ZFS replication.

:::caution The 3-2-1 rule is a minimum, not a maximum. For critical data, consider extending to 3-2-1-1-0: 3 copies, 2 media, 1 offsite, 1 air-gapped (disconnected), 0 errors (verified restores). :::

graph LR
    A[Create Snapshot] --> B[Accumulate Changed Blocks]
    B --> C[Replicate to Remote]
    C --> D[Prune Old Snapshots]
    D --> E[Verify Integrity]

TrueNAS provides built-in snapshot task scheduling with configurable retention:

Snapshots only consume space when data is modified or deleted after the snapshot is taken. Monitor Snapshot space usage:

# List all snapshots with space usage
zfs list -t snapshot -o name,used,refer,written

# Check how much space snapshots are using on a dataset
zfs list -o name,used,usedbysnapshots,usedbydataset,usedbyrefreservation tank/data

# Find the largest snapshots
zfs list -t snapshot -S used -o name,used | head -20

Snapshots can be destroyed to reclaim space, but you cannot destroy an individual snapshot if it has Clones. Destroy all clones first, or use zfs destroy -R to recursively destroy a snapshot and all Its dependents.

# Destroy a specific snapshot
zfs destroy tank/data@daily.2024-01-01

# Destroy all snapshots older than 90 days (using zfs-auto-snapshot naming)
zfs list -t snapshot -o name -s creation | \
  awk "/tank\/data@autosnap_daily/ && $1 < "tank/data@autosnap_daily_$(date -d '90 days ago' +%Y-%m-%d)"' | \
  xargs -n1 zfs destroy

ZFS send/receive is the native mechanism for creating exact copies of datasets. It works at the Block level, transferring only the changed blocks.

# Initial full replication
zfs send -Rv tank/data@snapshot1 | ssh remote-nas zfs recv -F backup/data

# Incremental replication
zfs send -Rvi tank/data@snapshot1 tank/data@snapshot2 | \
  ssh remote-nas zfs recv -F backup/data

# Encrypted replication over SSH
zfs send -Rcv tank/data@snapshot2 | \
  ssh -c aes128-ctr remote-nas zfs recv -F backup/data

For bandwidth-constrained network links, use pv or trickle to limit transfer rate:

# Limit transfer to 50 MB/s using pv
zfs send -Rv tank/data@snapshot2 | \
  pv -rat -b -s $(zfs send -nPv tank/data@snapshot2 | tail -1 | awk '{print $NF}') | \
  ssh remote-nas zfs recv -F backup/data

TrueNAS Cloud Sync supports:

1. Navigate to Data Protection → Cloud Sync → Add.
2. Select the source (local dataset or snapshot).
3. Select the cloud provider and configure credentials.
4. Choose the transfer mode:

• Sync: One-way mirror from local to cloud.
• Move: Transfer to cloud and delete local copies.

5. Set the schedule (real-time, hourly, daily).
6. Configure snapshot retention on the cloud side.

• Egress costs: Most cloud providers charge for data egress (download). Backblaze B2 and Wasabi are exceptions with no egress fees.
• Upload bandwidth: Uploading to cloud is limited by your ISP’s upload speed. A 1 TB backup over a 50 Mbps upload connection takes ~48 hours.
• Encryption: TrueNAS can encrypt data before uploading (client-side encryption), ensuring the cloud provider cannot read your data. Configure this under “Encryption” in the Cloud Sync task.
• Versioning: Enable cloud bucket versioning to protect against accidental deletion or ransomware.

A backup that has never been tested is not a backup — it is a hope. Regularly test your backup Restore procedure:

1. Monthly: Restore a random subset of files from the most recent backup and verify integrity.
2. Quarterly: Perform a full dataset restore to a test environment and validate application functionality.
3. Annually: Test a bare-metal restore (pool recovery from replicated snapshots).

# Test restore from a snapshot
zfs clone tank/data@daily.2024-01-15 tank/data-test-restore
ls -la /mnt/tank/data-test-restore
# Verify file contents, permissions, timestamps
zfs destroy tank/data-test-restore

Set up monitoring and alerting to detect backup failures:

1. Snapshot task failures: TrueNAS sends alerts when snapshot tasks fail. Configure email notifications under System → Alert Settings.
2. Replication failures: Monitor the replication task status and ensure the lag time between source and destination is within your RPO target.
3. Cloud sync failures: Cloud sync tasks can fail due to credential expiration, network issues, or quota limits. Set up alerting for these failures.
4. Storage capacity: Monitor both local and remote backup storage capacity. A full backup destination is as bad as no backup.

RPO defines the maximum acceptable data loss measured in time. If your RPO is 1 hour, your backup Strategy must ensure that no more than 1 hour of data can be lost.

RTO defines the maximum acceptable downtime after a disaster. If your RTO is 4 hours, you must be Able to restore service within 4 hours.

TrueNAS supports snapshot retention policies that prevent snapshot deletion within a configured time Window. This protects against ransomware that attempts to encrypt files and delete snapshots:

1. Configure a snapshot task with a retention period that exceeds your recovery window.
2. TrueNAS SCALE supports “protected” snapshots that cannot be deleted manually within the retention period.
3. For maximum protection, replicate snapshots to a separate system where the replication destination has its own snapshot retention policy.

1. Immutable snapshots: Prevent snapshot deletion.
2. Offsite replication: Even if the primary system is compromised, the offsite copy is protected.
3. Air-gapped backup: Periodically create a backup that is disconnected from the network.
4. User education: Train users on phishing and suspicious attachments.
5. Network segmentation: Limit access to the NAS from untrusted networks.

TrueNAS supports dataset-level encryption using AES-256-GCM:

1. Create an encrypted dataset or pool.
2. The encryption key is protected by a passphrase or a key file.
3. Without the key, the data is unreadable. Even if the drives are physically stolen.

# Create an encrypted dataset
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt \
    tank/encrypted-data

# Mount the encrypted dataset (requires the passphrase)
zfs mount -l tank/encrypted-data

• Performance impact: AES-NI hardware acceleration makes the overhead negligible on modern CPUs ( 1–3%).
• Key management: You must securely store the encryption key/passphrase. Losing the key means losing the data permanently. Store keys in a password manager, hardware security module, or offline location.
• Send/receive: Encrypted datasets can be sent with raw mode (-w), preserving encryption without needing to decrypt and re-encrypt.
• Backup implications: If you replicate an encrypted dataset to an untrusted location, the destination cannot read the data without the key.

The most common backup failure is not testing the restore procedure. Backups can fail silently (corrupt snapshots, incomplete transfers, credential expiration) and you will not discover the Problem until you need to restore. Test restores monthly at minimum.

A single backup copy (even if it is offsite) is vulnerable to the same failure that destroyed the Primary (ransomware, fire, flood). Always maintain at least two independent backup copies on Different media.

Large backups over slow network links can take longer than the backup window allows, causing backup Tasks to overlap and potentially fail. Calculate your backup size and network bandwidth to ensure Backups complete within the window:

Where efficiency accounts for compression and deduplication ( 0.5–0.8 for compressed data).

If your encryption keys are compromised, all data encrypted with those keys is at risk. Implement a Key rotation policy (e.g., annually) and ensure you can re-encrypt data with new keys. ZFS does not Natively support key rotation on existing datasets — you must create a new encrypted dataset and Copy the data.

Creating too many snapshots (e.g., every 5 minutes without pruning) can consume all available pool Space. Always configure retention policies that limit the total number of snapshots per dataset. Monitor snapshot space usage with zfs list -o name,usedbysnapshots.

The most storage-efficient backup strategy combines periodic full backups with frequent incremental Backups:

graph TD
    A[Sunday: Full Backup] --> B[Monday: Incremental]
    B --> C[Tuesday: Incremental]
    C --> D[Wednesday: Incremental]
    D --> E[Thursday: Incremental]
    E --> F[Friday: Incremental]
    F --> G[Saturday: Incremental]
    G --> H[Sunday: Full Backup]
    H --> I[Monday: Incremental]

Storage calculation:

For a 1 TB dataset with 5% daily change rate:

A synthetic full backup constructs a full backup from the last full backup and all subsequent Incrementals, without reading the source data again. This reduces the load on the production system:

1. Initial full backup: Read all data from source.
2. Daily incrementals: Read only changed blocks from source.
3. Weekly synthetic full: Construct full backup from incremental chain on the backup destination.

TrueNAS supports synthetic full backups through its replication task configuration.

Calculate the time required for each backup type:

Where Effective_Bandwidth accounts for compression, deduplication, and network overhead ( 50–80% of raw bandwidth).

# TrueNAS snapshot task configuration (via web UI):
# Navigate to Data Protection → Snapshot Tasks → Add
#
# For daily snapshots with 2-week retention:
#   Dataset: tank/data
#   Schedule: Daily at 00:00
#   Retention: Keep 14 snapshots with naming convention: auto-daily-%Y%m%d-%H%M%S
#
# For weekly snapshots with 3-month retention:
#   Dataset: tank/data
#   Schedule: Weekly on Sunday at 02:00
#   Retention: Keep 12 snapshots with naming convention: auto-weekly-%Y%m%d-%H%M%S
#
# For monthly snapshots with 1-year retention:
#   Dataset: tank/data
#   Schedule: Monthly on the 1st at 03:00
#   Retention: Keep 12 snapshots with naming convention: auto-monthly-%Y%m%d-%H%M%S

# Check snapshot space usage per dataset
zfs list -o name,used,usedbysnapshots,usedbydataset -r tank

# Find datasets where snapshots use more than 10% of total space
zfs list -o name,used,usedbysnapshots -r tank | awk '$3 > 0.1 * $2 {print}'

# Destroy snapshots matching a pattern (older than 30 days)
zfs list -t snapshot -o name -s creation | \
  awk '/tank\/data@auto-daily-/ && substr($1, length($1)-7) < strftime("%Y%m%d", systime()-30*86400)' | \
  xargs -n1 zfs destroy

# Or use TrueNAS's built-in snapshot task with retention policy

# TrueNAS Cloud Sync configuration for AWS S3:
# Provider: Amazon S3
# Bucket: my-nas-backup
# Region: us-east-1
# Access Key: AKIA...
# Secret Key: ...
# Folder: /tank/data/
# Transfer Mode: SYNC (one-way mirror)
# Schedule: Daily at 04:00
# Snapshot: Include latest snapshot
# Encryption: AES-256 (client-side)
# Compression: Enabled (gzip)

Backblaze B2 is a cost-effective alternative to AWS S3 for backup storage:

• No egress fees: Download data for free.
• Storage cost: Approximately USD 5/TB/month.
• Download cost: Free.
• API compatible with S3 (use S3-compatible provider in TrueNAS).

1. Enable client-side encryption and compression: Reduces upload data volume.
2. Schedule during off-peak hours: Minimize impact on production network.
3. Use multipart uploads: Large files are uploaded in parts, improving reliability.
4. Monitor transfer logs: Check for errors, timeouts, or throttling.

1. Assess the disaster. What was lost? Pool, server, site?
2. Verify backups are intact. Log into the backup destination and verify recent snapshots exist and are readable.
3. Prioritize recovery. Restore critical data first, then less critical data.
4. Test the restore. Restore a sample of data and verify integrity.
5. Document the recovery. Record what was restored, what was lost, and the timeline.

If the entire TrueNAS server is lost (fire, flood, theft):

1. Procure replacement hardware. Match the original specifications if possible.
2. Install TrueNAS on the new hardware.
3. Import the remote replica pool. If using ZFS replication, the remote pool can be imported directly.
4. Configure services. Restore SMB shares, NFS exports, users, and permissions from the configuration backup.
5. Verify data integrity. Run a scrub on the imported pool.
6. Restore any data not in the replication. Use cloud sync or offline backups.

TrueNAS SCALE supports ZFS native encryption:

# Create an encrypted dataset
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt \
    -o compression=zstd tank/encrypted

# Mount (requires passphrase)
zfs mount -l tank/encrypted

# Change passphrase
zfs change-key tank/encrypted

# Check encryption status
zfs get encryption tank/encrypted

Encryption keys can be managed in several ways:

Store encryption keys in a secure, offsite location:

1. Password manager: Store the passphrase in a password manager (Bitwarden, 1Password).
2. Physical copy: Write the passphrase on paper and store in a safe deposit box.
3. Key escrow service: Some password managers offer key escrow for trusted contacts.

:::danger If you lose the encryption key, all data on the encrypted dataset is permanently Irrecoverable. There is no backdoor. Always have a verified backup of the key. :::

Configure alerts under System → Alert Settings → Advanced:

Create a Grafana dashboard to monitor backup health:

• Panel 1: Replication lag (hours since last successful replication)
• Panel 2: Snapshot count per dataset (line chart over time)
• Panel 3: Snapshot space usage per dataset (stacked area chart)
• Panel 4: Cloud sync status (success/failure count over time)
• Panel 5: Pool capacity gauge
• Panel 6: SMART health status table

#!/bin/bash
# Verify backup integrity by comparing source and destination checksums
# Run weekly as a cron job

SOURCE="tank/data"
DEST="backup/data"
SNAPSHOT="verify-$(date +%Y%m%d)"

# Create a snapshot of the source
zfs snapshot ${SOURCE}@${SNAPSHOT}

# Send to backup destination (dry run to compare)
zfs send -Rnv ${SOURCE}@${SNAPSHOT} > /tmp/verify_output.txt

# Check the output for any errors
if grep -q "error\|failed\|corrupt" /tmp/verify_output.txt; then
    echo "BACKUP VERIFICATION FAILED" | mail -s "Backup Alert" admin@example.com
else
    echo "BACKUP VERIFICATION PASSED" | mail -s "Backup OK" admin@example.com
fi

# Clean up verification snapshot
zfs destroy ${SOURCE}@${SNAPSHOT}

ZFS incremental send/receive creates a chain of snapshots where each new snapshot contains only the Changes since the previous one. Managing these chains correctly is critical for both efficiency and Recoverability.

# Full send (baseline)
zfs send tank/data@base | zfs recv backup/data

# Incremental send (changes between base and snap1)
zfs send -i tank/data@base tank/data@snap1 | zfs recv backup/data

# Another incremental (changes between snap1 and snap2)
zfs send -i tank/data@snap1 tank/data@snap2 | zfs recv backup/data

The receiving side must have the base snapshot (@base) and all intermediate snapshots to apply an Incremental send. If any intermediate snapshot is missing, the receive fails.

A broken chain occurs when an intermediate snapshot is destroyed on either side. To recover:

# Scenario: @snap1 was destroyed on the backup, breaking the chain
# Option 1: Find the last common snapshot and do a new full send
LAST_COMMON=$(zfs list -t snapshot -o name -s creation backup/data | tail -1)
echo "Last common snapshot: $LAST_COMMON"

# Option 2: Use zfs send -R to send all snapshots including intermediates
# This re-creates the full history on the receiving side
zfs send -R tank/data@snap2 | zfs recv -F backup/data

When using ZFS deduplication (not recommended for most workloads due to RAM overhead), incremental Sends automatically deduplicate. However, the dedup table must be present on both sides:

# Check dedup table size (warning: can be enormous)
zpool get dedupratio tank
zdb -D tank 2>/dev/null | head -5

# If dedup ratio is close to 1.00x, dedup is not saving space but still consuming RAM
# Consider disabling dedup: zfs set dedup=off tank/data

TrueNAS and OpenZFS 2.0+ support resumable send/receive, which is critical for large datasets where Network interruptions are possible:

# Send with resume token (TrueNAS UI handles this automatically for replication tasks)
# From CLI:
zfs send -v tank/data@snap1 | zfs recv -s backup/data

# If interrupted, the receive side generates a token
# Check for resume tokens:
zfs get receive_resume_token backup/data

# Resume the transfer using the token
zfs recv -s backup/data <<< "$(zfs get -H -o value receive_resume_token backup/data)"

:::tip Resume tokens expire after approximately 5 minutes of inactivity in some implementations. For Very large transfers over unreliable networks, consider using mbuffer as a network buffer to Absorb short interruptions:

# Sender side
zfs send tank/data@snap1 | mbuffer -W 300 -s 128k -m 1G 10.0.0.20:9090

# Receiver side
mbuffer -s 128k -m 1G -I 9090 | zfs recv backup/data

:::

A backup that has never been tested is not a backup. Establish a regular verification cadence to Catch silent corruption, configuration drift, and restoration procedure issues.

#!/bin/bash
# Run on the first Saturday of each month via cron

set -euo pipefail
LOG="/var/log/backup-verify-$(date +%Y-%m).log"
VERIFIED_COUNT=0
FAILED_COUNT=0

echo "=== Monthly Backup Verification: $(date) ===" | tee -a "$LOG"

# 1. Verify all replication tasks completed successfully
for task in $(midclt call replication.query | jq -r '.[].id'); do
  status=$(midclt call replication.get_instance "$task" | jq -r '.status')
  if [ "$status" != "SUCCESS" ]; then
    echo "FAIL: Replication task $task last status: $status" | tee -a "$LOG"
    FAILED_COUNT=$((FAILED_COUNT + 1))
  else
    echo "OK: Replication task $task" | tee -a "$LOG"
    VERIFIED_COUNT=$((VERIFIED_COUNT + 1))
  fi
done

# 2. Test restore of a random file from each major dataset
for dataset in tank/data tank/photos tank/documents; do
  # Pick a random file
  RANDOM_FILE=$(find /mnt/$dataset -type f | shuf -n 1)
  RELATIVE_PATH="${RANDOM_FILE#/mnt/}"

  # Find the backup location and verify the file exists there
  echo "Checking: $RELATIVE_PATH" | tee -a "$LOG"

  # Compute checksums on source and backup
  SOURCE_CKSUM=$(sha256sum "$RANDOM_FILE" | awk '{print $1}')
  BACKUP_FILE="/mnt/backup_pool/$RELATIVE_PATH"
  if [ -f "$BACKUP_FILE" ]; then
    BACKUP_CKSUM=$(sha256sum "$BACKUP_FILE" | awk '{print $1}')
    if [ "$SOURCE_CKSUM" = "$BACKUP_CKSUM" ]; then
      echo "OK: $RELATIVE_PATH checksums match" | tee -a "$LOG"
      VERIFIED_COUNT=$((VERIFIED_COUNT + 1))
    else
      echo "FAIL: $RELATIVE_PATH checksum MISMATCH" | tee -a "$LOG"
      FAILED_COUNT=$((FAILED_COUNT + 1))
    fi
  else
    echo "FAIL: $RELATIVE_PATH not found in backup" | tee -a "$LOG"
    FAILED_COUNT=$((FAILED_COUNT + 1))
  fi
done

# 3. Summary
echo "=== Results: $VERIFIED_COUNT passed, $FAILED_COUNT failed ===" | tee -a "$LOG"
if [ "$FAILED_COUNT" -gt 0 ]; then
  echo "BACKUP VERIFICATION FAILED" | mail -s "Monthly Backup FAIL" admin@example.com
  exit 1
fi

Every quarter, perform a full dataset restoration to a temporary location and verify application Integrity:

# 1. Receive a full backup to a test pool
zfs send -R tank/data@quarterly-test | zfs recv -o mountpoint=/mnt/test_restore test_pool/data

# 2. Run application-specific integrity checks
# For a database:
docker run --rm -v /mnt/test_restore/pgdata:/var/lib/postgresql/data \
  postgres:15 pg_verifybackup /var/lib/postgresql/data

# For a file server:
find /mnt/test_restore -type f -exec md5sum {} \; > /tmp/test_checksums.txt
# Compare against production checksums taken at snapshot time
diff /tmp/production_checksums.txt /tmp/test_checksums.txt

# 3. Measure restoration time for RTO validation
START=$(date +%s)
zfs send -R tank/data@quarterly-test | zfs recv -F test_pool/data_restore
END=$(date +%s)
echo "Full restoration time: $((END - START)) seconds"
# Compare against your RTO target

RPO defines the maximum acceptable data loss measured in time. If your RPO is 1 hour, you can afford To lose up to 1 hour of data.

# Calculate actual RPO from snapshot schedule
# Snapshots: hourly, replication: every 4 hours
# Worst case: replication fails right after snapshot, next successful replication is 4 hours later
# Actual RPO = snapshot_interval + replication_interval = 1h + 4h = 5 hours

# If business requires RPO of 1 hour:
# Option A: Replicate every hour (increases bandwidth usage)
# Option B: Use synchronous replication (requires low-latency link, <5ms)
# Option C: Use application-level replication (e.g., PostgreSQL streaming replication)

RTO defines the maximum acceptable downtime. If your RTO is 4 hours, the system must be fully Restored within 4 hours of a disaster.

graph LR
    A[Disaster Occurs] --> B[Detection: 15 min]
    B --> C[Failover Initiation: 30 min]
    C --> D[Data Restoration: 2-3 hours]
    D --> E[Verification: 30 min]
    E --> F[Production: RTO met]

    style A fill:#f66
    style F fill:#6f6

TrueNAS Cloud Sync tasks can fail silently if not properly monitored. Common failure modes include Authentication token expiration, rate limiting, and network timeouts.

# List all cloud sync tasks and their last run status
midclt call cloudsync.query | jq '.[] | {id, description, job: .job.name, state: .job.state}'

# Get detailed error information for a failed task
TASK_ID=1
midclt call cloudsync.get_instance "$TASK_ID" | jq '.'

# Check cloud sync logs
ls -la /var/log/cloudsync/
cat /var/log/cloudsync/cloudsync.log | tail -50

Cloud sync retention policies prevent cloud storage costs from growing unboundedly:

# Configure retention via CLI (TrueNAS SCALE)
midclt call cloudsync.update 1 '{
  "snapshot": true,
  "retention_policy": "CUSTOM",
  "lifetime": 90,
  "retention_count": 10
}'

Retention strategies:

Cloud providers impose API rate limits that can cause sync failures:

# AWS S3: 5,500 GET/HEAD requests per second per prefix
# Google Cloud Storage: 4,000 read operations per second
# Backblaze B2: varies by plan

# Reduce rate by:
# 1. Increasing transfer chunk size
# 2. Reducing concurrent connections
# 3. Syncing less frequently
# 4. Using multipart uploads for large files

# Configure chunk size and concurrency in TrueNAS
midclt call cloudsync.update 1 '{
  "transfer_threads": 4,
  "chunk_size": 524288000
}'

ZFS snapshots are inherently resistant to ransomware because they are read-only and cannot be Modified by compromised clients. However, additional measures are needed for comprehensive Protection.

TrueNAS supports holding snapshots to prevent even administrators from deleting them:

# Hold a snapshot (prevents deletion until released)
zfs hold keep_forever tank/data@snap1

# List holds on a dataset
zfs holds tank/data

# Release a hold (requires explicit admin action)
zfs release keep_forever tank/data@snap1

# Create a periodic hold script for critical datasets
#!/bin/bash
# Run weekly: hold the latest snapshot for 1 year
DATASETS="tank/data tank/photos tank/documents"
TAG="annual-hold-$(date +%Y)"

for ds in $DATASETS; do
  LATEST=$(zfs list -t snapshot -o name -s creation "$ds" | tail -1)
  zfs hold "$TAG" "$LATEST"
done

Monitor for sudden increases in file modification or deletion rates:

# Alert on rapid file changes (potential ransomware)
#!/bin/bash
THRESHOLD=1000  # files changed per minute
DATASET="tank/data"

while true; do
  CHANGES=$(zfs diff -FH "$DATASET@1min-ago" 2>/dev/null | wc -l)
  if [ "$CHANGES" -gt "$THRESHOLD" ]; then
    echo "ALERT: $CHANGES file changes detected in $DATASET in the last minute" \
      | mail -s "RANSOMWARE ALERT" admin@example.com
    # Optionally create an emergency snapshot before more damage occurs
    zfs snapshot "$DATASET@emergency-$(date +%Y%m%d-%H%M%S)"
  fi
  sleep 60
done

If ransomware is detected:

# 1. Immediately disconnect affected shares to stop the spread
midclt call smb.update '{"shares": [{"name": "infected-share", "enabled": false}]}'

# 2. Identify the last clean snapshot (before infection)
zfs list -t snapshot -o name,creation -s creation tank/data | tail -20

# 3. Roll back to the last clean snapshot
zfs rollback -r tank/data@clean-snapshot

# 4. Verify data integrity after rollback
find /mnt/tank/data -type f -mtime -1 | head -50  # Check recently modified files

When syncing backups to cloud storage, encrypt data before it leaves your network. TrueNAS supports Client-side encryption for cloud sync tasks.

# Create an encryption key (store this securely offline)
openssl rand -base64 32 > /root/cloud-sync-key.enc
chmod 400 /root/cloud-sync-key.enc

# Configure cloud sync to use encryption
midclt call cloudsync.update 1 '{
  "encryption": true,
  "encryption_key": "'"$(cat /root/cloud-sync-key.enc)"'",
  "encryption_cipher": "AES-256-GCM"
}'

:::caution If you lose the encryption key, all cloud backups become permanently unrecoverable. Store Encryption keys in multiple secure locations: a password manager, a hardware security key, and a Printed copy in a physical safe. Never store encryption keys alongside the backups themselves. :::

For environments subject to regulatory requirements (GDPR, HIPAA, SOC 2), document your backup and Encryption procedures:

• Data classification: Identify which datasets contain regulated data.
• Encryption at rest: Cloud storage providers encrypt at rest, but client-side encryption adds a layer of protection against provider-side breaches.
• Data residency: Some regulations require data to remain in specific geographic regions. Choose cloud providers with data centers in compliant regions.
• Retention policies: Regulations may mandate minimum or maximum retention periods. Configure snapshot and cloud sync retention accordingly.
• Audit trail: Enable logging for all backup and restoration operations.

# Enable audit logging for ZFS operations
# In /etc/sysctl.conf:
echo "vfs.zfs.zfs_events.class.include=\"all\"" > /etc/sysctl.d/99-zfs-audit.conf
sysctl -p /etc/sysctl.d/99-zfs-audit.conf

#!/bin/bash
# Run daily at 06:00 via cron

set -euo pipefail
ALERT_EMAIL="admin@example.com"
ERRORS=()

# Check 1: All datasets have recent snapshots
for ds in $(zfs list -o name -H | grep -v "^tank/boot$"); do
  NEWEST=$(zfs list -t snapshot -o creation -s creation "$ds" | tail -1)
  AGE_HOURS=$(( ($(date +%s) - $(date -d "$NEWEST" +%s)) / 3600 ))
  if [ "$AGE_HOURS" -gt 26 ]; then
    ERRORS+=("WARNING: $ds has no snapshot in ${AGE_HOURS}h")
  fi
done

# Check 2: Replication tasks are healthy
for task in $(midclt call replication.query | jq -r '.[].id'); do
  LAST_RUN=$(midclt call replication.get_instance "$task" | jq -r '.job.time_started')
  LAST_STATUS=$(midclt call replication.get_instance "$task" | jq -r '.job.state')
  if [ "$LAST_STATUS" != "SUCCESS" ]; then
    ERRORS+=("FAIL: Replication task $task status=$LAST_STATUS (last run: $LAST_RUN)")
  fi
done

# Check 3: Pool health
POOL_HEALTH=$(zpool status -x)
if echo "$POOL_HEALTH" | grep -q "DEGRADED\|FAULTED\|OFFLINE"; then
  ERRORS+=("CRITICAL: Pool health issue - $POOL_HEALTH")
fi

# Report
if [ ${#ERRORS[@]} -gt 0 ]; then
  printf "%s\n" "${ERRORS[@]}" | mail -s "BACKUP HEALTH ALERT" "$ALERT_EMAIL"
fi

This topic covers the essential concepts and techniques related to backup and replication, including key principles and practical applications.

Key concepts include:

• core concepts and definitions
• key principles and frameworks
• practical applications
• common techniques and methods
• evaluation and critical analysis

A thorough understanding of these concepts, combined with regular practice and review, is essential for mastery of this topic.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.