Monitoring and Alerting

The TrueNAS dashboard provides a real-time overview of system health:

Navigate to System → Advanced for detailed system information:

• System: Hostname, uptime, kernel version, platform
• Hardware: CPU model, RAM amount, PCI devices
• Network: Interface configuration, IP addresses, MAC addresses
• Storage: Pool status, disk information, SLOG/L2ARC status

1. Navigate to Data Protection → S.M.A.R.T. Tests → Add.
2. Select the disk type (All, HDD, SSD, NVMe).
3. Select the test type (Short or Long).
4. Set the schedule (Daily, Weekly, Monthly).
5. Save.

# Check SMART health
smartctl -H /dev/sda

# Full SMART attributes
smartctl -a /dev/sda

# Check self-test log
smartctl -l selftest /dev/sda

Critical attributes to monitor:

1. Navigate to Data Protection → Scrub Tasks → Add.
2. Select the pool to scrub.
3. Set the schedule (Monthly is standard for HDD pools; Weekly for SSD pools).
4. Set the threshold (minimum days between scrubs).
5. Enable or disable scrub when resilvering is in progress.

# Check scrub status
zpool status tank

# Example output:
# pool: tank
# status: scrub in progress since ...
# scan: scrub repaired 0 in 12h34m with 0 errors on ...

1. Navigate to System → Alert Settings.
2. Configure the email settings:

• SMTP server address and port
• Encryption (TLS/SSL)
• Authentication (username/password or app-specific password)
• From address
• To addresses (comma-separated)

3. Send a test email to verify the configuration.

TrueNAS classifies alerts into severity levels:

Configure alert rules for:

• Disk failures: Any drive with SMART status not “OK”
• Temperature: Disk or CPU temperature exceeding threshold
• Pool capacity: Pool usage exceeding 80% or 90%
• Scrub errors: Any errors found during scrub
• Replication lag: Replication destination falling behind source
• UPS events: Power failure, battery low, on battery

NUT provides UPS monitoring and automated shutdown for TrueNAS:

1. Connect the UPS to the TrueNAS server via USB.
2. Navigate to System → UPS → Add.
3. Configure the UPS settings:

• Identifier (e.g., ups)
• Driver (e.g., usbhid-ups for USB-connected UPS)
• Port (e.g., auto)
• Shutdown mode (UPS goes to battery, or low battery)

4. Configure the shutdown behavior:

• Minutes on battery before shutdown (e.g., 5 minutes)
• Load percentage threshold (e.g., shut down at 50% battery)

For environments with multiple systems on the same UPS:

1. Configure TrueNAS as the UPS master (connected to the UPS via USB).
2. Configure other systems as UPS slaves (connected to TrueNAS via the network).
3. TrueNAS notifies slave systems to shut down before shutting down itself.

# On TrueNAS (master), allow network access to NUT:
# Configure the UPS service to listen on the network interface
# Set up NUT users and passwords for slave systems

TrueNAS supports SNMPv2c and SNMPv3:

1. Navigate to System → Advanced → SNMP.
2. Enable SNMP.
3. Configure community string (SNMPv2c) or user credentials (SNMPv3).
4. Set the contact and location information.
5. Select which OIDs to expose (system, interfaces, storage, etc.).

TrueNAS SNMP exposes:

• System information (hostname, uptime, OS version)
• Interface statistics (bytes in/out, errors, drops)
• Storage pool information (capacity, health, I/O)
• Disk information (temperature, SMART health)
• CPU and memory utilization

Use the SNMP data with:

• Zabbix: Full-featured monitoring platform with TrueNAS templates.
• LibreNMS: Auto-discovering network monitoring.
• PRTG: Windows-based monitoring with SNMP sensors.
• Check_MK: Enterprise monitoring with native TrueNAS checks.

TrueNAS can export metrics to Prometheus for visualization in Grafana:

1. Deploy the Prometheus + Grafana stack from the TrueNAS app catalog.
2. Configure Prometheus to scrape TrueNAS metrics (via the node exporter or the TrueNAS API).
3. Import pre-built Grafana dashboards for TrueNAS.

Common dashboard panels for TrueNAS:

• Pool capacity over time (bar chart or gauge)
• I/O throughput per pool (time series)
• Disk temperature heatmap
• ARC hit ratio and size (time series)
• Network throughput per interface (time series)
• CPU and memory utilization (time series)
• SMART predictive failure indicators (table)

# System logs
cat /var/log/messages

# ZFS-related logs
dmesg | grep zfs

# SMB/CIFS logs
cat /var/log/samba4/log.smbd

# NFS logs
cat /var/log/messages | grep nfsd

# Boot logs
journalctl -b

# Check pool capacity and growth
zpool list -v
zfs list -o name,used,avail,refer,mountpoint

# Historical capacity tracking (via zpool history)
zpool history tank | grep create

Track capacity growth over time to predict when you will need to add storage. A pool that grows by 1 TB/month will fill a 20 TB pool in approximately 20 months (accounting for RAIDZ overhead).

Alert fatigue occurs when too many low-priority alerts desensitize administrators to critical ones.

Strategies:

1. Set appropriate thresholds: Do not alert on every minor event. Temperature warnings at 45 °C are noise; warnings at 55 °C are actionable.
2. Aggregate alerts: Group related alerts (e.g., “SMART errors on pool tank” rather than individual errors per disk).
3. Use escalation: Critical alerts go to immediate notification (email + SMS); warnings go to a daily digest.
4. Review and prune: Regularly review alert rules and remove or adjust ones that trigger frequently without action.
5. Acknowledge and track: Use a ticketing system to track alert acknowledgment and resolution.

A TrueNAS system without email alerts is a silent system. Disk failures, pool degradation, and other Critical events will go unnoticed until data is lost. Always configure email alerts and verify they Work with a test email.

SMART tests are predictive — they detect problems before they become failures. Running SMART tests Only when you suspect a problem defeats the purpose. Schedule regular short and long tests to catch Failures early.

A SMART predictive failure warning means the drive has a high probability of failing. Replace the Drive immediately — do not wait for it to fail completely. The longer you wait, the higher the risk Of a second drive failing before the resilver completes.

A UPS that is configured but never tested may fail when needed. Test the UPS shutdown procedure Quarterly:

1. Disconnect the UPS from mains power.
2. Verify TrueNAS detects the power loss.
3. Verify TrueNAS initiates shutdown at the configured threshold.
4. Verify other systems on the UPS also shut down.
5. Reconnect mains power and verify systems restart cleanly.

Collecting metrics without understanding what they mean leads to either panic (false alarms) or Complacency (missed warnings). Define clear thresholds for each metric, document them, and ensure The team understands what each alert means and what action to take.

The TrueNAS dashboard provides real-time metrics for:

TrueNAS stores historical metrics using RRDtool (Round Robin Database). Historical data is retained For approximately:

• 1-minute resolution: 24 hours
• 5-minute resolution: 7 days
• 1-hour resolution: 30 days

For longer retention or higher resolution, use external monitoring (Prometheus + Grafana).

The TrueNAS dashboard is customizable. Navigate to the dashboard and click the gear icon to:

1. Add widgets: CPU, memory, pool, network, disk, system temperature, uptime.
2. Rearrange widgets: Drag and drop to customize the layout.
3. Resize widgets: Some widgets support resizing for more detail.
4. Remove widgets: Remove widgets you do not need to reduce visual clutter.

Understanding SMART attributes in detail:

Reallocated Sector Count (ID 5):

• Count of sectors that have been reallocated due to read errors.
• Any increase indicates the drive is failing. Plan for immediate replacement.
• This is the single most important SMART attribute for HDDs.

Current Pending Sector Count (ID 197):

• Count of sectors that are unstable and awaiting reallocation.
• Non-zero value means the drive has detected potential bad sectors.
• If the count increases over time, the drive is deteriorating.

Uncorrectable Sector Count (ID 198):

• Count of sectors that could not be recovered after multiple read attempts.
• Any non-zero value means data has been lost. Replace the drive immediately.

Command Timeout (ID 188):

• Count of aborted operations due to timeout.
• Non-zero value can indicate cable issues, controller problems, or drive failure.

UDMA CRC Error Count (ID 199):

• Count of CRC errors on the UDMA interface.
• indicates a bad SATA/SAS cable or connector.
• Replace the cable before replacing the drive.

Media Wear Indicator (ID 173, SSD-specific):

• Percentage of rated endurance used.
• When this reaches 0%, the drive has reached its rated write endurance.
• Most drives continue to function beyond 0% but with increased risk.

Available Spare (ID 232, SSD-specific):

• Percentage of spare blocks remaining.
• When this drops below 10%, the drive is running out of spare blocks for wear leveling.
• Plan for replacement.

# View self-test log
sudo smartctl -l selftest /dev/sda

# Example output interpretation:
# Num  Test_Description    Status                  Remaining  LifeTime  LBA_of_first_error
# # 1  Extended offline    Completed: read failure  90%        1200h     123456789

# Status values:
# Completed without error    — Healthy
# Completed: read failure    — Bad sectors detected
# Aborted by host            — Test was interrupted
# Interrupted (host reset)    — System rebooted during test
# Not started                — Test has not begun

# Monitor temperature continuously
watch -n 5 "smartctl -A /dev/sda | grep -i temperature'

# Set up temperature alerting with smartd
# /etc/smartd.conf:
# /dev/sda -W 5,45,55 -R 5 -m admin@example.com
# -W: Monitor temperature, warn at 45°C, critical at 55°C
# -R: Report changes in raw attributes
# -m: Email alerts

During a scrub, ZFS reads every block in the pool and verifies its checksum:

1. Read the block and its stored checksum.
2. Compute the checksum of the read data.
3. Compare computed vs. Stored checksum.
4. If they match: data is intact. Continue.
5. If they do not match: a. Read the redundant copy (mirror) or recompute from parity (RAIDZ). B. Verify the corrected data. C. Write the corrected data back to the bad block. D. Log the error.

During a scrub of an HDD pool, normal I/O performance can degrade by 30–50%. Schedule scrubs during Off-peak hours.

# After a scrub completes, check the results
zpool status tank

# Example output:
# scan: scrub repaired 0 in 12h34m with 0 errors on Mon Jan 15 10:30
# scan: scrub in progress since Mon Jan 15 10:00
#       42.5% done, 0h47m to go
#       0 repaired, 0 unrepairable errors

# Key fields:
# repaired      — Number of blocks repaired from redundancy
# unrepairable — Number of blocks that could not be repaired (DATA LOSS)
# errors        — Total checksum errors found

If unrepairable is non-zero, you have experienced data corruption that could not be recovered from Redundancy. Identify which files were affected and restore from backup.

Gmail (App Password):

SMTP Server: smtp.gmail.com
Port: 465
Encryption: SSL/TLS
Username: your-email@gmail.com
Password: <app-specific-password>
From: your-email@gmail.com
To: admin@example.com

SendGrid:

SMTP Server: smtp.sendgrid.net
Port: 465
Encryption: SSL/TLS
Username: apikey
Password: <sendgrid-api-key>
From: notifications@example.com
To: admin@example.com

Mailgun:

SMTP Server: smtp.mailgun.org
Port: 587
Encryption: STARTTLS
Username: postmaster@mg.example.com
Password: <mailgun-password>
From: notifications@example.com
To: admin@example.com

For environments with multiple administrators, route alerts based on severity:

# Option 1: Use the TrueNAS app catalog
# Install "Prometheus Node Exporter" from the catalog

# Option 2: Manual installation
# SSH into TrueNAS and install node_exporter
curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

Import pre-built TrueNAS dashboards from Grafana.com or build custom ones. Key panels:

1. Pool Health Status: Single stat panel showing pool state (ONLINE/DEGRADED/FAULTED).
2. Pool Capacity Gauge: Gauge showing % used, with thresholds at 70% (yellow) and 90% (red).
3. ARC Hit Ratio Time Series: Line chart showing hit ratio over 24 hours.
4. Disk Temperature Heatmap: Color-coded table of all disk temperatures.
5. I/O Throughput: Stacked area chart of read/write throughput per pool.
6. SMART Predictive Failures: Table showing drives with SMART warnings.

TrueNAS NUT configuration is stored in /etc/nut/:

[myups]
    driver = usbhid-ups
    port = auto
    desc = "APC Back-UPS 1500"

# /etc/nut/upsd.conf
LISTEN 0.0.0.0 3493
MAXAGE 15

# Check UPS status
upsc myups

# Key fields:
# battery.charge    — Remaining charge percentage
# battery.runtime   — Estimated minutes remaining
# ups.status        — "OL" (online), "OB" (on battery), "LB" (low battery)
# ups.load          — Load percentage
# input.voltage     — Input voltage
# output.voltage    — Output voltage
# ups.temperature   — UPS internal temperature

When the UPS detects a power failure:

1. Power failure detected. UPS switches to battery.
2. NUT notifies TrueNAS. The UPS status changes to “OB” (on battery).
3. Timer starts. TrueNAS waits for the configured delay (e.g., 5 minutes).
4. If power is restored within the delay: Normal operation resumes. No shutdown.
5. If timer expires: TrueNAS initiates shutdown: a. Stop all services (SMB, NFS, apps). B. Sync all ZFS pools (zpool sync). C. Export all pools (zpool export -a). D. Shutdown the system (shutdown -p now).
6. UPS signals low battery: UPS sends the final “LB” signal and shuts itself down.

For environments with multiple servers on one UPS:

# TrueNAS (master): Connected to UPS via USB
# Server 2 (slave): Monitors via NUT network

# On the master (TrueNAS):
# /etc/nut/upsd.users
[upsmonitor]
    password = <password>
    upsmon master

# On the slave (Server 2):
# /etc/nut/upsmon.conf
MONITOR myups@truenas-ip 1 upsmonitor <password> master
SHUTDOWNCMD "/sbin/shutdown -h +0"

For environments with multiple systems, centralize logs using:

• syslog: Configure TrueNAS to forward logs to a central syslog server.
• ELK Stack: Elasticsearch + Logstash + Kibana for log analysis.
• Loki + Grafana: Lightweight log aggregation with Grafana integration.

TrueNAS retains logs according to:

#!/bin/bash
# Check for common error patterns in system logs
LOG_FILE="/var/log/messages"

echo "=== ZFS Errors ==="
grep -i "zfs.*error\|zfs.*fault\|zfs.*degraded" "$LOG_FILE" | tail -20

echo "=== Disk Errors ==="
grep -i "ata.*error\|scsi.*error\|i/o error\|medium error" "$LOG_FILE" | tail -20

echo "=== Network Errors ==="
grep -i "link.*down\|carrier.*lost\|crc.*error" "$LOG_FILE" | tail -20

echo "=== Memory Errors ==="
grep -i "out of memory\|oom\|page allocation failure" "$LOG_FILE" | tail -20

echo "=== UPS Events ==="
grep -i "ups\|battery\|power failure" "$LOG_FILE" | tail -20

# Track pool capacity over time
zpool list -v -p | awk '{print strftime("%Y-%m-%d"), $0}' >> /var/log/pool_capacity.log

# Analyze growth rate (daily)
cat /var/log/pool_capacity.log | awk '{print $1, $4}' | \
  awk 'NR>1 {printf "%s %s %+.1f GB/day\n", $1, $2, ($2-prev)/((NR>2)?1:1)} {prev=$2}'

Based on historical growth rate, project when the pool will reach capacity thresholds:

Where:

• $C_{total}$ is the total pool capacity
• $C_{used}$ is the current used capacity
• $R_{growth}$ is the daily growth rate

Suppress alerts during known maintenance windows:

# TrueNAS does not have built-in alert suppression.
# Workaround: Disable email alerts during maintenance, re-enable after.
# Or use an external alerting system (PagerDuty) with maintenance windows.

# prometheus.yml for TrueNAS monitoring
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "truenas''
    static_configs:
      - targets: ["truenas.local:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: "(.*):(.*)''
        replacement: "${1}'

  - job_name: "smartmon''
    static_configs:
      - targets: ["truenas.local:9633']

  - job_name: "zfs''
    static_configs:
      - targets: ["truenas.local:9133']

Export and share Grafana dashboards:

1. Navigate to the dashboard in Grafana.
2. Click the share icon (top right).
3. Select “Export” → “View JSON”.
4. Save the JSON file.
5. Import on another Grafana instance.

# Prometheus alert rule for ZFS pool capacity
groups:
  - name: truenas_alerts
    rules:
      - alert: ZFSPoolCapacityHigh
        expr: zfs_pool_used_bytes / zfs_pool_size_bytes > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "ZFS pool {{ $labels.pool }} is above 85% capacity''
description: "The TrueNAS dashboard provides a real-time overview of system health: Comprehensive educational content coverage with definitions and practice problems."

For environments with multiple TrueNAS systems or other servers:

ELK Stack (Elasticsearch, Logstash, Kibana):

1. Install Filebeat on TrueNAS to forward logs to Logstash.
2. Logstash parses and enriches the logs.
3. Elasticsearch stores and indexes the logs.
4. Kibana provides visualization and search.

Loki + Grafana (lightweight alternative):

1. Install Promtail on TrueNAS to forward logs to Loki.
2. Loki stores logs in a compressed index.
3. Grafana provides LogQL queries and visualization.
4. Much lighter than ELK, suitable for smaller deployments.

# Configure log rotation in TrueNAS
# Navigate to System → Advanced → Syslog
# Set maximum log file size (default: 10 MB)
# Set maximum number of archived log files (default: 5)

# Track pool usage over time
#!/bin/bash
LOG="/var/log/pool_capacity.log"
echo "$(date "+%Y-%m-%d'),$(zpool list -Hp -o capacity tank),$(zpool list -Hp -o used tank)" >> "$LOG"

# Calculate growth rate (last 30 days)
tail -30 "$LOG" | awk -F',' '
  NR>1 {
    used_diff = $3 - prev_used
    days_diff = NR - 1
    printf "Daily growth: %.1f GB/day\n", used_diff / days_diff
  }
  { prev_used = $3 }
'

1. Add vdevs: Add new vdevs to the pool (increases both capacity and performance).
2. Replace with larger drives: Replace drives one at a time with larger drives (increases capacity only).
3. Cloud sync archival: Move cold data to cloud storage.
4. Data pruning: Delete unnecessary snapshots, old logs, and temporary files.

This topic covers the essential concepts and techniques related to monitoring and alerting, including key principles and practical applications.

Key concepts include:

• core concepts and definitions
• key principles and frameworks
• practical applications
• common techniques and methods
• evaluation and critical analysis

A thorough understanding of these concepts, combined with regular practice and review, is essential for mastery of this topic.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.