zfs

Pools are managed with the zpool(8) command and have the following hierarchy:

pool: consists of one or more virtual devices (vdev)
vdev: consists of one or more physical devices (dev) and come in different kinds such as disk, mirror, raidzX, ...
- disk: single physical disk (vdev == dev)
- mirror: data is identically replicated on all devs (requires at least 2 physical devices).

Data stored in a pool is distributed and stored across all vdevs by zfs. Therefore a total failure of a single vdev can lead to total loss of a pool.

A dataset is a logical volume which can be created on top of a pool. Each dataset can be configured with its own set of properties like encryption, quota, .... Datasets are managed with the zfs(8) command.

zfs pool management

Pools are by default mounted at /<POOL>.

Create, modify and destroy zfs pools

# Create a pool MOOSE with a two mirror vdevs.
zpool create moose mirror <dev1> <dev2> mirror <dev3> <dev4>..

# Add new raidz1 vdev to a pool.
zpool add moose raidz1 <devA> <devB> <devC>..

# Remove a vdev from a pool.
zpool remove moose <vdevX>

# Destroy a pool.
zpool destroy moose

For stable device names in small home setups it is recommended to use names from /dev/disk/by-id.

Inspect zfs pools

# Show status of all pools or a single one.
zpool status [<pool>]

# Show information / statistics about pools or single one.
zpool list [<pool>]

# Show statistics for all devices.
zpool list -v

# Show command history for pools.
zpool history

Modify `vdevs`

# vdev MIRROR-0 with two devs.
zpool status
    NAME            STATE     READ WRITE CKSUM
    moose           ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        virtio-200  ONLINE       0     0     0
        virtio-300  ONLINE       0     0     0

# Attach new device to an existing vdev.
zpool attach moose virtio-200 virtio-400

# vdev MIRROR-0 with three devs.
zpool status
    NAME            STATE     READ WRITE CKSUM
    moose           ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        virtio-200  ONLINE       0     0     0
        virtio-300  ONLINE       0     0     0
        virtio-400  ONLINE       0     0     0

# Detach device from vdev.
zpool detach moose virtio-200

Replace faulty disk

# MIRROR-0 is degraded as one disk failed, but still intact.
zpool status
    NAME            STATE     READ WRITE CKSUM
    moose           DEGRADED     0     0     0
      mirror-0      DEGRADED     0     0     0
        virtio-200  UNAVAIL      0     0     0  invalid label
        virtio-300  ONLINE       0     0     0

# Replace faulty disk, in mirror.
# No data is lost since mirror still has one good disk.
zpool replace moose virtio-200 virtio-400

# MIRROR-0 back in ONLINE (good) state.
zpool status
    NAME            STATE     READ WRITE CKSUM
    moose           ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        virtio-400  ONLINE       0     0     0
        virtio-300  ONLINE       0     0     0

Import or export zfs pools

When moving pools between hosts, the pool must be exported on the currently active host and imported on the new host.

# Export a pool called MOOSE.
zpool export moose
# If datasets are busy, use lsof to check which processes keep it busy.
#   lsof <mntpoint>

# List pools that can be imported using BY-ID deivce names (for example).
zpool import -d /dev/disk/by-id

# Import pool MOOSE using BY-ID device names (for example).
zpool import -d /dev/disk/by-id moose

Device names used by an existing pool can be changed by exporting and importing a pool again.

zfs dataset management

Datasets are by default mounted at /<POOL>/<DATASET>.

Create and destroy zfs datasets

# Create dataset FOO on pool MOOSE.
zfs create moose/foo

# Destroy dataset.
zfs destroy moose/foo

List all zfs datasets

# List all zfs datasets.
zfs list

Mount zfs datasets

# List currently mounted datasets.
zfs mount

# Mount dataset.
zfs mount moose/foo

# Unmount dataset.
zfs unmount moose/foo

Encrypted datasets

Encryption is a readonly property, can only be set when creating a dataset.

# Create encrypted dataset FOO on pool MOOSE.
zfs create -o encryption=on -o keyformat=passphrase moose/foo

# Mount encrypte dataset and load encryption key (if not loaded).
zfs mount -l moose/foo
# -l is equivalent to first loading the key via zfs load-key moose/foo.

# Unmount dataset and unload encryption key (unload is optional).
zfs umount -u moose/foo

Manage zfs encryption keys

# Preload encryption key for dataset.
zfs load-key moose/foo

# Preload encryption key for all datasets.
zfs load-key -a

# Change encryption key for dataset.
zfs change-key moose/foo

# Unload encryption key for dataset.
zfs unload-key moose/foo

Manage dataset properties

# Get all properties for dataset.
zfs get quota moose/foo

# Get single property for dataset.
zfs get all moose/foo

# Get single property for all datasets.
zfs get quota

# Set property on dataset.
zfs set quota=10G moose/foo

Snapshots

# Create snapshot called V2 for dataset moose/foo.
zfs snapshot moose/foo@v2

# List all snapshots.
zfs list -t snapshot

# Make .zfs direcotry visible in the root of the dataset.
zfs set snapdir=visible moose/foo

# Browse available snapshots in visible .zfs direcotry (readonly).
ls /moose/foo/.zfs/snapshot
v1/  v2/

# Create a new dataset based on the V1 snapshot
zfs clone moose/foo@v1 moose/foov1

# Destroy snapshot.
zfs destroy moose/foo@v1

Access control list

Focus on posix acl.

# Set the ACL type for the FOO dataset to POSIXACL.
zfs set acltype=posixacl moose/foo

# Get the ACL type of a given dataset.
zfs get acltype moose/foo

For performance reasons it is recommended to also set zfs set xattr=sa moose/foo [ref].

Example: zfs pool import during startup (`systemd`)

The default zpool cache file is /etc/zfs/zpool.cache. When pools are imported the cache is updated.

Enable the following targets / services to automatically import pools from the cache.

systemctl list-dependencies
  ...
    └─zfs.target
      └─zfs-import.target
        └─zfs-import-cache.service

Notes