Introduction

After my initial storage migration to Ceph for my main VM storage and exploration of iSCSI, I still needed to upgrade my back-up managment and making my bulk storage shared in my Proxmox cluster. Currently, my bulk storage is a ZFS pool local to one of my hosts. This happens to be the host with the least RAM and least compute. I want the bulk storage to be shared so I can use it to host large datasets for VMs as well as to serve as a backup target. Additionally, my fileserver for large amounts of media and simple file backups is also shared via SMB with the other PCs on my network.

One of my main goals was to keep ZFS as main bulk storage mechanism, instead of going to something like CIFS/NFS for sharing the bulk storage. Snapshotting and such are quite nicely integrated with ZFS on Proxmox and I like to be able to snapshot volumes before doing major software upgrades (Although ideally all running services should be on Ceph and the HDD pool should just be data at rest...).

Current setup

host1: (SuperMicro)
|-240GB Boot SSD
|-500GB SSD Ceph OSD
|-500GB SSD ZFS tank-spool
|-500GB SSD ZFS tank-spool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-8TB HDD raid10-pool
\-8TB HDD raid10-pool

My main host has a mix of storage. It has a local boot SSD with LVM (should be replaced by a ZFS mirror...) as well as a ZFS SSD mirror pool for fast local storage. For Ceph, it has a single SSD as OSD.
Finally, the bulk storage is ZFS pool with 3 mirrored vdevs. Some people prefer RAIDZ, but I chose to go with the reasoning laid out in You should use mirror vdevs, not RAIDZ.. This also makes it a lot easier for me to just add mirrored vdevs when I want to expand the pool.

hosts 2 and 3 (R320)
|-60GB SSD Boot zfs-mirror
|-60GB SSD Boot zfs-mirror
|-800GB SSD Ceph OSD
\-500GB SSD Ceph OSD

My two other Proxmox hosts, my 'compute' hosts currently can't access the bulk storage and use Ceph for shared storage. This means that I also can't really backup the VMs on Ceph, because Ceph itself doesn't have that much capacity (only 2TB or so, most already in use).

Goals

VM Bulk storage with snapshots and thin provisioning
Proxmox backups (basically just compressed files)
Fileserver for local network

Option 1: Proxmox native iSCSI and fileserver

Pros:

Minimal performance overhead
"single pane of glass" for storage management
Less moving parts/software to maintain
Cons:
Requires iSCSI target software on Proxmox host
Requires manual user administration on Proxmox host
No 'nice' features for storage managment like NAS software
No isolation of storage and compute

In this scenario I would do everything natively on my Proxmox host. The only real change to the current setup would be to add an iSCSI target to the current bulk tank with initiators on each of my Proxmox hosts. In practice, it entail far more. Setting up an iSCSI target on Proxmox is not trivial as iSCSI target software seems to be more of an BSD thing (see also my exploration of XigmaNAS). Furthermore, Proxmox's native iSCSI capabilities are very lacking both as initator and target. It's native implementations use SSH for direct iSCSI operations on the targets.

Option 2: Native NFS or CIFS and native fileserver

Pros:

NFS is well supported on Linux
No native snapshotting of single VM vols, only of the entire underlying dataset (or qcow2s)
No native thin provisioning (again would require qcow2)

Cons:

Requires manual user administration on Proxmox host
Abstraction layer on top of ZFS

For this scenario I'd have to give up my ZFS capabilities for the most part and rely on other storage mechanisms on top of the bulk storage pool. The ZFS as such would remain local only, but I would make specific datasets on the pool available as shared storage to the cluster. This is somewhat similar to the way I currently already have the fileserver in place. Likely, I would make a dataset specifically for the NFS server. I would have to move all current VMs to qcow2 images on top of NFS, giving up all their current snapshots etc.

Option 3: Virtualized TrueNAS without passthrough HBA

Pros:

Easy user administration
Isolation of user data from compute host
Easy setup

Cons:

Yet another VM
TrueNAS can't properly use its ZFS capabilities
Non ideal storage layering
How to export bulk storage to Proxmox?
Storage only comes online after the main host has booted its VMs

Another option is to virtualize TrueNAS on my main Proxmox box and either passthrough the individual disks or a zvol. However, in both these cases, TrueNAS wouldn't have low level access to the drives and can't really properly use its own ZFS capabilities. Furthermore, sharing the bulk storage wouldn't be ideal. If I'd still want ZFS over iSCSI then TrueNAS would have to do ZFS on its passthrough disks or zvol. Neither is ideal. The advantage is that TrueNAS has a good community plugin for Proxmox iSCSI.

Option 4: Virtualized TrueNAS with passthrough HBA

Pros:

Easy iSCSI administration
Easy user administration
All bulk storage handled by TrueNAS

Cons:

Yet another VM
PCIe passthrough is harder to set up
IOMMU groups are not great on my main server
Storage only comes online after the main host has booted its VMs

A better option for TrueNAS is to PCIe passthrough my HBA so that it can properly do its own ZFS things with the drives. I could then nicely use the TrueNAS iSCSI plugin for Proxmox to share the bulk storage. The fileserver function would just be handled be handled by TrueNAS itself, providing easy administration.

Conclusion

Regardless of which of the above options I would pick, I need to rearchitect a bunch of my VMs, moving their storage. Some of the containers on the current SuperMicro host even have local mount points with mapped UIDs. That needs to change regardless of which option I pick.

Although I quite like the administration options offered by TrueNAS, I don't really like needing another abstraction layer for my ZFS pool. I think I initially would like the explore Option 1, but I am not looking forward to manually doing my iSCSI target configuration...