Introduction
After my initial storage migration to Ceph for my main VM storage and exploration of iSCSI, I still needed to upgrade my back-up managment and making my bulk storage shared in my Proxmox cluster. Currently, my bulk storage is a ZFS pool local to one of my hosts. This happens to be the host with the least RAM and least compute. I want the bulk storage to be shared so I can use it to host large datasets for VMs as well as to serve as a backup target. Additionally, my fileserver for large amounts of media and simple file backups is also shared via SMB with the other PCs on my network.
One of my main goals was to keep ZFS as main bulk storage mechanism, instead of going to something like CIFS/NFS for sharing the bulk storage. Snapshotting and such are quite nicely integrated with ZFS on Proxmox and I like to be able to snapshot volumes before doing major software upgrades (Although ideally all running services should be on Ceph and the HDD pool should just be data at rest...).
Current setup
host1: (SuperMicro)
|-240GB Boot SSD
|-500GB SSD Ceph OSD
|-500GB SSD ZFS tank-spool
|-500GB SSD ZFS tank-spool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-4TB HDD raid10-pool
|-8TB HDD raid10-pool
\-8TB HDD raid10-pool
My main host has a mix of storage. It has a local boot SSD with LVM (should be replaced by a ZFS mirror...) as well as a ZFS SSD mirror pool for fast local storage.
For Ceph, it has a single SSD as OSD.
Finally, the bulk storage is ZFS pool with 3 mirrored vdevs. Some people prefer RAIDZ, but I chose to go with the reasoning laid out in You should use mirror vdevs, not RAIDZ..
This also makes it a lot easier for me to just add mirrored vdevs when I want to expand the pool.
hosts 2 and 3 (R320)
|-60GB SSD Boot zfs-mirror
|-60GB SSD Boot zfs-mirror
|-800GB SSD Ceph OSD
\-500GB SSD Ceph OSD
My two other Proxmox hosts, my 'compute' hosts currently can't access the bulk storage and use Ceph for shared storage. This means that I also can't really backup the VMs on Ceph, because Ceph itself doesn't have that much capacity (only 2TB or so, most already in use).
Goals
- VM Bulk storage with snapshots and thin provisioning
- Proxmox backups (basically just compressed files)
- Fileserver for local network
Option 1: Proxmox native iSCSI and fileserver
Pros:
- Minimal performance overhead
- "single pane of glass" for storage management
- Less moving parts/software to maintain
Cons: - Requires iSCSI target software on Proxmox host
- Requires manual user administration on Proxmox host
- No 'nice' features for storage managment like NAS software
- No isolation of storage and compute
In this scenario I would do everything natively on my Proxmox host. The only real change to the current setup would be to add an iSCSI target to the current bulk tank with initiators on each of my Proxmox hosts. In practice, it entail far more. Setting up an iSCSI target on Proxmox is not trivial as iSCSI target software seems to be more of an BSD thing (see also my exploration of XigmaNAS). Furthermore, Proxmox's native iSCSI capabilities are very lacking both as initator and target. It's native implementations use SSH for direct iSCSI operations on the targets.
Option 2: Native NFS or CIFS and native fileserver
Pros:
- NFS is well supported on Linux
- No native snapshotting of single VM vols, only of the entire underlying dataset (or qcow2s)
- No native thin provisioning (again would require qcow2)
Cons:
- Requires manual user administration on Proxmox host
- Abstraction layer on top of ZFS
For this scenario I'd have to give up my ZFS capabilities for the most part and rely on other storage mechanisms on top of the bulk storage pool. The ZFS as such would remain local only, but I would make specific datasets on the pool available as shared storage to the cluster. This is somewhat similar to the way I currently already have the fileserver in place. Likely, I would make a dataset specifically for the NFS server. I would have to move all current VMs to qcow2 images on top of NFS, giving up all their current snapshots etc.
Option 3: Virtualized TrueNAS without passthrough HBA
Pros:
- Easy user administration
- Isolation of user data from compute host
- Easy setup
Cons:
- Yet another VM
- TrueNAS can't properly use its ZFS capabilities
- Non ideal storage layering
- How to export bulk storage to Proxmox?
- Storage only comes online after the main host has booted its VMs
Another option is to virtualize TrueNAS on my main Proxmox box and either passthrough the individual disks or a zvol. However, in both these cases, TrueNAS wouldn't have low level access to the drives and can't really properly use its own ZFS capabilities. Furthermore, sharing the bulk storage wouldn't be ideal. If I'd still want ZFS over iSCSI then TrueNAS would have to do ZFS on its passthrough disks or zvol. Neither is ideal. The advantage is that TrueNAS has a good community plugin for Proxmox iSCSI.
Option 4: Virtualized TrueNAS with passthrough HBA
Pros:
- Easy iSCSI administration
- Easy user administration
- All bulk storage handled by TrueNAS
Cons:
- Yet another VM
- PCIe passthrough is harder to set up
- IOMMU groups are not great on my main server
- Storage only comes online after the main host has booted its VMs
A better option for TrueNAS is to PCIe passthrough my HBA so that it can properly do its own ZFS things with the drives. I could then nicely use the TrueNAS iSCSI plugin for Proxmox to share the bulk storage. The fileserver function would just be handled be handled by TrueNAS itself, providing easy administration.
Conclusion
Regardless of which of the above options I would pick, I need to rearchitect a bunch of my VMs, moving their storage. Some of the containers on the current SuperMicro host even have local mount points with mapped UIDs. That needs to change regardless of which option I pick.
Although I quite like the administration options offered by TrueNAS, I don't really like needing another abstraction layer for my ZFS pool. I think I initially would like the explore Option 1, but I am not looking forward to manually doing my iSCSI target configuration...