I tried this in my homelab, and I wanted to convert my Proxmox with it’s LXC’s over to it…but the problem I ran into was that it was a little too obtuse for what I wanted to use it for.
Long story:
I wanted to get away from Proxmox as a platform and LXCs in general over to another hypervisor and using standard containers. I was tired of having a full fat web GUI to manage things with. Yes you can use pct and other CLIs tools in Proxmox to get some speed, but you are still dealing with an opinionated, dare I say proprietary stack.
Incus is really cool in that it’s CLI driven first, and you can add a GUI later if you want. I thought that was nice, and I was able to get some VMs started in my lab cluster and get clustering kinda working. I thought I was in a good place to switch.
But then, my main file server died (thankfully just the mobo, not the ZFS disks). And I was scrambling to figure out how to get all of my functionality back without having to go back to adding the storage back to Proxmox. So I took it as an opportunity to go to incus.
Well it turns out, it’s not so turn key depending on your method of deployment. For one, the actual “IncusOS” deployments are really particular about having a newer version of TPM (not something the old boxes I purchase typically have). So you are still likely to install something like Debian + incus-cli tools.
Once you have your distro + incus package setup though, there are a few things you have to tackle. For one, if you actually want separate IP addresses per VM (or LXC) you’ll likely need to solve the chicken and egg problem with macvlan, otherwise you’ll have to figure out something for a reverse proxy with the network bridge.
So because I was pressed against the wall (my family was irritated about no jellfyin or immich), and I knew that I didn’t want to do LXCs again and simply wanted to do most things I can in containers, I went the podman route instead.
Every app gets it’s own local user. You can install podman-compose, and use the same commands as docker compose basically to get your app up. All rootless, and you can also setup podman compose apps with daemons so that they start on boot: https://www.it-hure.de/2024/02/podman-compose-and-systemd/. If you don’t care about rootless isolation, I think Docker is still fine. Everyone seems to make documentation and build their apps with it in mind still.
I still had a need for VMs, so now I just use the standard Qemu+KVM+libvirtd combo for the one or two apps that need that need a full on separation from other apps.
All of it is in Ansible playbooks now, and I’m pretty happy with the setup I think.
TL; DR
Incus is really cool. I probably wouldn’t bother with IncusOS on old hardware, I’d just install a standard Linux distro + incus packages. Spend your time evaluating it (and making it infrastructure code for easy deployment maintenance). Maybe try out the other “standard” ways to do it instead to compare (Docker, Podman, Qemu+KVM, etc).
You typically use a RAID configuration for redundancy on a media server, so when a disk fails (it is when, not if) you don’t lose all your data. Having a mirror or a striped volume (or both with RAID 10) can also give a performance boost since your read and writes are going across multiple disks at the same time (depending on your RAID configuration).
However, I can’t think of a good reason to do hardware RAID anymore. You can get the same redundancy and have it be “safer” through setting up ZFS.
I ran OMV with ZFS with two disks as a ZFS mirror and it accomplished the same thing as a similar RAID mirror would do. I got faster read speeds (think serving media faster so higher bitrate) and if one disk dies, I can recover.
omv-extras.org has a plugin that will give you the ability to setup ZFS, as it doesn’t come with it by default unfortunately.
With RAID 10, you would consume all the drives in your configuration and use it like a bucket so to speak. If you did the same thing in ZFS, you could have a couple of mirrors and then pool them together to increase performance.
Either setup (RAID or ZFS) should accomodate k8s or other devops workflows. You could simply make another folder for these things, or in ZFS you could make a special vdev for it.
Anyways, theres lots of ways to accomplish what youre seeking. You can definitely do all those things things you asked about. I think you just have to do some research on what kind of setup works best for you and the kind of work this machine will do.
Edit: And make a backup! RAID or ZFS may help keep your data available, but you should always have a backup somewhere too for important things like photos. Cant stress it hard enough!!!