I thought I learned this lesson years ago, but apparently not. I have a small homelab. It is just Proxmox on an old HP workstation that has way more memory and CPU power than I really need. I love it, though, and it has been fun to tinker with. I have known for ages that I needed a better backup solution for the server itself. Proxmox has a backup server project that would have been perfect.

But I never got around to it, and now I’m a bit screwed.

I upgraded from version 8.4 to 9 tonight. I didn’t think anything of it, really. It’s Debian. Debian doesn’t really break. Oops. The thing is, the upgrade worked fine. I was able to launch back into the server, my VMs started up proper and all the Docker containers were working fine. I walked away for some family time and thought, ‘job done.’ But when I came back I noticed that my Glance homepage wasn’t loading. Neither was portainer. So, I tried to ssh into the Proxmox server and it was completely offline.

Figured it was a one off thing, but nope. It happens over and over again. The error logs say it is something to do with an offline member of the cluster, but I fixed that and it’s still doing it.

Theoretically, all my data is stored on my nfs server, which is separate. In theory, I should be able to reinstall Proxmox, re-setup my containers and VMs, and the data should repopulate. I doubt it will be that easy, though. Nothing ever is. (As I was editing this, I realized that a lot of my smaller data, stuff like FreshRSS, isn’t on the NFS server at all. FML).

The good news is that I have saved my backups from my main computer. I did run my borg backups through a VM on the Proxmox server, but it was pretty easy to just point my backup script to my NFS server where the backups are actually stored. Hopefully everything else goes that easily. (It won’t)

I think that I will use this as an opportunity to redo some things. I’ll either go smaller and use some of the ThinkCentres I have and abandon the HP monstrosity. Or I’ll keep with my old setup, but do a better job of utilizing Proxmox. More containers and VMs, instead of everything just being shoved in two VMs. I’ll also, if that’s the route I go, set up that Proxmox backup server. I’ll make sure this nonsense doesn’t happen again.

If I do lose the data, I’ll be out quite some work. I host one of my podcasts locally, and that would be a pain to setup elsewhere. I think the part that would suck the most is that all of my RSS feeds are in FreshRSS, and I have no backup of that at all. If I can’t get that docker container to work with the old data, I’m going to be very sad. Nextcloud also sucks, because I sync my notes through there. But I do have all of that stuff saved locally, so it won’t be a big deal. I will also lose a lot of saved for later content in Linkding. Hopefully, I’ll be able to recover this, but if not, it will be a painful lesson.

So learn from me, guys. Your main PC isn’t the only thing that needs backups. Your servers to do too. Granted everyone knows this. I did too. But I was too lazy. So, don’t be lazy.

This was day 22 of Blaugust 2025