20240916, Day 30
Situation report:
Almost a month into the challenge and I have done something Homelab every day.
Vacation is over, back to computer work that pays bills. Need to emphasize on Ceph, Proxmox’ networking, and Mikrotik. Installing applications, server stuff, and running fiber optic across the house was fun (ymmv) but those three technologies are pillars and it’s hard to goof off developing on the fun little projects when big pieces aren’t working yet.
Started reading Ceph documentation after building the test cluster (classic).
Read up to Switching & Bridging in Mtk.
20240917, Day 31
Ceph fundamentals & requirements. Going to need to wire up that 10Gb….
Booted up Eve-NG, fixed it up, made sure CHR image is still working. Need that all, because:
Getting to a lab section of the Mtk manual.
U r smelly
20240918, Day 32
I was getting into the Ceph reading, it’s what it needs to be but can be exhaustive. Checking out the Proxmox writings on Ceph and it might be much better to get started quickly.
I tried to fit in a Mtk lab and found out VPC doesn’t support vlan tagging (…does it?). Perhaps I’d be better off using a router as a PC.
20240919, Day 33
Just a big read on the Proxmox Ceph implementation today. It gets to things like operation & maintenance very quickly. I’m fearing that to mess with it i need to build another host out of something.
I did rollback and start a documented reload of Ceph in the cluster but i am out of energy. A friend asked why i wasn’t out goofing off and I took this picture.
20240920, Day 34
Spec’ing out host hardware morning, evening, and between work. Doesn’t feel like “lab” work, but it basically is, plus I’m stupid with hardware so doing anything with that is good.
20240921, Day 35
More brooding over hardware details, make a list.
Off to computer store. Redo almost all of list except for mobo and memory. Microcenter staff was a big help.
But alas; while getting help online about PCIe stuff, the Internet assured me Proxmox cannot cluster, and Ceph cannot run on three nodes!!1
Will those technologies do that?? Stay tuned to find out.
20240922, Day 36
I built a host~ Haven’t built a computer from scratch in a long time, new ones are very nice.
Despite all the big pieces powering up and working, I think I made a mistake with the CPU selection so I’m dealing with that.
20240923, Day 37
Had to track down another processor for the new host, turns out the version of that family with ECC support is rare. Oh well, learning.
Got the virtual Ceph lab going and picked up where I left off, made a cluster. Next is sharing out storage to userland.
20240923, Day 38
Fun morning, I broke things~
It’s all fake, so it’s fine, but wow
Breakthrough evening:
Got CephFS going and the block storage accessible, and used both
and they said it couldn’t be done…
Here it is installing Debian VM’s. Installing VM’s, on a cluster who’s hosts are VM’s, running inside another cluster. So Yes the performance was abyssmal, IOPS cat is doing his best over there. But i did the thing.
20240924, Day 39
Crashed one of the vhosts, Ceph blew out but fixed itself before i got a screenshot
Although parts for the old cluster showed up today, and parts for the new host are in the mail, I had a big think about another new host and how to make networking go for that. Watching a lot of workstation cooling vids and nerds building their own servers vids.
Just so the day wasn’t all planning and hypothetical i make & break some Ceph things a few times just to see what it would do. It’s fun taking out a drive and watching a host keep going, but more fun when you do a Global All Stop and forgot you had a VM running on that.
(it lived!!! Global Started and everything I’d tried to test just popped up)
20240926, Day 40
Asking questions on support forums, browsing around network parts.
Working out how to make a reliable lil’ hyperconverged cluster.
But alas, I took time to figure out how to get CephFS mounted from a client.
woohoo I Made a Folder. It is truly a special day.
… that’s not much to look at but I’d wondered for months how filesystem access to Ceph really worked. I’d always just assumed it’d be something like an NFS but instead it’s a mount -t cephfs option.
20240927, Day 41
Checking with a Pxmx forum about networking, mesh should be workable. Ordered 40Gb dual NIC’s on used market for surprisingly cheap. Received cable management swingarms for R-series.
20240928, Day 42
I ordered another motherboard/CPU/RAM slam. Someone, send help
Spent way too long brooding over the specs for that. Instead of a computer pic, here’s some kittens I cleaned house for at the animal shelter today.
Ember and Oreo. <3 ^_^
20240929, Day 43
the sysadmin: throws away the cable management arms
the nerd: nabs cable management arms, installs at home
20240930, Day 44
First day with Corosync editing and I decided to clean up cluster networking. Did not go very well, lost half the cluster at one point. I’ve got 75% of it back and quitting for today.
20241001, Day 45
WELL THEN: i dumped the whole enterprise half of the stack getting things going, but fixed it! Not sure I want to fess up to how that got done
And then I broke it again, and fixed it again
Learned a lot about Corosync but gee whiz. Always means good things when you are four wide in terminal sessions.
Still one host that’s not on a dedicated Coro connection but I can’t stand to break it all again tonight.
20241002, Day 46
Bad news: I still haven’t gotten the Corosync network set up the way I want it
Good news: Every time I’ve seemingly irrecoverably broken the cluster I’ve learned some new way to get it running again
The cluster is just staying misnumbered until I’m able to simulate this and try again. Moving on.
20241003, Day 47
Mail call~
Including:
A pro series Ryzen so my memory now counts as ECC memory
Secondhand mini-ATX server board
A four-pack of kinda-fast network cards
I didn’t get the NICs plugged into anything but figured that at least testing that board and proc deserved a special dinner.
What do you mean it’s not healthy. There’s a salad
20241004, Day 48
Homelab… takes all shapes, right?
Brain transplant, de-pasting a CPU & fan, cleaning stuff up for a return. Gather up network card & micrometer for fan install as I read these cards throttle down or pause if you don’t active cool. Make a list of parts & pieces.
Also bring new fan to bribe the returns desk, if needed.
20241005, Day 49
Computer store haul:
Besides dropping cash and being annoying, I swapped the free workstation ram from the server motherboard in the workstation and installed all the workstation’s ECC ram onto the server board.
Everything rings up correctly but to test ECC & really do anything I need to brain transplant the two CPU’s including full de- and repaste.
Not a lot done, but also not bad for Caturday.
20241006, Day 50
Halfway there. Today was musical chairs:
Swap server brains out of workstation, swap workstation brains out of server, get workstation going.
Pro tip: Just because you got a free processor doesn’t mean it included video for free.
When the workstation counted up right I took that all apart since I’ve had a different plan for it. Re-cased all the server guts into the workstation case for now and got it counted up right.
Yup it’s old server guts on the floor but away we go
And unlike all the other parties I didn’t go to during the 100, I went to one.
Sometimes it’s good to see yourself in a new way.
