There’s a few threads on here sporting homelab builds, repurposed hardware from gaming PCs, servers, networking and general tinkering for the sake of tinkering and role-playing as a sys-admin. Today I too have decided it is time to post my would be junk in to a thread now I’ve spent a few quid tidying it all up and getting everything working as I’d like (sort of). There’s a few goals I have to try and meet and a few nice to haves that I’ve thrown in for good measure so let’s have a look. Goals. - Repurpose some old server hardware as well as an old casual gaming PC to allow for a Plex server, lossless music streaming (as close to as I am able), generic storage, networked high(ish) speed storage and a few VMs. - Rethink and execute/deploy a shared VPN server, network management tools, CCTV. - Patch in cat6 locations in each room Nice to have. - Host game servers and team speak - Facility to remotely hard power down devices so that I don’t have to exert energy walking to it or god forbid, be unable to reset something until I’m home. - Monitor environmental factors such as temp, static pressure, humidity etc. Anyway, the hardware isn’t great. It’s either not very powerful but reasonably low powered or it packs a punch but draws significant power. We also have no punch and loads of power draw too but the hardware is that old that replacing it would take years in utility bill extras to allow new efficient hardware to make up for it so it’s a none starter. Some of the available kit below, not all of it will be used. - 3400G b450i system to repurpose. (Nvme + 16gb ram) - 5950x b550i system (with a2000) to repurpose. (Nvme + 64gb ram) - HP ML350e with an E3-1240v3, b120i raid controller, dual 430w psu, 32gb ram.. The reality is I’ll likely ditch the HP server and probably the 5950x system, upgrading the ram in the 3400g based system assuming it’s up to the job of running Plex and a few other snip bits for media. I’m unsure at present. I’ve got plenty of 5 port, 8 port gigabit switches, POE and standard. All unmanaged. I didn’t think these were up to the job of anything useful but I have them to hand just in case I need to commit a crime against networking. I’ve also got a Trendnet 18 port (8port POE, 8port POe+, dual 10G SFP) switch of sorts but it needs modifying as it sounds like a sodding hoover. After a bit of research I’ve decided to be lazy and to buy a Ubiquiti UDM SE to act as the main hub within this set up as well as a pair of G3 flex cameras and a Gx bullet something or other. I can throw a cheapo 2tb HDD in there and call it done for a CCTV/NVR. It’s a lazy way to do things I know but I want to be lazy in this case for ease of use more than anything. I’ve also managed to nab an 1U smart, connected, by port switching PDU with OOB management as well as the facility to attach environmental sensors which ticks a good few boxes too. The issue is, I need to patch the UDM and the switch in to keep things tidy so that’s 3U gone. Add with 3U for the server chassis and 1U for the PDU and we are at a point where my space (8-9U) is about filled up as the space available will only allow for a mini rack of that sort of height. Over all I think I have my work cut out. I’ve got a reasonable idea of what to do and where to chuck it all but it’s getting it all together and executing this reliably that’s going to be the challenge. Before I bowl in head first and give myself trauma, does anyone have any suggestions?
I’d say you’re pretty much there in terms of hardware. Ex-corporate rackmount kit can be great value, but there’s often a reason people don’t want them at home: they’re hot, noisy, and suck down a lot of power. You’ll have a lot more options for cooling and controlling power usage with a consumer CPU, and the 5950X provides bags of cores/threads. The trade-off is the patchy (or complete lack of) support for enterprise-grade features like ECC RAM or PCIe passthrough, though these may not be important for something that isn’t mission-critical. The only thing I’d add on the hardware side is that I’d usually prefer networked storage to live on its own dedicated box. You can run things like TrueNAS Scale on a virtual machine, but you may have to piddle about with things like PCIe passthrough for storage controllers. And if other VMs are dependent on networked storage then you may have issues in terms of the order in which the VMs boot. About the only other thing I could recommend is to use Proxmox as a VM/container hypervisor. Broadcom has killed off the free tier for ESXi, which was often the hypervisor of choice for homelab tinkerers.
Suggestion buy a QNAP 5950x does do pci-e passthrough, it also has a nice Eco mode that keeps power down but still runs decents clocks on all threads, not sure about the ECC RAM but the onboard memory controller of 5xxx does a lot of that error correcting anyway, if you need ECC the PRO series of CPU support it and can be had cheap, downsides are of course you can't really get good memory speeds, I think mine run at 2933MHZ CL22 it was 3200 but dropped down once I put 4 sticks of ECC in and the only sort of speed bump I could get out of it was a Cas latency change which when my 5950 was running 3800CL16 most of its life seems a downgrade, in reality you don't notice. 3400G is OK but being a very early Ryzen it won't give you much in the way of PCI lanes if you are going to be utilizing add in cards for passthrough it is less than later chips but just fine if you don't need that functionality. B450 with a 35w PRO is a nice compromise sips power when doing very little, in the teens Watts wise. I ended up buying an x570 server board for mine from Asrockrack, OTT but gave me a lot of stuff onboard like dual 10G NIC, IPMI, lots of SATA and NVMe freeing up slots so I could passthrough for my Nas and router etc. of course downside was a big bump in power use, you don't get something for nothing though. IPMI is great for remote server control, can be achieved with something like PiKVM if you don't have a board with it https://github.com/pikvm/pikvm but perhaps that is not the remote control you were looking for real ballache to get working, and when running getting performance out of its VMs, so getting full speed from router or 20Gb link from NAS NVMes wasn't straight forward, IOMMU groupings were tragic too, frightened to touch it now it does and really should have just bought a Qnap where a lot of the hardwork is done, I'll never know how to get it going again if something fails though having your own datacentre, duplication setup and firing up another node is pretty easy as is moving services with proxmox.
Thanks for the replies guys. Proxmox was always on the cards as a potential base layer to stick some VMs on top of but as you’ve mentioned, isolating PCI-E lanes and allocating hardware to dedicated VMs isn’t always that straight forward and from experience can be a bit of a pain. I’ve spent the evening putting together a small 3U *400mm deep server chassis with the B450i and 3400G. I’ve thrown on Ubuntu as a test bed to see how everything functions before settling for anything else, it’s also allowed me to drop the power down significantly when compared to an 5950x with an A2000. I can migrate that gear in at a later point but at the moment the issue is needing the PCI-E lane for an network card that supports 10G SFP as 2.5g Ethernet I don’t feel will cut it for when I inevitably decided to start moving large files over the network between devices. Anyway, the crux of it is, I will lose my vga when using the 5950x and I absolutely hate running headless systems as when something goes wrong it becomes a pain in the ass pretty sharpish. The HP server has ILO4 which is great but my god does it suck down the juice and make some noise! I’d completely forgotten it has an RDX hot swap and a whole box of 1TB drives which is pretty cool but cool doesn’t pay the bills ultimately. xpenology would have been a gem on that machine if it was a little more “friendly”. The UDM SE arrived so I’ve set that up and have had a play. It needs a modification to one of the internal fans with a replacement for an 40*20 Noctua part, I’ve found the issue with a Google search, it seems like a well documented issue and there’s a good few guides on how to do it so I’ll order up a fan and crack on with that when it arrives. Tomorrow I’ll sort some pics out and will throw up a little bit of progress. In the meantime I’ve been eyeing up a few of those n100 mini PCs as you can find them for as little as £140 each with 8GB of ram and a small 120GB SSD. I’ve got plenty of 16GB sodimms kicking about so a small upgrade there and I may look to cluster three of them. I could even 3D print a nice customer facia for a 1U shelf, we shall see. I know you’ve mentioned QNAP above, how would an Terramaster 1U with 2.5gbe networking fair if I was to ditch the OS and instead run TrueNas? Anywho, that’s enough 2am rambling for the time being. Here’s to a few updates over the weekend. IMG_0729 by andrew8200m posted 23 Mar 2024 at 02:18 IMG_0730 by andrew8200m posted 23 Mar 2024 at 02:18 IMG_0731 by andrew8200m posted 23 Mar 2024 at 02:18
I only mention QNAP as when doing my research they had all the software in built for everything I wanted to do and was GUI based with lots of youtube guides, it may not be for you, rolling my own was very difficult for me with a lack of Linux knowledge and understanding of what is some very powerful software (despite using Linux daily all my working life, I use it like a windows user, it's just a layer of software that does what I need for my applications, I don't need to know a thing about it ) I thinking having it wrapping behind a nice GUI would be easier, I am sure any hardware running software you are happy with will be fine, I think at the time I tried true NAS it did not support all my NICs, the Intel stuff was fine but my 4x 2.5 card from realtek was a no, couple of years ago now when they were shifting from trueNAS to scale and it was a bit sketchy for support, may have improved since? This is the board I use ASRock Rack > Server Motherboard > X570D4U-2L2T you can see in the specs 3xxx G series doesn't have lanes for second slot, you need newer chips. It's onboard IPMI means you don't need a CPU with a GPU though but it adds 7w to power budget to be always on system for control and I've never needed to use it.
Due to the lack of depth on offer in the mini rack I have, I need to keep it ITX with the facility to run an NVME slot to 4x SATA as well as having 4x SATA ports built in. Newer boards generally have the heat sinks built in that aren’t very friendly for this or I need to add a SATA pcie card in and run an APU. Yes I can look in to something with built in IPMI as this would be ideal but when the cost of buying something like that is considered as well as leaving existing hardware redundant, it leaves me in a position of e-waste as well as the disgruntled feeling of buying new when what I have will do the job perfectly well (with some work arounds of course). Having said all of the above, I’ve just found a startech USB3.0 to HDMI adaptor that adds basic 1080p processing and display for £35.99.. it’s looking like I could end up using the 5950x after all but then decode encode isn’t going to be anywhere near as efficient as that of the 3400G when using this particular product. I’ll also be limiting myself to windows only which I don’t particularly want to do. Lastly, I could sell the 3400G system as well as the 5950x system and the look at getting a more modern ITX AM5 platform or something that uses a low end 14xxx i3 from intel. There’s a lot to think about but the cost of moving to the intel or AM5 product would likely be around £550 given how much 64GB of ddr5 4800 is currently. Rambling aside, I’m still playing around with what I have in order to get something together over the weekend that does the job. I can then mess about making decisions on what I need to do moving forwards. I appreciate the feedback on the QNAP. I didn’t like the idea of the synology as it didn’t do all I wanted it to hence why I was looking at xpenology as it’s free all be it a little questionable. The truenas feedback also rings true of some hardware driver complications with the gear I have too as I know certain features on the boards aren’t supported at driver level which would make it also less than ideal. Anyway, I’m not going to get exactly what I want over night and I’ll likely be swapping bits in and out over the coming years. I’m chuffed to have gotten as far as I have with it in the short time I’ve had. Here’s a cheeky finished up mini rack, now to sort the software and functionality out as well as getting the CCTV wired in IMG_0735 by andrew8200m posted 23 Mar 2024 at 15:37
I did that I pretty much set tried all this stuff out on my threadripper and on my 3400g/b450 before deciding yup the desktop AMD platform can do it what I wanted which was at the lowest level just deliver fast storage and run a couple of game servers If you do look for new boards that asrockrack may have you covered, for example they do a variant of my board in itX, it drops a couple of gb lan ports and obviously pci slots but with dual 10Gb NICs onboard, 2 occulink ports for either PCI e or say 8 sata, plus NVMe, IPMI etc. its pretty well featured. one that uses sodimms and one thay uses normal dimms https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T They also have ITX for pretty much any chip whether desktop/server intel or amd, they are not bad value for what they have on board. Looks like its only deep itx for am5 though, their boards are built for deployment in racks being the server side company so heatsinks aren't OTT. Downsides of such high combo boards, is if you want to passthrough some of the onboard hardware, it takes other stuff you might not want out of action, but thats a problem on all these desktop systems for what I gather and only gets sorted with proper server hardware. Other downside you will need a VGA connection to monitor for IPMI if you don't remote through web, HDMI can be used if you have onboard GPU but with a 5950x it might need VGA, that was my experience at least, it wasn't clever enough to auto switch, but perhaps bios has been improved, quite a few new versions since I setuop.
Mini update in terms of power draw. 0.26-0.41 amps depending on what’s happening. That makes for 60-100w for the entire rack hourly and an average according to the logging on PDU of 1.728kwh of power draw. Given my average unit cost of 11.7p that’s a healthy £75 a year in electricity. I can cope with that, a nice result all in
The ASRock board has come across the radar and does seem to be very feature rich however when you consider the cost of the board and the switch to SODIMMs it again ramps up in cost just to use the 5950x. An i5-14500 with an B760i board as well as an add on SFP comes in at a similar price point and I get to sell the 5950x and other bits of kit on rendering the lovely asrock option a bit of a none starter. I have a display port IP KVM that I could use or I could be lazy and just use RDP at this point (xRDP in the case of Linux or VNC perhaps).. I’m happy for the time being. I’ve already come across a few limitations which is great news as routing out the issues now and creating a list so I can upgrade accordingly is ideal rather than getting a few months in to a newer build and finding I’m short with something detrimental. I also like the idea of an B760i with an intel chip due to the drivers being readily available on most operating systems with little fuss.
Not at all. All the hallmarks are there for a cool lab your side, the difference between mine and yours? I put it in a small rack to polish it up a little. A turd is still a turd no matter how you dress it. I have an ex wife so know this all too well
