Hello, I'm looking at setting up a render farm. I'm a student in an Architecture department and I do work with the computers/plotters/laser cutters. This is kind of a two part question, but please tell me about your experiences with render farms for hardware setup. I know it's Proc ram board and HDD at minimum, but did you build them yourselves, buy from a vendor? Also once that was done any software you loved or hated, maybe even just stuff you've heard was good. Thanks Regards, Brett EDIT: Looking at all price ranges, leaning away from a blade server after hearing some issues. This will be for students to render their animations or images on from modelling programs. The Rendering engines we use are Revit, 3DS Max, and Maxwell.
How large of a render farm is it? If it's <10 machines, I prefer homebuilt machines using consumer parts. We've got a bunch of ITX machines about to be built using completely consumer parts i7 2600k's and 16gb of RAM each. They are about a foot in each dimension, and fast enough for heavy duty professional work. However depending how many people you have, though, a rack-based system might be better for you.
We were looking at 20-ish machines. Budget is assumed to be 20,000 USD. i7 3820, 16GB of DDR3 1666, ATX boards. Will look into the i7-2600s considering the space you mentioned. Also a BOXX setup, albeit that'd be about 6 of those with that price. We've just been getting the ball rolling so the serious hardware investigation is starting.
personally i would go consumer end even at 20 nodes. I've ran maxwell across 3 nodes in the past which was fun, but personally i've been making the workflow switch to blender and soon the new new cycles engine which enables CUDA based rendering, currently its at beta but it works. I've pegged a dream build around the use of two 680's and a cheap i3 just to use as a render box, cycles currently only supports two cuda devices aswell as a CPU so I am waiting for it to mature a little longer and allow for a multi GPU setup as thats effectively up to 4 or more nodes in one box.
Which software did you use to have Maxwell work? If this goes into play we'll be replacing this every so often, so that's good stuff to know with the CUDA based rendering. The absolute max is 30k, and we'd like to stay away from that if possible, so we've got that figured in too. That stuff is indeed expensive.
20k for 20 machines is certainly doable. In fact, the very build I'm using now would work for it, if you take the SSD we're using for heavy shader work out. Here, let me remake it for you, and replace the 2600k's with 3770k's. How much space and noise tolerance do you have to work with?
I've specced out 3820 i7s, 64GB SSD, 16GB DDR3, and some intel ATX board; that gets us 22 for 20k. And space is an issue. if it's small as in ITX scale, not an issue, if its ATX, it becomes more an issue. We've got a server closet with it's own AC system at our disposal, but that's not infinite space. I'd be up for make a custom enclosure, but that does affect the final cost as my hours working go into this too. But please do tell.
I've been playing with Blender and it's built in render engine 'Cycles' which is CUDA enabled. Last time i used maxwell was version 1.7 installed on W7 with the Solidworks Plugin for SW2010. Hardware across the 3 nodes in use was just a mix of my workstation Q6600, and two old boxes which had a E6600 and a Opteron 165. The major headache i had was the network switch bottlenecking the process, (consumer grade sh*t) and me forgetting to store the project in a location which all the nodes could see on the network. Other than those simple mistakes it was easy to setup and enjoy. I've been trying to learn blender as it's come on a long way in the last 18 months, but this video wetted my apetite! And this one
I've rebuilt the spec of the computers to the mini-ITX form factor as per the previous recommendation. It is as follows: LGA 1155 Core i7 3770S Intel Mobo H77 64GB Samsung SSD SATA III 16GB Corsair Vengeance DDR3 1600 Corsair Builder 500W PSU Lian-Li Mini-ITX case The good news (anyone) is 20k gets 25 of those! I specced the 3770S because Newegg only allows for 2 of the 3770Ks to be purchased at once. Also I'd assume that when under load the turbo would kick in anyways to the same clock speed. Other than that it'd be a cooler less power-thirsty idle. Any criticisms? Also, anyone recommend any specific software to manage the farm that allows Maxwell, 3DS Max, Revit, and some others. Really any diverse progams to manage them. Backburner is Autodesk's thing I know. OS can be Windows or Linux, I'd prefer a GUI
how are you managing windows installs do you already have the required software for network install. only thing id also mension is. consumer grade networking is pretty crap for render farms as they reach there choke point a bit easily under heavy use. if you can get business level networking in budget its something you should look at. render prices seem insane, Maxwell Render Standard license is just short of 1k dollars on there own site seems a bit excessive ( per license i assume you need 1 per machine the software is ran on ) ( legally you need 1 per machine ) theres network licenses out there. i think for your budget your gonna struggle to spec the machines you want with the license costs included. if it was me id be setting aside a good chunk of the budget on network and licence costs ( 5-6k minimum ) then work out what you can get from there. dif between 3770s and 3770k is a bit more than just turbo 3770s is a 65 watt cpu 3770k is a 77watt cpu. the 3770s in uk at least is only used in all in one pcs which have to fit a certain power envolope. they also turbo a bit differently enough finding a review that is not a review of the entire system is problematic. 2600k is alot cheaper than the 3770k or S for that matter and would be near enough identical performance for an £20 gap in uk money bit more id imagine in dollars. 20 dollars cheaper on newegg is the 2600k than the 3770s if your buying 20 of them thats 400 dollars which could be put towards networking hardware.
I was thinking an image of Windows 7 and just throwing it onto all of the machines. If that's a problem waiting to happen then please let me know. This is for an educational program, so educational discounts are being counted upon too. We already to have a Maxwell licence set up for our normal lab computers. This is all being managed within our department, but the schools ITS department could help me with that I'm sure. We have an absolute max of 30k, but have made room in the budget for hardware to take up 20k of that only. Thanks for the heads up on the 3770S Thus far I've been finalising the hardware, and this week it'll be more about the software involved. So forgive my lack of knowledge, as I'm only starting to really dive into this.
How big is your server closet? Depending on your choice of cooling etcetera there are 1U enclosures which will hold 2 mITX boards so this might help cut the cost of cases- if you're putting taller coolers on the cpu's then you may need to go to 2U. And if you don't mind getting blank rack cases and putting a bit of work in they you could probably shoe-horn 4 boards per case, though I doubt you'd fit PSU's in (unless you went pico-PSU, which *might* be an option, see below) so you'd need to distribute power somehow. Also I think you've over-speced the PSU's by a massive amount - at a guess 150-200W per system should do nicely and a that low range of power requirements you can get pico-PSU's or other similar PSU's which are tiny by comparison to standard ATX. Added to this you'll probably find plenty of rack mount cases come with adequate PSU's included. And by saving money on the cases and PSU's you can get (or go part way towards) a good quality switch to slot in there. As Burnout21 and rollo already said, no sense spending a lot of money on systems and software and then being crippled by your network speeds. As I see it if you've got a cabinet sitting there at your disposal, why not make use of it, unless it's a tiny quarter size thing as that simply wouldn't be big enough. Unless you need to be able to move the systems around?
Technically that's illegal unless it's some multi install windows copy you would have to check and make sure it's the correct version
Thank you Asura, I'll keep that in mind and rework it all. I was unaware about how much of a PSU I'd need exactly, so I went by brand. Any particulars you'd recommend? And Rollo, we've got a MSDN licence so we have OEM licences for Microsoft software underneath that. We would've be cloning a retail image over many many computers. The software on this will all be on the up and up to our knowledge. Thanks for your input though everyone, this has been refined much more and still can be. Plans are that it'll be on hold til August with the fiscal school year coming to and end and the lack of a budget to fill out for the year.
A quality 400w PSU should be fine, something with a 120mm fan. Seasonic is my preference, but they also make the higher end corsair PSU's.
corsair cx430 its £35 here and iv used it in afew build with no problems http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story&reid=214 $45 on newegg for the v2 http://www.newegg.com/Product/Product.aspx?Item=N82E16817139026
