1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Uni's Rack Farm [updated thread structure 2 accomodate new 20K character post limit]

Discussion in 'bit-tech Folding Team' started by Unicorn, 9 May 2009.

  1. Unicorn

    Unicorn Uniform November India

    Joined:
    25 Jul 2006
    Posts:
    12,726
    Likes Received:
    456
    Please note that because the forum now only allows a post length of 20,000 characters, I have had to cut this OP and add the updates from it to the end of the thread. They are linked below, at the end of this first post. All future updates will be posted in this way.

    Current Farm Specs (GPU's only);

    Geforce 8800GTS 320MB - 345.6 Gigaflops
    Geforce 8800GTS 320MB - 345.6 Gigaflops
    Geforce 8800GS 384MB - 396.0 Gigaflops
    Geforce 8800GS 384MB - 396.0 Gigaflops
    Geforce 9800GT 512MB - 576 Gigaflops
    *NEW* Geforce GTS 250 - 705 Gigaflops
    *NEW* Geforce 8800GS 384MB - 396.0 Gigaflops
    *NEW* Geforce 8800GS 384MB - 396.0 Gigaflops

    Total - 3556.2 GigaFLOPS / 3.5562 TeraFLOPS

    ---------------------------------------------------------------
    I thought since we (Read: I) have been discussing this left right and center in some way in several threads both here and on the old CPC forum, I would consolidate what I have worked out so far into one single thread, and then any questions that I or any of you have to ask can go here as well. I plan on making this a project log once the build is underway, but for now it is in the planning stages. Hopefully by the time I come to build it, I will have planned everything well enough to make sure that I don't hit any problems.

    Outline of the project
    For those of you who don't know what I'm talking about; I have been planning to build a Folding @ Home farm for the past month or so. The loose definition of a farm is "The concept of taking two or more computers and organizing them to work together". At the moment I plan on including 4 rigs in the farm dedicated to folding, with another acting as a large capacity file server. The GPU2 client will be what the farm will run mainly, although I also will run notfreds VMware SMP client on whichever CPU's will support it to obtain maximum points.

    The cases
    I had originally planned on using 19" rack cases the same as what Nitteo's farm is built in, they are these from Xcase although in his footage and stills you will notice that he has removed the locking front panel presumably for better airflow. Nitteo has a very impressive farm and I want to mimick his in some ways, but in others I want to employ my own design features and put my mark on it.

    I also like the setup that Atlas Folder has, and although it's a very neat and well put together farm, it is built in open frame type cases, which would mean lots of dust and dirt and not much directional airflow over hot components in my case. I can only assume that Atlas Folder runs his farm in an Air Conditioned, clean air server room and for this reason the open frame design of his farm makes sense.

    Saspro showed me these, which are available with the option of a 19 PCI slot back plate. This feature, as he rightfully pointed out, brings up the possibility of having 4x dual slot graphics cards in a rack case, which would be ideal because most dual slot cards have coolers that exhaust hot air out of the case, keeping the internal temps lower. I think that these cases are ideal for my folding rigs. They have 2x 120mm fans to draw cool air in at the front, which have the additional benefit over the intakes on the XCase XC RM 400/10 of having dust filters. The Compucase also has an optional 14 slot backplate, which will allow the case to hold 4x dual slot PCI-E graphics cards, or possibly more with the right motherboard and some tweaking. I had placed an order for one XCase XC RM 410 yesterday (08/05/09), however after seeing the better equipped Compucase I changed my order to one of those instead. This case is going to house 2x 8800GS's and an A64 X2 6400 Black Edition in an MSI KA92 Platinum for now.

    Hardware list - currently owned – Updated 09 October 2009
    This is a quick list of what I currently have to put into the farm. I will be adding to this once the cabinet is started, although I still don't know what exactly I want to end up with in there when all is said and done. Needless to say I will be looking for bargains on nvidia graphics cards between now and August! What's below is what I currenly have running for the Chimps Challenge (as of 07/05/09) and will not necessarily be what makes it into the farm in the end (i.e. my lowly single core media PC - that's just a host for the 9800 until my new KA92 comes!).

    Rig 1
    2x 320MB 8800GTS heavily OC - 720 core, 1010 memory (achieved with Accelero 8800 Xtreme's & Aircon)
    C2D E7400 @ 2.8Ghz – Soon to be changed for a Q6600 because the E7400 does not support VT
    2GB OCZ PC2-8500 Reaper HPC RAM
    Gigabyte GA965P-DS4 Motherboard

    Rig 2
    4x 384MB 8800GS's (thanks JackOfAll! ;))
    Athlon 64 X2 3.0Ghz
    2GB OCZ PC2-8500 Reaper HPC RAM
    MSI K8N Neo Platinum Motherboard

    Rig 3
    512MB 9800GT - Stock (have an Accelero Xtreme 9800 for it so I can OC, not yet installed)
    Athlon 64 single core AM2 (lol) 2.6Ghz
    2GB Corsair RAM
    Micro ATX Biostar A740G M2+ Motherboard

    Rig 4 (all updated)
    512MB Geforce GTS 250
    C2Q Q6600 - OC to ~ 3.8Ghz
    4GB OCZ PC2-8500 Reaper HPC RAM (1066Mhz)
    Biostar TPower I45 motherboard


    That's all I got for now, I will update this OP with more info (and date and time stamp each update) as the project progresses, for now if anyone has any suggestions or comments on what I have already said, please send them my way! More to come later tonight on what cabinet to use, location of the 6' tall farm and cooling. Gotta get back to work now, it's taken longer than I thought to write all that!


    Useful links within the thread:

    Thanks to Phoenicis for his contribution on prifinitty. A quick guide for beginners is here.
     
    Last edited: 20 Oct 2009
  2. SazBard

    SazBard 10 PRINT "C64 FTW"

    Joined:
    28 Apr 2009
    Posts:
    328
    Likes Received:
    5
    there are some nice vids on you tube of someones folding farm racks. Have a look to get some ideas :)
     
  3. SkiDave

    SkiDave Minimodder

    Joined:
    16 Mar 2008
    Posts:
    742
    Likes Received:
    56
    :jawdrop: Wow that is quite an impressive challenge! The other guys you linked seemed very serious :eeek: I hope that you have deep pockets! :lol:

    A couple of questions (forgive a folding noob); with folding farms does each computer have OS and clients installed on it? If so how are they all controlled simultaneously?

    Best of luck!

    Dave
     
  4. ArthurBuse

    ArthurBuse CustomBitChimps member

    Joined:
    26 Apr 2009
    Posts:
    260
    Likes Received:
    15
    Yes, to fold on the graphics cards you need Windows on each PC. Luckily, there is a
    free demonstration version of the new Windows 7 Release Candidate.

    To control the PCs, there is a utility called Log-me-in (others are available) which can
    control other machines over a local network, or even over the internet. Another option
    is to buy a device called a KVM switch (Keyboard Video and Mouse) which allows one
    keyboard, mouse and monitor to switch between machines.
     
  5. Unicorn

    Unicorn Uniform November India

    Joined:
    25 Jul 2006
    Posts:
    12,726
    Likes Received:
    456
    @ Sazbard: Yeah I think the videos you are referring to are from Atlas Folder. They are excellent because he explains every detail of his farm in them!

    @ Davetherave: Yep it's going to be quite a challenge, and of course Atlas and Nitteo are two of the best known and most serious folders around these days. I'm not aiming for anything on the scale of what either of them have (because honestly, I haven't got a spare $20K lying around :eeek:) but I should be able to build a farm more than capable of 100K per day, which I would be more than happy with :) About the costs; It will be pretty expensive to do but like I have already said it won't be happening all at once. By spreading the cost of it I can take the sting out of the overall cost of the project ;) Each PC has an OS installed on it yes, and each core that you fold on has a client. So with 4 PC's with 4 cards in each, I will have 4 licenses of Windows and 16 GPU clients running.

    Everything is going to be controlled via a KVM using one keyboard and one mouse, and a single monitor also, but more on that later!

    Just on another note, Atlas Folder's farm is about to get a lot bigger... He just bought what looks to be more than half of Nitteo's farm as Nitteo is winding down his efforts to attend to other things. From the AF website:

    I wonder what Nitteo is doing with that rack and the 11 cases from it? :rolleyes:
     
    Last edited: 9 May 2009
  6. JackOfAll

    JackOfAll What's a Dremel?

    Joined:
    23 Apr 2009
    Posts:
    671
    Likes Received:
    6
    Uni, take a couple of pictures when it arrives, please.
     
  7. saspro

    saspro IT monkey

    Joined:
    23 Apr 2009
    Posts:
    9,606
    Likes Received:
    400
    Atlas actually is using most of Nitteo's kit. Nitteo hit on hard times and sold his farm. Atlas and another guy bought it.

    Unicorn, there are ways of legally getting you the OS's for a much reduced rate (£200 + VAT for 10 copies of Vista, + a shed load more software).

    PM me for more details.

    I'll be interested to see how this is going as I might recase my rigs in to a rack to save space.
     
  8. JackOfAll

    JackOfAll What's a Dremel?

    Joined:
    23 Apr 2009
    Posts:
    671
    Likes Received:
    6
    Saspro, not wishing to steal your thunder, but I can possibly go one better than that. Free copies of Vista business edition. I think I have at least 5 spare licenses that are never going to be used, depending on how many Uni needs.
     
  9. saspro

    saspro IT monkey

    Joined:
    23 Apr 2009
    Posts:
    9,606
    Likes Received:
    400
    Even better
     
  10. Unicorn

    Unicorn Uniform November India

    Joined:
    25 Jul 2006
    Posts:
    12,726
    Likes Received:
    456
    Yep I will definately get some pics of it when it arrives and post them as an update, as I will be moving the 2 8800GS cards to that case (they're currently in a Lian Li V2000 and its BIG!). I am also getting a small floor cabinet (only 6 or 8u as far as I know) to hold the first rack case until I get the full cab bought, so as soon as I get that I'll get you all pics of that too :)

    The windows licenses would be excellent guys! This was actually going to be a question that I asked in an update tonight (almost finished, coming in a bit). I thought that running XP would be the better option but then I realized that all the 3 or 4 card rigs that I have seen are running Vista. I don't know much about quad card folding at present, but I am assuming that Vista is the only OS that will support 4 cards when not in SLI? Please clarify this for me, thanks!

    JackOfAll, it would be great if you could get me just 1 copy for now, as everything else is running on XP and I have no need for 4 or 5 copies straight away. Eventually I will need 5 licenses for it, as I plan on the farm having 5x quad GPU rigs in it by the end of Sept. Thanks very much for the offer!

    About nitteos farm: I watched the video of the cards that Atlas got from him earlier on this evening, and have to say that it's a seriously impressive amount of hardware! I don't believe I have ever seen as many GX2's in one place :p That lot (and anything else he has bought off N) is going to give him one serious boost in points! It's a shame to see Nitteo go but I suppose you have to remember how much he has contributed to OCN already. I'd safely say he's the inspiration that got a lot of the guys on their team folding in the first place. Fair play to him, I hope that whatever has forced him to stop folding doesn't last.
     
  11. JackOfAll

    JackOfAll What's a Dremel?

    Joined:
    23 Apr 2009
    Posts:
    671
    Likes Received:
    6
    Can't help you on that, maybe some else can?

    Sure thing, I'll PM you on Tuesday when I'm back in the office.
     
  12. JackOfAll

    JackOfAll What's a Dremel?

    Joined:
    23 Apr 2009
    Posts:
    671
    Likes Received:
    6
    Uni, just to lob in a curve ball. Why are we talking about XP and Vista? How about converting you to Linux? :D
     
  13. Unicorn

    Unicorn Uniform November India

    Joined:
    25 Jul 2006
    Posts:
    12,726
    Likes Received:
    456
    Oh no he didn't?! :lol: I have some experience using Linux for coding projects in uni and messing around with it on my own machine, but have never tackled it for folding. I'm willing to give anything a try, and I don't see why I couldn't use it in the farm. I would need lots of guidance from you on that though! If you're willing to show me how it's done (F@H in Unix that is) and if there is no performance sacrifice, then I would be willing to give it a go!

    Btw I updated my OP tonight... It's getting looong :p
     
  14. saspro

    saspro IT monkey

    Joined:
    23 Apr 2009
    Posts:
    9,606
    Likes Received:
    400
    GPU on Linux is possible bit complicated. XP or Vista are the easiest. (I generally use Vista 64bit due to XP32's 4GB memory limit and drivers).

    Unicorn - I hope you're feeling rich. To power your rack for a few hours with a UPS is going to cost thousands (if not tens of).
    I have a APC 3000 on mine and that lasts long enough for them to shutdown safely.
    I use WOL to turn them back on if needed (I'm usually need the server room)
     
    Last edited: 10 May 2009
  15. Christopher N. Lew

    Christopher N. Lew Folding in memory of my father

    Joined:
    23 Apr 2009
    Posts:
    1,358
    Likes Received:
    46
    I use Win XP 32 bit on all my machines. No problem with folding 4*8800GT or 4*9800GX2. Using drivers 181.xx frees up the CPU for doing more folding - drivers prior to that showed the CPU as 100% busy. XP used to be easier than Vista because of the issue of dummy plugs required by Vista. I believe from other postings that CUDA 2.1 and above have 'fixed' this.
     
  16. JackOfAll

    JackOfAll What's a Dremel?

    Joined:
    23 Apr 2009
    Posts:
    671
    Likes Received:
    6
    Swings and roundabouts, in as much as you'll gain PPD on the CPU smp a2 core WU's, which are not available on Windows, but lose a little PPD on GPU2 work units by virtue of needing to running them via the wine layer.

    I'll have a think about advantages and disadvantages, put together a list and you can decide if the benefit for you outweighs the cost. Two immediate disadvantages that come to mind are GPU fan speed and upping the shader clock. The Linux equiv of rivatuner, called nvclock, is not as advanced. I know it is only possible to change fan speed on 'earlier' series cards. Overclocking shaders is not really possible 'on the fly'. Most of the people I know who do this under Linux, modify the card bios and then re-upload it with nvflash. I suppose you could argue this is actually a better way to do it, in as much as it survives a reboot. Let me ponder a little more.

    The nVidia drivers are not really a problem. Most distributions package them up in a distribution friendly way. Installing the drivers, a CUDA toolkit into /usr/local and the wrapper dll, doesn't take very long. I reckon I'd do very well in a race to get a machine up and folding, you starting with a Vista CD and me starting with a Fedora CD.

    For a farm, I see Linux as being an advantage. For a start, in a headless environment there is no need to run any sort of GUI or XServer. I'd probably run up my CPU and GPU clients using a single instance of 'screen' and then be able to ssh in from any other location to look at or control the individual console windows. You can export the directories with either nfs or samba, the latter probably better if you're going to monitor from a Windows machine with fahmon.

    I'll give it some more thought and we'll see if anyone else has any overwhelming reasons for you not to use Linux for the farm. Whilst I'm an open source advocate, I believe in using the best tool for the job, be that Windows or Linux. Linux may well be the best tool for your rack farm if you can deal with the learning curve. Not that that should be a problem if you already have a little experience. Of course, I can shut the hell up (if you so desire) and just sort you out a M$ license! ;)
     
  17. phoenicis

    phoenicis Retired Chimp

    Joined:
    26 Apr 2009
    Posts:
    493
    Likes Received:
    9
    This is fascinating stuff guys both in terms of the rack project and the subsequent discussion on Linux vs Windows. I may have dabbled in running SMP on Linux under VMware but Linux GPU folding and a rack farm is way beyond my current skill set and therefore it sounds like fun and really interests me :)

    It appears the only real drawback for me in moving towards GPU under Linux is, as you mentioned JackOfAll, the lack of ability to adjust clock speeds via software. I like to OC shaders a little but then tweak GPU settings to avoid EUEs when a new types of WUs come along.

    Keep up the good work on updating us on the project Unicorn :thumb:

    p.s. Unless they've changed it, Vista can support upto 5 GPUs without losing the plot. I came across this in an early 3 x GX2 rig and had to move to XP to get all 6 clients running. Will not be a problem either way with quad rigs but I'm not aware of any fix for Vista removing the requirement for dummy plugs.
     
  18. manxminx

    manxminx Opps, I broke my Dremel . . .

    Joined:
    9 May 2009
    Posts:
    50
    Likes Received:
    0
    Here's a few off the wall ideas.

    Rather than each PC having it's own power supply, it might be more efficient to have one large industrial 12v power supply feeding everything. Purchase cost may be prohibitive though, unless you can pick one up cheep second hand.

    Also, Google Datacentres don't use UPS. Each server has a 12V battery attached, charged directly from each power supply. It has enough power to shut down the server. No more, no less. That's all that's needed. Simple, cheep, easy to implement.

    And here's a really off the wall idea. Water cool everything, then use a heat transfer pump to heat your hot water/radiators. Reclaiming waste heat is the big new thing in Data centre ideas. There's a new one in Switzerland which is reclaiming all the heat to use in the public swimming pool next door! Might be too expensive to implement on a small scale though.

    Lastly, invest in a wind turbine. It'll cut your electricity costs. If you get one with a 12V ouput it can power your rack (or part of the rack) directly when the wind blows!

    Ali.
     
    Last edited: 10 May 2009
  19. Unicorn

    Unicorn Uniform November India

    Joined:
    25 Jul 2006
    Posts:
    12,726
    Likes Received:
    456
    Lots to reply to here!

    @ Saspro: I have researched this more thoroughly and you are of course right, there's no way I can afford to have a UPS with a large enough capacity to run the rigs once the power goes off... I will make do with either one or 2 UPS's to allow the rigs to shut down safely or investigate manxminx's 12V battery per PC idea.

    @ CNL & Phonetics: Thank's for clearing up Vista & XP support for multiple cards, I now have the choice of Win XP, Vista and Unix. OS choice is shaping up to be the big decision I have to make here, so any further help or suggestions from anyone experienced with quad card folding on any of the three will be welcomed! Thanks for your advice so far JackOfAll, overclocking sounds like a lot more work in Linux than it is in Windows, and most of my hardware is currently overclocked... but then there are the obvious advantages of Linux on a number of headless rigs. I will think about it a bit more in the meantime and do some research on Linux folding!

    @ manxminx: All of your ideas are truly awesome... and I have to admit I want to have a go at as many of them as I can. Industrial PSU's are probably out of my league cost wise, but it's not a biggie as I have enough 600W+ PSU's in stock to build the whole farm at the moment anyway ;) I have been chewing over the idea of watercooling at least the CPU's in the rig for a while now, and wondeting what the best way to go about it is. I want to utilize the waste heat from the farm in some way, but I haven't quite nailed a solid idea yet. I'll keep you all posted on new ideas that I have for this. Lastly, I am considering the possibility of using either a wind turbine or (more likely) an array of solar panels on the garage roof. As I mentioned in an old thread on the CPC forums, my brother works for a heating & plumbing company that supplies and installs renewable energy equipment like biomass biolders, solar panels and wind turbines. I think he could supply them to me for a lot less than a member of the public so I will pick his brains on power output etc and see if it would be worth it :)
     
  20. saspro

    saspro IT monkey

    Joined:
    23 Apr 2009
    Posts:
    9,606
    Likes Received:
    400
    Unicorn - what do you need to know about quad card folding?
    All my farms are quad card.

    I'd also ditch the rack mounted monitor idea for now as you're looking at £500 for a 2nd hand 1u slide out monitor. VNC & Logmein will work in almost every situation
     

Share This Page