Scratch Build – In Progress The 1 Million Points Per Day Dream Folding Machine

I do fold. Usually for The Secret Folding Team. A group of friends folding, we are currently growing bigger and bigger. For foldathons I join the Overclock.net team, same with the yearly Chimp Challenge. I fold with two gtx670s currently and building a 4p 64 core amd system. I agree with you that you either need to go 4P xeon (8c16t per cpu). Or go a full on gpu factory. But the hd7990 is not the smart route in my opinion. Certainly not aircooled. You need two gpu cards for 200k a piece. That is not enough. And the power draw is enormous. You would be better off investing in nvidia GTX 780 TI. The Titan already did like 220k a piece alone. But is silly expensive. The TI should be stronger and costs less. You do need a 4C at least as the cpu is needed to "feed" the gfx cards. One or two cores / threads per GPU. So a quad 780 TI with waterblocks and overclocked to the max should be able to do 1 million PPD.

Good luck with your build! A 1 million PPD system is not an easy feat.

 

Hi Hukkel.

Thanks for the input. A 4 processor Xeon setup would probably be much more efficient in terms of its power draw from the wall, but I just don't think that it is a possibility in terms of cost and the project budget, though this should be an avenue that I should explore. Do you have any suggestions in terms of motherboards and CPU's which might come in on budget? Does anyone out there have experience of 4P Xeon computers and their likely costs? Please let us all know.

In terms of GPU's the advantages of the HD7990 are that it is both cheap (ish) (£300 + VAT) and productive (220K PPD). The disadvantage would be possibly its reliability (Unknown quantity) and it's rather massive power draw (375w apparently) I need to purchase an in the wall power meter to check on how much these babies actually use. The GTX780TI has a better use of power and possibly runs cooler, but has the disadvantage of costing a lot more (Cheapest is £420 + VAT) Their folding potential is also unquantified at the moment but they would need to do close to 250K PPD per card to get me to a Million PPD, or I would need 5 of them as well. If anyone out their is folding on a GTX 780TI, please let us all see the figures so we can compare them to the Titan and other cards. Does the PPD productivity make up for the extra cost? I've been having a few ideas as to how I might be able to keep the HD7990's cool under air cooling, but I'll not reveal that just yet! Secrets, secrets. I have to keep a few plans up my sleeve.

 

It's a long shot, but it might just work....

Well, I'm not sure whether this is really worth trying, but I thought that I had better have a go at putting 3 HD7990's into my Corsair Obsidian 550D case. I am currently running two of the cards in there, but spaced apart with the fans on full bore. I'm also running a 10% overclock on both the GPU's and the memory, and they seem very happy and stable in this configuration. I have acquired a third HD7990 from Scan, (They just seem to get cheaper and cheaper) so I'm ready to roll.







I first put the card in Slot No 2 of 4, with slots 1 and 3 already filled.





And fired her up. The first thing I noticed was that the fans were fouling on the card behind, so I had to try and increase the spacing between the cards slightly to eliminate this. The machine booted up OK, though more slowly than usual, but seemed to have a few teething problems in Windows coping with the 3 cards installed. The Folding @ Home interface did not automatically recognise the 2 new GPU's present, so I had to manually install then as 2 new GPU slots. Once I had done this it was then happy to recognise all 6 GPU's and the CPU slot.





I loaded up afterburner and forced the fans to 100%. This was not enough to stop the temperatures from going sky high on the two cards that had no space on the fanward side of the card. This I suppose was not surprising, and I had to an extent expected this, though maybe not the super high temperatures of 100 degrees. I quickly paused all the folding work units to bring the temperatures back down to normal levels, though it was clear that this configuration of cards was never going to work. They were just too close together and couldn't breathe.





Note the yellow lines where the GPU's are over 100 degrees, and the resulting throttling of the GPU to compensate.





Having cooled things down somewhat, I thought for the sake of completeness that I would try the third card in slot number 4. The would allow cards 1 and 4 to breathe but I suspected that the card in Slot 3 would cook. I set it up and again the fans hit the card behind. This was sorted with a plastic spacer, though not very high tech!





This time everything seemed to load smoothly first time and I quickly put all the Folding @ Home units on pause to stop any build up of heat. I again put the fans on 100% and then started folding and watched the temperatures rise once more.





As expected, two of the cards stayed within acceptable temperature ranges, but the third card in Slot 3 again rose to over 100 degrees and started throttling back. Experiment over for this evening as far as I was concerned, as the case was just not capable of cooling 2 cards adjacent to each other even with the fans on full. There was just too restricted an airflow to the card with another card immediately behind it.

So it would appear that we could use 5 HD7990's to get us to a million points, and this could probably be done on budget, but we have some serious airflow, spacing and cooling issues to address. Class dismissed!

 

This is getting on to be a pretty awesome problem solving build, and I really can't wait to see what you do!

 

Water.

 

PCI-E Riser cards are what is needed, they will need to be powered ones as the mobo will seriously struggle to provide the power required through all of the slots once they are full. You can get X16-X16 powered risers, and X1-X16 powered risers for not very much cash.

Next you will need a case that is custom build with a bar 8-10" above the motherboard which you can screw the cards into and keep space between them.

Like this:



You will also likely need to start using Linux, as 5 cards technically means 10 GPUs in the system, and I suspect Windows will throw a complete wobbler and refuse to recognise some of them.

 

Sorry if I've missed something, but why don't you simply watercool them ?
Especially if you will use this setup for long time.

 

When running this amount of such cards you really need watercooling. It also gives you room for a decent overclock which will rise your PPD.

I still think it is not smart to run 3 or 4 dual GPU cards. Maybe more expensive but getting 3 or 4 single GPU cards will use much less power and produce less heat. Your 1050W is not going to be enough.

PS try and remove the cpu thread and see if anything happens to your PPD.

 

Hi There. Thanks ever so much for your comments and positive feedback. I know my "Project Log" is a little different to the normal format, but I was hoping that it would be interesting to some people never the less, so your comments are really encouraging and much appreciated. I'm also pretty blown away with the number of people reading it. Thanks to everyone else who has taken the time to read my waffle! and do feel free to comment and make suggestions. I'm now particularly looking for open air, test bed cases, so let's have your favourites.

 

Hi there. Wowwhhh. Now that's what I call a cool bit of kit!!! I was about to start investigating different types of open air test bed cases, so I think you might have beaten me to the punch there. Whilst looking at other peoples logs I had also come across the riser cards, and thought that this might be a way of lifting some of the cards out of the way to get more air to each card. So I'm currently looking for an open air case with good facilities for modding, and also a decent motherboard with lots of PCI-E slots to take all my cards.

Linux would be a completely new departure for me, so again that may be another bridge that I need to cross later in the project log. I'm sure that there are many people out there who are Windows fanboys through and through, and the prospect of Linux is a bit scary! I'm certainly in that boat.

 

Hi Tarikk. Thanks for the comment. I think water cooling is a distinct possibility especially if air cooling becomes too much of an issue. The big downside of water cooling is the extra cost involved, as each of the 5 cards will need it's own rather expensive water block, and I'm really hoping to stick to my budget. What I am hoping is that I can come up with an incredible folding machine that anyone could put together with just basic PC building skills for the price of just a standard high end gaming PC. I considered that there would be quite a few people out there who would consider a budget of £2,000 + VAT as a reasonable amount for the parts of their new PC, and that is part of the fun of the exercise to try and keep the amount of money to a reasonable level, for normal people to aspire to. If I can encourage a few people along the way to do a bit of Protein Folding and joint our folding team (Currently 7th in the world!) then that would be brilliant. Many people looking at the Bit-Tech site must have awesome computing machines, so why not put them to good use when you are not using them? Preaching over!!!

 

Hi Hukkel. Thanks again for your useful input. Yes I agree that watercooling is a possibility however it would be quite expensive, and air cooling would be cheaper.

I hear what you say about the dual GPU cards, but I have been investigating the PPD of the GTX Titan, 780TI and the 780. As far as the figures that I have seen (See the folding section on this forum) none of these single GPU cards can reach 250k PPD and so I would need at least 5 of them, which would blow a big hole in my budget. This whole project was originally inspired by purchasing a pair of AMD HD 7990's and seeing that they would fold 110K + on each GPU or 440K + all together. That got me thinking about the magic Million mark, and I decided to choose a low budget to make it more fun and interesting, and maybe of more relevance to more people who have a lot less cash. I've only limited funds myself, or the wife starts shouting at me!

You are dead right about the PSU's. I'm either going to need one huge one with the facility to power and supply the five GPU cards and with enough PCI-E cables, or more realistically I'll need two PSU, one to supply the computer and 2 GPU's and the other to supply the remaining 3 GPU's. This will again be new ground for me, and probably many other people too, so that should be interesting. I'll probably go second hand for the PSU's as they can be pretty expensive for a decent high end one.

Thanks for the folding tip. I'll try that out and see what happens and report back. The 8 core CPU folding doesn't produce many PPD in any case. It is the GPU's that produce the points.

 

Well Peta, I have you covered there as well.

A single large PSU would be extortionate, you would need a 1600W and even then it would struggle.

A pair of 850W or so would suffice and be considerably cheaper, you can do one of two things then:

The PSU that connects just to the GPUs could be shorted with a small length of wire on the 24pin connector between the green cable and any black one. This will turn it on and leave it on permanently, and is completely safe to do.

The other option (which I suspect is mire for you) is to buy this: http://www.add2psu.com/

 

Well, in this case I would get 1 or 2 180mm air penetrator fans and try to blow some more fresh air directly to the cards.

But not sure if it would help at all.

 

Thanks for the tip. I think that I will investigate air cooling first off, as I think that it will be the cheaper option and the 5 x HD7990 graphics card will take out most of my budget in one foul swoop. I'll look into those fans as they sound promising. Cheers. Pete

 

Open Air Cases and Test Benches.

We've established that things rapidly get too hot to handle in my Corsair Obsidian 550D Quiet case, though it is not very quiet with the side off and two HD7990's with the fans at 100%! So I've been looking at options available for open air cases and test benches.

Antec Skeleton Open Case.





This is a great looking piece of kit and apparently is a popular choice for an open case. However, I've quickly ruled this case out for my particular needs due to the fixed ribs and fan assembly over the motherboard area. I need this area to be open with lots of vertical space above it, where I can build some sort of framework to raise some of the graphics cards. So whilst it wins on good looks, it loses out on moddability above the motherboard area.


DOMA Pro PCI.


This looks to be a really basic bit of kit and is definitely at the "No Frills" end of the market. It doesn't exactly do it for me in the looks department, though I can imagine that it is fairly functional and workmanlike in its usage. It is made from cut acrylic panels with no metal frame to support it.














As a case for my folding machine I just don't think that the structure will have the strength to take the mods and weight of the graphics cards that it needs to take. It also doesn't excite me at all, and I want a bit of wow factor if at all possible. Cheapness isn't everything in this particular case.


Microcool Banchetto 101 Chassis.

When I saw this test bench / case I was genuinely gobsmacked by its killer good looks. It certainly floats my boat visually, and is a lust worthy, quality object in itself.
















I would certainly love the opportunity to work with this case, but that became a non starter when I found out the price of about £300, without any of the extra bells and whistles either. Once I have won the lottery, this will definitely be my test bench of choice!


HPTX Tech Station Chassis.

This looks a well made, quality item that looks like it would lend itself well to modding and further development.










I think that this case would be seriously in the running as I like both the looks, the quality and the functionality, but unfortunately it only seems to be available in the USA, retailing at about $200ish dollars.

 

have you also looked at




http://www.scan.co.uk/products/aero...case-e-atx-07mm-usb3-20cm-red-led-fan-w-o-psu

looks great for what you want to use it for

just replace the fan with
http://www.scan.co.uk/products/180mm-silverstone-sst-ap182-air-penetrator-series-case-fan-white

and you should be good to go

 

Don't be a *****, Man !



Made in Ukraine. 85USD

 

Hi Lilgoth89.

You beat me to the punch! I called it a day yesterday and went to bed without finishing my review of open cases. I had two more to review; the Lain Li Pitstop PC-T60 open air test bench, and you've guessed it; the Aerocool Strike X Air. I was saving the best two till last. I'll still do my review of both, just so people can see what is out there in this style of case, as this Project Log is all about being informative and discussing the issues. Sure, I could speed things along a bit, but what's the fun in that. Many thanks for your input and taking the time to write a post. I really appreciate it, as it's great to see people reading the Project Log, and hopefully finding it interesting. Cheers. Pete

 

Hi ALS. Yes you are right, maybe I am not being adventurous enough. Maybe I should be going for the 2 Million points Per Day Folding machine. Your photo certainly looks like it might be capable of that! Those cards look rather too close together to stay cool if they were HD 7990's. They really suck up some air and dump out some heat. I have turned the radiator off in the room that I have the computer in, and I have to leave the door open to let the heat out. And that just a pair of them. 5 together is going to be volcanic! Thanks for your post though. It is interesting to see racks like that, and what other people have come up with as solutions to similar problems. Much appreciated. Cheers. Pete