Thoughts on this Folding Rig Config?

[jrweiss]

At least based on prices for new stuff, I think that X99 + Haswell Xeon makes sense.

Any glaring issues with using this mobo / CPU / RAM?
Haswell Xeon E5-2603 v3 (+$70)
MSI X99S (+$110)
2x4GB DDR4-2133 (+$15)
CPU cooler of some sort to handle 85W (+$60)
Total cost increase: $255

So, basically in doing this I would be able to run 3 GPUs at X16/X16/X8 (versus X8/X4/X4), and I would either buy 2 GTX 970 cards or 1 GTX 980/980Ti to start. Then down the road I could add more GPUs, up to the max of 3. Maybe I do want to stick with the 1000W PSU if I am thinking about the 980!

I would look closely at cooling air flow. If you're running large GPUs 24/7, you want to make sure you have good air flow. If you "need" a 1000W PSU, you will need to exhaust a significant part of 1000W of heat...

That case has 1 120mm fan included, set up as a rear exhaust fan. Your PSU fan will also likely be set up as an exhaust fan, but at the bottom of the case. The GPU cards will exhaust SOME heat out the back, depending on individual designs.

You will want at least 2 120mm or larger, high-flow intake fans to force air into the case - 1 in front and 1 or 2 in the side. Intake air filters and "positive pressure" (more intake airflow than exhaust flow from fans) will help avoid sucking dust in through crevices. The open top on that case may help natural circulation as well. Many fan vendors will not show airflow specs because the fans are weak in flow or noise or both. I'd invest in 2 or 3 Noctua NF-S12A fans (noctua.at) to get max flow with minimum noise.

You might also reconsider the case and find one that will take 2x140mm intake fans up front.

As long as you don't overclock the CPU, the stock heat sink should be sufficient. I have Noctua NH-D14 coolers on my CPUs, but I run CPU Folding WUs in addition to the GPU WUs. Self-contained water coolers for CPUs are now available and popular, so that would be another option.

[Nathan_P]

No, get the smallest, cheapest ram kits you can find for the x99 platform, the only WU that need gobs of ram are the bvigadv WU and you don't have enough cpu cores to run them.

[bollix47]

There are a few core 21 projects like 10494 that do use a substantial amount of system ram ... I have one running currently and the core is using ~1.5GB of resident ram. Since you're thinking about having 2 or 3 GPUs you could consider 6GB of ram as a minimum.

[Rel25917]

Older versions of core 21 used several gigs during unit start up, new versions don't seem to but I'm guessing as we go for bigger and bigger proteins the ram requirements will only go up. I would go for at least 8gigs for a multi gpu setup personally. You could also go for 4 now and another 4 when you add a card if that's the way you decide to go.

[bmwman91]

Cool, thanks you all for the input and help. I have made my purchases and am eagerly awaiting the goods so that I can get things rolling. A couple of additional changes/upgrades were made since I forgot that I had an extra ~$500 sitting in my Paypal account (yay for cleaning out the garage and selling a bunch of old car parts that have accumulated over the better part of the last decade!). While these upgrades are not going to translate directly/immediately into WU throughput, they should help to ensure that this machine is adequately future-proofed and that it has sufficient headroom to keep chugging for a long time. The upgrades were twofold: 4x4GB DDR4-2133 (vs. 2x4GB), and an E5-2620V3 CPU (vs. E5-2603V3). While this CPU is most likely overkill, I can at least run some SMP units on it, and it looks like it is right at the limit of the sensible performance-per-dollar area for a CPU with 40 PCIe lanes.

So, here is the final system configuration:
Case: DEEPCOOL TESSERACT WH Mid Tower Computer Case
PSU: XFX ProSeries P1-1250-BEFX 1250W (NOTE 1)
Mobo: MSI X99S SLI Krait Edition LGA 2011-v3 Intel X99 Motherboard
CPU: Intel Xeon E5-2620 v3 Processor, 85W TDP
CPU Cooling: DEEPCOOL Gamer Storm Lucifer V2 CPU Cooler 6 Heatpipes 140mm Silent PWM Fan Support LGA 2011-v3 (NOTE 2)
RAM: G.SKILL Ripjaws 4 Series 16GB (4 x 4GB) 288-Pin DDR4 SDRAM DDR4 2133
GPU: ZOTAC ZT-90503-10P GeForce GTX 980 Ti AMP! 6GB 384-Bit GDDR5 PCI Express 3.0 SLI Support Video Card
HDD: (FREE) Kingston 240GB SSD. This has been laying around for quite some time. It was a freebie that I got with an eBay laptop purchase.
OS: Ubuntu or (FREE) Win 8.1. (NOTE 3)

NOTE 1: Yes, overkill. No doubt. Two reasons. One, it has a $30 rebate AND a 10% discount deal that ended last night. Two, since this will be running continuously for an indefinite period, I want plenty of overhead so that it is never fully stressed. Additionally, the plan is to run 3 x GTX 980Ti or Titan cards (eventually).

NOTE 2: The CPU does not come with a cooler. This seemed to be a decent deal for a 140mm HSF setup.

NOTE 3: I have a spare copy of Win8.1 Pro, so I might just go with that rather than Ubuntu. Drivers will be a heck of a lot easier to deal with in Windows than Ubuntu.



That leaves thermals. With the single GTX 980Ti card I am starting with I assume that it is not a huge deal, but I will try to use sensible cable routing and air flow since the plan is to eventually run 3 GPUs. I bought 3 inexpensive 120mm fans to go with the one included in the case. I plan to put 2 of them in the side to suck air in and blow directly onto the CPU and GPU, and two in the top to blow air out. It may take some experimentation to get things dialed in nicely. Push/pull combos can just as easily hurt performance as help since flow mismatches between fans can be troublesome. This case will then have open vents in the front and rear, and I may play with blocking those off to see if I can create more of a "wind tunnel" between the side intake and top exhaust. The Zotac GPU I am using has 3 90mm axial fans and while there is some perforation in the rear metal plate, I may mess with blocking that as well. CPU and GPU core temperatures will be the variable that I monitor in my cooling experiments, and I will do my best to document that little project in here.

So, this is exciting. I have not built a rig in close to 10 years, and I certainly did not have much money to do it with back then (broke college student with loans). It should be lots of fun!

[bruce]

Most PCs have a reasonable amount of main RAM. FAH is comfortable with 2GB or 4GB Paging can manage a lot of inactive programs even if they're listed as running when they don't have anything to do. The real requirement is based on the total of the working sets of all programs which are busy processing at any one time.

That depends on how you measure "WU throughput" Most people pay attention to PPD (points per day) and/or the historical total of their points.

Baseline points are established for each project. Then bonus points are added based on the total time from download through processing and including uploading. Adding a minute or two for processing initialization will be a pretty small decrease in total point. for WUs that take many hours or many days.

[7im]

The old recommendation was to allow for whatever runs the OS at the best speed, then add enough for FAH. Obviously the versions of the OS have changed over time. 2GB for XP, 4 GB for Vista, at least 4 if not 8 GB for Win 7. Win 10 seems much like Win 7. Linux hasn't changed quite as much. I just always mirrored the Windows equivalent at the time.

Same for FAH, but now with the multiple slots and client types, it grows even faster and is more complicated.

[jrweiss]

That leaves thermals. With the single GTX 980Ti card I am starting with I assume that it is not a huge deal, but I will try to use sensible cable routing and air flow since the plan is to eventually run 3 GPUs. I bought 3 inexpensive 120mm fans to go with the one included in the case. I plan to put 2 of them in the side to suck air in and blow directly onto the CPU and GPU, and two in the top to blow air out. It may take some experimentation to get things dialed in nicely. Push/pull combos can just as easily hurt performance as help since flow mismatches between fans can be troublesome. This case will then have open vents in the front and rear, and I may play with blocking those off to see if I can create more of a "wind tunnel" between the side intake and top exhaust. The Zotac GPU I am using has 3 90mm axial fans and while there is some perforation in the rear metal plate, I may mess with blocking that as well. CPU and GPU core temperatures will be the variable that I monitor in my cooling experiments, and I will do my best to document that little project in here.

You may want to rethink the thermals... You should ALWAYS have more intake fan air flow than exhaust flow. Then tailor the remaining case slots to direct excess air where you want it.

With an 85W CPU and a single GPU, you're not likely to run into problems. However, you should learn a lot from the single-GPU setup before you add more of them.

The PSU is good. Its fan won't likely come on at all with the single GPU.

The orientation of the fins under the coolers on the GPU does not appear to encourage venting the hot air out the back, so most of it will have to go out via the exhaust fans. The cooler fans will suck air in and push most of it out the "top" of the card (long edge away from the MoBo). That might tend to fight a fan blowing directly onto the card from the case side.

I would place the intake fans on the front and the lower side mounts; cover the upper side fan slot (I use clear packaging tape on he inside of the case). Hopefully that will put the most positive pressure between the GPU and PSU. Remove the lower HDD cage if possible. If you can put the SSD elsewhere (in the top 5.25" bay?), remove the upper drive cage as well. Run the front fan at full speed; the side intake and top (or rear) exhaust could be PWM.

The orientation of the CPU cooler will depend on whether you decide to put the exhaust fan in back or on top. I believe 1 PWM fan will be sufficient with the 85W CPU. Set it up at the bottom (or front) of the heat sink, blowing into the heat sink. Put your case exhaust fan directly opposite the CPU cooler fan, to encourage direct flow out the case. The pic in the Deepcool gallery for the case shows CPU fan front, case exhaust fan rear; better option may be to rotate the cooler 90 deg and put CPU fan below (blowing up and into the heat sink), case exhaust top forward. Cover the back case fan slot and keep the aft top slot open.

[bmwman91]

Thanks. Yes, I suppose that I will need to look at how the GPU cooler is exhausting air and try to work with it rather than against it. I'll try arranging the fans as you have recommended.

Unrelated question...the CPU I am using has 6 physical cores and 12 logical ones (hyperthreaded). The folding client will reserve one logical core per GPU slot, at least from what I have seen on the low-end machines I have picked up to fold with over the years. These machines are folding with both the CPU and GPU. Is it better to set things up so that 2 logical cores (1 physical core) are free per GPU? I have no idea if the 2 logical cores that don't get allocated for CPU folding will actually be on the same physical core, so maybe it is a moot point. In the case of the CPU that is coming for my new rig, I imagine that CPU folding will be running on 10 cores anyway since 11 is prime. Either way, I'd like to set things up so that the GPU has as many resources available as it may need. Is there a table somewhere that has the number of CPU cores (physical or logical) that can be used for folding? It sounds like primes don't get used, powers of 2 are ideal and some even numbers don't get fully utilized either. Also, would disabling hyperthreading help/hurt anything, or is it a non-issue with the current client and cores?

[jrweiss]

In my experience, hyperthreading improves CPU Folding somewhat. While the GPU slot needs a CPU core (or a thread), it doesn't actually use it full time - more like 10% of the time (maybe more with more powerful GPUs). The more cores/threads left over for the CPU slot, the better it will [usually] run.

I've been using CPU7 on my quad-cores with HT, but went to 6 with the recent lack of WUs for 7 cores. If it's a dedicated Folding machine, CPU-1 may be the best choice, and let the client figure it out. Otherwise, start with CPU10 and see what happens.

[bruce]

Your OS assigns tasks to each logical CPU and unless you specify affinity, they'll be chosen at random and tasks will tend to float from one thread to another.

CPU-based Folding spends most of it's time on SSE instructions and therefore tends to saturate the FPU. [For simplicity sake, it's generally assumed FAH for CPU depends only on the FPU, but that's not precisely correct.]

If a pair of FahCore_a4 (or other FPU-based analysis) threads happen to land on the same pair of logical CPUs, they'll spend maybe 85% of the time competing with each other. Thus CPU:12 would be about 15% more productive than CPU:6 for a pure FPU-based application such as FAH (provided all 6 land on different pairs of threads and have a dedicated FPU).

Now allocate 2 of your threads to support the GPU leaving only 10 to support the CPU tasks. You'll still have a total of 6 FPU, so some pairs will see competition and others will not. [It's generally assumed that the GPU support tasks use mostly ALU resources and negligible FPU resources (since that has been offloaded to the GPU).] The CPU throughput will be somewhat faster than CPU:6 (with optimum affinity setting) and somewhat slower than CPU:12. bit since we're talking about a range of between 0% and 15%, it's not going to be a critical issue.

EDIT: jrweiss responded while I was typing. My overly technical explanation agrees with what he said.

[Foxbat]

I was recently in a discussion with one of the members of my Folding Team about two GTX 970 cards vs. a single GTX 980 Ti for the same approximate cost. His argument was "it's all about the CUDA cores" and 3,328 shaders (for two 970s) vs. 2,816 is more cores. Plus, the benchmarks over on AnandTech seem to show the 970 performs at 66% of the 980 Ti, so two 970s should be better than a single 980 Ti. I accused him of trying to use Logic on me.

My find was a used Dell Precision T7400 on Amazon for $256 (no O/S), delivered. Dual 3.0 GHz Xeon 4-core (no HT) CPUs with a pair of PCI-e x16 slots (but they're only 2.0, not 3.0), 16 GB of RAM, and 2x500 GB SATA drives in a RAID 1 configuration. Plus, a 1000 W power supply. I installed Ubuntu and while I ponder my GPU options, it's currently crunching Core_A4 projects at 25K PPD on all 8 cores.

[Mstenholm]

There are a few core 21 projects like 10494 that do use a substantial amount of system ram ... I have one running currently and the core is using ~1.5GB of resident ram. Since you're thinking about having 2 or 3 GPUs you could consider 6GB of ram as a minimum.

True. And get a fast 4 core Intel w/o HT or disable. PCI-E lance have very little bearing. You will gain next to nothing by going the X99 way.

[foldy]

His argument was "it's all about the CUDA cores" and 3,328 shaders (for two 970s) vs. 2,816 [for 980ti] is more cores.

This is half the truth it is also about the clock for each shader. And 980ti has faster shaders.

[Nathan_P]

Ubuntu video drivers have come a long way - there are a couple of guides out there that can get you up and running in minutes with just a few commands in terminal, even I managed to get my gpu box up and running without difficulty and I hate Linux with a passion.