[help] advice needed to build FAH rig

[ari2asem]

i am totally noob to FAH. but i would like to build a rig of 7 or 8 gpu's.

my idea is to use mining frame to house so many gpu's, with about 4-5cm space between each gpu. meaning open frame.
this rig would be ONLY used for FAH, no any other projects (like WCG, Rosetta@Home or else).
trying to keep costs as low as possible, meaning: usage of second hand (used ) hardware parts (gpu, cpu, mainboard, ram).

my question:
- has the cpu core-clock-speed much effect on folding-speed? ( i now know 1 gpu needs 1 cpu thread)
- has ram memory speed (ddr3 or ddr4) (2100MHz, 3200MHz) effect on folding-speed?
- for gpu-cards....what is more important, number of shader cores? or gpu core clock speed?
- which brand (amd or nvidia) are better suited for FAH, meaning faster folding ?
- regarding 7-8 gpu's....most mainboards with 7x pcie x16-slots are using 2 PEX PLX switches for mostly x16-x8-x8-x8-x8-x8-x8 configuration. have 2 PLX switches any negative effect on folding speed?
- or should i build 2 systems with amd threadripper tr4 sockets, so having totall of 5 gpu per system?

so, what are your thougths?

[X-Wing]

I'm also new (comparatively), but from what I have read, my advice is as follows:

RAM doesn't seem to have much of an effect as long as there is enough to support your OS + whatever you want to do, but I would say no less than 8GB DDR4 @ 2133MHz.
As far as the CPU goes, I would say that since you're going to need a huge one already, I wouldn't worry about folding on the CPU and make it simply feed the GPU's. Given that you're going to need at least an 8-core CPU (or 4 core with Hyperthreading), the stock clocks shouldn't matter much as most high-end chips are overclockable, be careful with that though, as you could damage your CPU and motherboard if you don't do it right. I think anything that's 3GHz+ should be fine though.
GPU: AMD is cheaper, but uses more power and has lower performance per watt. Nvidia uses less power/better performance per watt, but is more expensive. I personally went with a 1660Ti from EVGA (XC Ultra) because it was in my sweet spot. Your system is going to draw huge amounts of power, so performance per watt needs to be a big factor for you.
As for everything else, I don't have enough info to help out.

Hope this helps!

[bruce]

FAH's official position has always been that you should fold with what ever equipment you have ... don't build a FAH-Only system ... but of course they like it whenever people have bigger/more powerful/faster systems.

None of the questions you've asked have an absolute Y/N answer... but most have a commonly offered answer where the opposite answer isn't very significant.

* CPU core-clock is relatively unimportant as is RAM speed. FAH doesn't use a lot of main ram so you'll probably automatically have enough to avoid paging.
* Multiply the number of shaders by the GPU core clock speed and use that number to compare GPUs.
* Most donors recommend NVidia, but don't count AMD out if you get a good price. This answer also depends on whether you happen to be assigned mostly small proteins or large proteins which varies from time-to-time.

[Nathan_P]

PLX switches should not have any effect on folding speed, only when you drop below PCIe2.0 x8 will you start to see a drop off.

Personally I would recommend 2 system's, getting all those gpu's to work off one board will be difficult at best, I do not recall anyone getting 8 GPU's to fold off of 1 motherboard for years. The last instances I recall were with the GTX 295 cards back in 2009-2010.
Start small - build one and see how you get on, 1 rig with 2-3 gpu's in it can easily complete 30-40 WU a day and earn several million PPD doing so.

I would not recommend 2nd hand hardware unless you know exactly how its been treated, anyone who puts out an advert saying their GPU is not over clocked is not telling the truth - almost all cards boost over their design frequencies. If you must go second hand to keep in cost look at less critical parts of your system, cpu, Ram and a small harddrive/SSD, GPU's as my advice above and for a psu I would just buy a new one, get a decent brand with a long warranty and you won't have to worry for years.

[ari2asem]

thank you all for your answers.

so next questions i have:

is this good enough mainboard for 7 gpu setup? Asus X99-E WS ? socket 2011-3, ddr4 memory, 7x pcie-v3 x16 slots. it uses 2 plx8747 pcie-switches. and when all seven slots are populated, then these are the connections in case of 40 pcie-lane cpu: (x16/x8/x8/x8/x8/x8/x8). which makes max pcie-bandwidth of x4-pcie-v3 lanes.

gpu-part....gtx 1070 here (the netherlands) is at price around 250 euro (used cards, second hand), gtx 1060 around 150 euro (also used cards).
question is: what is more important factor to watch for it? PPD or FP32 flops? meaning to ask: should my choice for a gpu be based on PPD-numbers of FP32-numbers?

my psu is corsair 1200watt.thanks in advance

[X-Wing]

To be honest, I would go with a current gen Core X-Series processor, or better yet, something from AMD's new Ryzen 3000 series (as yet, they are unavailable, so you'd have to wait just a little bit). This is because the new processors (on Intel) will support 44 PCI-E lanes (in the case of the i7-9800X), or on AMD you will get (rumored) 40 lanes on X570 chipset. This also means you'll need a different motherboard.

For the GPU, multiply # of shaders by core clock speed and compare.

[MeeLee]

Have you considered the monthly/annual electric cost of running such a setup?
Regular PCs run best at 4 to 6 GPUs, provided they have enough cooling.
It is cheaper and better to run 4x RTX 2080 GPUs in a regular desktop, than running 6+ GTX cards on a larger system.
For more than 6 GPUs, it is best to go with ultra expensive Xeon or AMD server hardware, which can easily cost thousands of dollars, and is plagued with low support and compatibility issues.

Your best bet right now, for full time folding, is to run a 4 to 6 core Intel CPU (one core per GPU), of up to 3Ghz (more is just luxury), and up to 6 GPUs. Run 2 motherboards for more.
Run with 4GB of RAM, and a small ~32GB SSD per system. No need for more.
The investment should be in the GPUs.
Run linux, and tune the cards for low power.
6x GTX 1660s will cost you about $1320, with $550 a year on electricity, and do about 3M PPD
6x GTX 1660ti will cost you about $1620, with $600 a year on electricity, and do about 4-4.5M PPD.
6x RTX 2060s will cost you about $2100, with $900 a year on electricity, and do about 6M PPD.
6x RTX 2080s will cost you about $4200, with $1000 a year on electricity, and do about 8.4M PPD
6x RTX 2080Ti will cost you about $7200, with $1500 a year on electricity, and do about 12-13M PPD.

It would be a waste getting your hands on 12gpus and some cheap hardware, only to find you're not able to actually cover the cost of running this system.

If you're interested in performance per dollar, don't go lower than GTX 1660 graphics cards.
GTX 1660, 1660ti, or RTX 2060s are about as good as you can get in a system, without breaking the bank on initial purchase.
Older second hand cards use a lot of electricity, aren't as future proof, have often been abused (Bitcoin mining).

It really depends on what your initial budget is, what your annual budget on electricity is, and how efficient you'd like to fold.

For 24/7 folding machines, latest generation NVidia graphics cards and Intel 6th gen or up CPUs are recommend, due to their superior power handling and efficiency.

- Super efficient (RTX 2080Ti), means Extreme high PPD per watt, bit at a high initial cost. So high, that you end up paying more money than what the card can gain you back on electricity savings in it's lifetime. The initial purchase price can never be recouped. You buy these cards for high PPD count.

- Efficient, (RTX 2080), These cards still run quite efficient. The initial cost of purchase, can be recouped after a good 6 years in electricity, vs RTX 2060.

- Sweet spot (RTX 2060), Best trade off between PPD and $$$

- Lower efficiency (GTX 1660/1660ti), lower initial cost. Recommend if you plan on upgrading in 2 years, as an RTX 2060 makes more sense for folding 2+ years.

- Lowest efficiency (GTX 10 series cards, and GTX 1650, as well as modern AMD Vega cards). Good if you want to squeeze the remaining life out of the hardware.
They're good for their lower initial cost, but with their low PPD, and higher power consumption, aren't very good candidates for permanent folding. It often makes more sense to sell these cards as soon as possible (the sooner, the more money you'll get for them), and purchase less but more modern cards instead (which run higher PPD and lower electric cost).

There are some people who run their rigs off of solar power.
In such cases, low initial cost is a bigger factor, than running cost. They may lean towards GTX 16 series cards, or lower, provided the solar panels are providing enough power to feed them.

[ari2asem]

To be honest, I would go with a current gen Core X-Series processor, or better yet, something from AMD's new Ryzen 3000 series (as yet, they are unavailable, so you'd have to wait just a little bit). This is because the new processors (on Intel) will support 44 PCI-E lanes (in the case of the i7-9800X), or on AMD you will get (rumored) 40 lanes on X570 chipset. This also means you'll need a different motherboard.

For the GPU, multiply # of shaders by core clock speed and compare.

if you can find for me ryzen-mainboard with 7 pcie-slots, than i would pay you 1000 euro's -:)
so, Amd ryzen-3 is not an option for 7 gpu's connecting to mainboard

then i should maybe choose x299 intel chipset and socket 2066

[MeeLee]

I would not choose AMD period.
Not for CPU, nor for GPU. (at least not if you're planning on purchasing hardware specifically for FAH).
And at least not until PCIE 4.0 comes available. And even then, see if PCIE 4.0 is really beneficial or not...

I would also not recommend going for the latest hardware (intel Core i 9th gen cpus), if you're going to run Linux.
Not much is known if Linux supports these CPUs yet, at least not 18.10, which is the latest version of K/L/X/Ubuntu supported by FAH.

[ari2asem]

as i understand correct, better is cut the cost of CPU and mainboard and buy new GPU-cards. ok, smart advise.

i guess i am going to choose for RTX 2060 MSI Ventus XS https://www.msi.com/Graphics-card/GeFor ... cification.
Ventus XS version because it is 10mm shoter than MSI Ventus 6g oc.
price Ventus XS is around 370 euro (new, not used card)
Price Ventus is around 375 euro (new, not used card)

my mainboard and cpu....not yet decided

time span when i expect finishing this build: hmm....4 weeks ??

[MeeLee]

I have found the cheapest RTX cards to be performing about the same as the more expensive ones.
Most of the time, the more expensive ones, offer higher boost clocks, and higher overclocks, but if you're running them on a power limited profile, all that doesn't matter.
Most RTX 2060 GPUs will run well above their boost clock at 130 Watts (all the way to 1935 Mhz and some even more).
Overclocking at high power limits is only good when you want to create peak performance tests. For continuous use, it's better to run the cards at a lower power profile (with overclock); they'll run more efficiently.

Make sure you have sufficient cooling on your system!
I mount a case fan in front of the cards, removing the hot air, creating a fresh air stream.

You can also alternate between one blower type card, and one open air cooled card like the one you mentioned above.
Sometimes blower cards are cheaper because they run hotter, run less high overclocks, and they do make a lot more noise; but they fold just as fast as the other cards, and with a power cap, also overclock just as good!

I think the main issue with GPUs is, what kind of Customer service can you expect from them?
Since EVGA is located in USA, their customer service is awesome for US residents!
MSI, Gigabyte, Asus are secondary over here,
and PNY and Zotac are Chinese and hard to get any CS from them. Something breaks, and it's on you.
Though most of the time, it's cheap and easy to find a replacement fan (fans are the first one to go usually on cards like these).

[Nathan_P]

To be honest, I would go with a current gen Core X-Series processor, or better yet, something from AMD's new Ryzen 3000 series (as yet, they are unavailable, so you'd have to wait just a little bit). This is because the new processors (on Intel) will support 44 PCI-E lanes (in the case of the i7-9800X), or on AMD you will get (rumored) 40 lanes on X570 chipset. This also means you'll need a different motherboard.

For the GPU, multiply # of shaders by core clock speed and compare.

Core-X is going to cost lots for a 44 lane model;, an alternative would be a threadripper based x399 board with 4 x16 slots and a 1900x, cheap cpu, decent mother board, small SSD and then GPU's to suit.

Using AMD threadripper will also help avoid issues like this:- viewtopic.php?f=80&t=30682 Its not clear from teh thread if the problem was ever fully fixed or not.

[ari2asem]

pcie-lane-wise is threadripper is much and much better choice than any hedt intel solution.

rigth now i have 3x threadripper builds, running gpugrid.net and wcg.
@nathan_p, i am aware of the problems with 1080 Ti and x299.

next week i will get my stuff (asus p9x79-e ws with 7 slots pcie and 2 plx8747 switches, e5-2690 v2). rigth now i have 6 cards gtx 1070 doing nothing. at the end of the week this test setup will be launched, just for testing. will run test for about 1 week to gather enough data to make my choice for which mainboard and cpu.

i will give you update about my progress.

and...thanks everyone for ideas, suggestions and inputs -:)

[toTOW]

You should read this tread about motherboards with PLX chips : viewtopic.php?f=80&t=30682

[FldngForGrandparents]

I wrote this post specifically for what you are looking to do: viewtopic.php?f=38&t=28869

This is my eighteenth year folding so I have a little experience. I have two solid machines running with the AMD FX-8350/8370 CPU's for years now. They are cheap to acquire. Then I use the Gigabyte 990FXA-UD3/UD5 motherboards. On the UD5 I am running 5 Nvidia GPU's. (1080Ti, 1080,1080,1070,1070) with a 1300w PS. You do want a faster CPU clock as even on Linux it still takes advantage of it. Hyper-threaded CPU you would want to turn off as it actually impacts folding if you use multiple GPU's. My power bill for all 11 GPU's and machines is about $220/month.