New RTX3xxx cards

[Endgame124]

Given that SLI is not possible any more with these cards, NVidia assume you won't have multiple cards any more. Those of us building FAH multi-cpu rigs aren't realy a market consideration for Nvidia.

What's bizarre right now is pricing. Nvidia's online store still has 2070S at £489 for delivery now and the 3070 on "notify" at £469! I'm tempted by a 3070 at that price if it lets me retire multiple 1070s.

Even if SLI is supported, how will you connect them?
2 GPUs of ~250-350Watts each, is 500-700W of heat generated in an area smaller than a shoe in size, can't be good.
Most motherboards having 3x full sized PCIE slots, can only use the first and the last one, often the first running at 8x, and the last one running at 4x.
Most modern motherboards run the first slot at Gen 4, and slot 2 and 3 at Gen 3.0 speeds, save for the high end boards of $250 and up.

I would probably put 2x 3090s in a next gen Ryzen system (4950X equivalent) in series with water blocks on both the video cards and the CPU, probably with 3x radiators.

[HaloJones]

Indeed. I have two dedicated folding rigs that do nothing else. They both use Phanteks P600S cases. One has a 280 and a 420 and one has a 280 and a 240. I'm strongly considering getting two 3070's and replacing the three 1070s and TitanX which together produce around 3.5-4m ppd and consume about 700W so 2x3070 would only be around 450W. The big question is how well will they perform for FAH. Only time will tell.

[markdotgooley]

Indeed. I have two dedicated folding rigs that do nothing else. They both use Phanteks P600S cases. One has a 280 and a 420 and one has a 280 and a 240. I'm strongly considering getting two 3070's and replacing the three 1070s and TitanX which together produce around 3.5-4m ppd and consume about 700W so 2x3070 would only be around 450W. The big question is how well will they perform for FAH. Only time will tell.

I have a much smaller setup with a 2060 and a 2060KO on the same AMD motherboard in a Phanteks Enthoo Pro 614 case that doesn’t keep them particularly cool (the 2060 is next to the 750W power supply and runs around 76C, the KO higher up and usually 82C). The motherboard’s PCIe 3.0 maxes out at x8 for one slot (with the KO in it) and at x4 for the other; x4 seems to cut GPU utilization to 90% or less fairly often. 150W a card, over 2M ppd though with COVID Moonshot WUs it’s currently around 2.4M and was often over 3M for a while. Probably will get a single 3070 or 3080 and sell the 2060s cheap, because I don’t see any other simple upgrade. With a single GPU the motherboard does 3.0 x16, and things might stay reasonably cool. (Sorrows of the somewhat-rich.)

[JimboPalmer]

Just a reminder, F@H has never supported SLI, ever.

[PantherX]

...F@H using single precision FP32, not half precision FP16...

That was the case... right now F@H uses mixed precision, most of the calculations are single precision but the rest is double precision. Thus, a GPU without double precision is not supported by F@H.

Using the software method when hardware is missing for double precision took too long and wasn't feasible thus, the hardware having double precision is now a requirement since the code can use the single/double as per its need

[MeeLee]

Indeed. I have two dedicated folding rigs that do nothing else. They both use Phanteks P600S cases. One has a 280 and a 420 and one has a 280 and a 240. I'm strongly considering getting two 3070's and replacing the three 1070s and TitanX which together produce around 3.5-4m ppd and consume about 700W so 2x3070 would only be around 450W. The big question is how well will they perform for FAH. Only time will tell.

Higher core count gpus can work both ways.
More than likely, not enough WUs will be optimized for such high core count in the first few months.

Like the 2080Ti GPUs, they often run less than optimal WUs.
So while your 3070-3090 might net you a much higher PPD on high atom WUs, they also won't have a significant higher PPD score for the rest of the WUs. They'll probably run at lower TDPs while running those lower atom count WUs.

[gunnarre]

Ryan Smith of AnandTech reports that all GA10x processors have SR-IOV enabled. If that's correct, then all 3000 series GPUs will actually support vGPU. In other words, you'll be able to segment the GPU into several virtual machines - not just pass through the whole card to a virtual machine.

Update:

Unfortunately, while the RTX-3000 cards support SR-IOV in hardware, it will be disabled in the driver. Nvidia will only support vGPU in the enterprise cards. Source https://twitter.com/RyanSmithAT/status/ ... 0140964864

Passing through the whole GPU will still work though.

[Juggy]

This link explains the CUDA deployment relatively well albeit mainly gaming focussed

https://www.engadget.com/nvidia-rtx-309 ... 59544.html

[HaloJones]

Indeed. I have two dedicated folding rigs that do nothing else. They both use Phanteks P600S cases. One has a 280 and a 420 and one has a 280 and a 240. I'm strongly considering getting two 3070's and replacing the three 1070s and TitanX which together produce around 3.5-4m ppd and consume about 700W so 2x3070 would only be around 450W. The big question is how well will they perform for FAH. Only time will tell.

I have a much smaller setup with a 2060 and a 2060KO on the same AMD motherboard in a Phanteks Enthoo Pro 614 case that doesn’t keep them particularly cool (the 2060 is next to the 750W power supply and runs around 76C, the KO higher up and usually 82C). The motherboard’s PCIe 3.0 maxes out at x8 for one slot (with the KO in it) and at x4 for the other; x4 seems to cut GPU utilization to 90% or less fairly often. 150W a card, over 2M ppd though with COVID Moonshot WUs it’s currently around 2.4M and was often over 3M for a while. Probably will get a single 3070 or 3080 and sell the 2060s cheap, because I don’t see any other simple upgrade. With a single GPU the motherboard does 3.0 x16, and things might stay reasonably cool. (Sorrows of the somewhat-rich.)

Are the cards running that hot because you're letting NVidia control the fan curves? If you don't intervene that's what NVidia will do - run them up to 82C.

My 1070s never go much above 25C over ambient but that's what water-cooling will get you.

[ap1978]

This might give an indication of PPD on the new cards:

NVIDIA GeForce RTX 3080: 168% of RTX 2080 performance in CUDA and OpenCL benchmarks - VideoCardz.com

[markdotgooley]

Are the cards running that hot because you're letting NVidia control the fan curves? If you don't intervene that's what NVidia will do - run them up to 82C.

My 1070s never go much above 25C over ambient but that's what water-cooling will get you.

I’m running Ubuntu 18.04 (can’t get things to work under anything later) and I seem to have limited control of things (or I’m just too ignorant). I think I could maybe force power consumption to 125W a card rather than the roughly 150W each they’re currently using. Maybe I can use the nvidia-smi command on a Linux terminal if that’s working (haven’t tried recently) to try to tweak some settings. The cards aren’t supposed to throttle until 90C and if neither goes over 82C, maybe I shouldn’t worry. Ambient is typically 25C.

[MeeLee]

Are the cards running that hot because you're letting NVidia control the fan curves? If you don't intervene that's what NVidia will do - run them up to 82C.

My 1070s never go much above 25C over ambient but that's what water-cooling will get you.

I’m running Ubuntu 18.04 (can’t get things to work under anything later) and I seem to have limited control of things (or I’m just too ignorant). I think I could maybe force power consumption to 125W a card rather than the roughly 150W each they’re currently using. Maybe I can use the nvidia-smi command on a Linux terminal if that’s working (haven’t tried recently) to try to tweak some settings. The cards aren’t supposed to throttle until 90C and if neither goes over 82C, maybe I shouldn’t worry. Ambient is typically 25C.

I can only tell you my way of doing things.
First, enable coolbits=28 on your system:

Code: Select all

sudo nvidia-xconfig --enable-all-gpus
sudo nvidia-xconfig --cool-bits=28

then reboot
then in GUI adjust fan curve to max (or maximum you feel comfortable with. Most GPUs do 3000RPM quite fine with little noise, but some people prefer to run them 2800RPM or 2500RPM).
The lower the fan speed, the lower the performance.
Then cap the power. 2060 and Super (more than likely KO as well), should run at 125W. There's no performance gain above this value, just power loss.
Then do a safe OC, depending on your GPU manufacturer, those OC values are different from card to card.
I have EVGA GPUs that only allow +65Mhz safely (they do +100Mhz shortly, but will cause bad WUs). I have Gigabyte GPUs that do +120Mhz. I have Asus GPUs that do +35Mhz only for the ROG strix (factory overclocked), or +220Mhz for one of their blower GPUs. Every brand is different, and there even are differences between models too.
The memory I always set to +1400Mhz, as the memory modules are underclocked from the factory by default. Those memory modules are 15Gbps (both Samsung and Hynix), but run at 14Gbps or lower from the factory. The difference between +1000Mhz and +1400Mhz isn't really noticeable (less than 1 to 2% in PPD).
Once a safe OC is paired with a safe power cap, and a fan speed maxed out, you'll run way lower temps.
Also, make sure the PSU is helping the case fans suck the hot air from inside the case. Sometimes it requires mounting the PSU upside down.

Another thing you can do to reduce GPU temps, is get rid of the front mounting bracket. And if your GPUs are sandwiched in the case, remove the Pcie mounting bracket on the case that's between the GPUs, for better airflow. Use cardboard and tape to create a duct, in case hot exited air would be sucked back inside the PC.
Your setup should allow GPU temps to run below 75C.

[MeeLee]

Ryan Smith of AnandTech reports that all GA10x processors have SR-IOV enabled. If that's correct, then all 3000 series GPUs will actually support vGPU. In other words, you'll be able to segment the GPU into several virtual machines - not just pass through the whole card to a virtual machine.

That would be, if it works....
You know how picky FAH is with WU assignment.
Running a 2080WU and removing the GPU, to run the WU on a 2060, and the wu gets aborted.
The 2060 can easily continue the WU, albeit a bit slower..
I think a similar thing will be done when all of a sudden your 3080 has half the shaders...

But even if it would work,
running a GPU with 2 vGPUs, could allow you to run 2 smaller WUs instead of 1, resulting in a PPD boost in that scenario,
But will also cut PPD drastically on large atom WUs, to the point that running 2 large, or a small WU and a large WU might net you less points compared to just running a single bigger WU on the entire GPU.

[gunnarre]

Hopefully the developers figure out a way to multitask large work units on large GPUs in the folding core so we won't have to manually use virtualization for that. They've already started discussing it on the OpenMM GitHub.

[markdotgooley]

To MeeLee: I’ll give it a shot. Thanks for the suggestions.