PCIe bandwidth requirements (RTX 50xx edition)

[arisu]

I am preparing to build a new system with four RTX 5080 GPUs (with the plan to upgrade to eight when the 5080 prices drop in a year or two). That's too much heat to put in one box without some seriously loud cooling so it will all be open air. I'm considering a single board computer and using ribbon risers (and NOT those awful 1x USB risers used for mining) to make space for the GPUs.

What are the PCIe bandwidth requirements for folding on something like a 5080? 8x and PCIe 4.0 (or perhaps 4x on PCIe 5.0)?

As ribbon length goes up, signal quality decreases. Does this increase latency in a way that impacts folding (like if it means more retransmissions that leave CUDA cores idle waiting for work) or is it an all-or-nothing thing where eventually the ribbon is too long and reliability plummets?

[muziqaz]

X4 is the minimum
X1 definitely does not work

[Retvari Zoltan]

A motherboard and CPU that can drive 8 cards at least with x4 bandwith is quite expensive. Risers further increase the costs. Personally I prefer cheaper motherboards / CPUs (i3) (for builds dedicated to GPU crunching) and single GPUs. Crunching on CPUs and (Nvidia) GPUs simultaneously can significantly reduce the GPU crunching speed, the same is true to some extent for multiple GPUs in the same system. You should consider this, or something in between. For example dual (water cooled) GPU systems (without risers, and expensive motherboards).

[arisu]

X4 is the minimum
X1 definitely does not work

The minimum for what PCIe gen? x4 on PCIe 5.0 has about the same bandwidth as x8 on PCIe 4.0.

When you say minimum is there any performance loss for using x4? I don't want to find out that "minimum" just means that it can barely finish a WU before it expires. But I also don't want to spend a lot of money on a high-end motherboard with extra PCIe lanes if the bus doesn't get even close to saturated.

[muziqaz]

Pcie3. Since I really don't think one would still run pcie2 mobos in this day and age

[arisu]

Pcie3. Since I really don't think one would still run pcie2 mobos in this day and age

That's enough to keep a 5080 fully occupied? That would mean that even a single x1 PCIe 5.0 lane is enough (3.0@x4 = 4.0@x2 = 5.0@x1 = 32 GT/s).

[Joe_H]

I think the testing that showed an x4 lane connection was enough on PCIe 3 dates back to the era of the 3090 cards, and even then showed some loss of processing speed compared to x8 or wider connections. I don't recall any newer testing mentioned here, possibly on the discord but I don't follow that.

[arisu]

Then to be safe, a 5.0@x4 or 4.0@x8 would be a good bet for a 5080. Then I could use M.2 to PCIe adapters and risers since M.2 is x4 which should be enough if it's PCIe 5.0, right?

I might just build two machines, both with four 5080s, instead of trying to squeeze out bandwidth from a consumer motherboard.

[muziqaz]

Pcie3. Since I really don't think one would still run pcie2 mobos in this day and age

That's enough to keep a 5080 fully occupied? That would mean that even a single x1 PCIe 5.0 lane is enough (3.0@x4 = 4.0@x2 = 5.0@x1 = 32 GT/s).

Are you really asking about the just released GPU testing on decades old standards and speeds?
With every new generation user is on their own to find out what is and what is not working for them.
We can only provide what is definitely not working.

[arisu]

Well that's why this thread is the RTX 50xx edition!

Wouldn't it be enough for someone with a 5080 to see what their PCIe bandwidth usage is like? If they're using under 64 GT/s then even x2 is fine if it's PCIe 5.0 (which is the newest that is in common usage). I don't want to make a very expensive mistake.

Since 4.0 is more common today especially with cheaper motherboards, I am curious if a single x16 could be used for four 5080s with something like:



If 64 GT/s is enough (5.0@x2, 4.0@x4, 3.0@x8 etc) then that would work well. But without knowing the actual bandwidth requirements in GT/s, I can only guess. I just assumed someone might know what the requirements are based on the bandwidth usage on their system (doesn't nvidia-smi tell it?).

[muziqaz]

First you need to find someone who can afford or even find to buy one, let alone fold on it and afford the electricity bill.
As I said you are on your own in this with brand new hardware.
Buy one 5080, and test it, then buy the rest of the hardware

[calxalot]

I think some people on discord have it.
You should ask there about folding bandwidth used.

[FaaR]

What software can report how much PCIe bandwidth is used by a GPU? Genuinely curious here! Windows task manager will do it for storage drives, but not GPUs...

[arisu]

What software can report how much PCIe bandwidth is used by a GPU? Genuinely curious here! Windows task manager will do it for storage drives, but not GPUs...

On Linux for Nvidia GPUs there's nvidia-smi. On Windows with an Intel processor, I guess this thing might work? https://github.com/intel/pcm