PCI-e bandwidth/capacity limitations

[bruce]

Well, I've got a few more GPUs than slots so maybe I'll try running something on x1 and see if it can do ANYTHING USEFUL Clearly PCIe will be the limiting factor, not GPU throughput.

(I've never been good about selling stuff that has been replaced by an upgrade.)

Why not?

[Nathan_P]

OK, 1st test on an actual WU - haven't had a chance to do any proper testing as win7 won't install properly and windows is a lot easier to run a bunch of WU tests on the Linux. This is taken from my hfm logs on my main folding machine running the zorin 9 distro.

PCIe 2.0 @ x16 project 10490 tpf 2.40 to 2.42 on a GTX 970. PCIe 3.0 @ x16, project 10490 tpf 2.40 - no change between PCIe 2.0 and 3.0 x16 at this time - this bears out the gaming performance speeds found elsewehere.

PCIe 2.0 @ x8 project 9151 tpf 1.13 to 1.26 on a GTX 980ti, PCIe 3.0 x 16 project 9151 1.07 - 6 second change between current and previous recorded minimum time - this takes ppd from 422k PPD to 479k PPD. Again the gaming performance speeds do show a drop off at PCIe 2.0 x8

As projects run through over the next couple of days i'll post more results and try some PCIe 1.1 tests but early indications are that PCIe 2.0 x8 and below do affect performance and the faster the card the bigger the hit.

[artoar_11]

I'm not sure if the games and FAH scaled PCIe xx transfer equally. Just for comparison:


Source: http://www.techpowerup.com/reviews/AMD/ ... ng/18.html

[foldy]

We expect FAH to have a major performance hit like 50% on lower than pcie 2.0 x8 with fastest GPUs because of the high bus bandwidth usage.
Other than this games relative performance summary which has only 11% performance hit even on pcie 1.1 x4

[foldy]

@Nathan_P: Maybe you want to try this linux tool to monitor pcie bus usage
https://github.com/mountassir/gMonitor

So you can see what bus usage your GTX 980ti hit at PCIe 2.0 @ x8.
I guess around 80% which will already be high enough to increase work unit data transfer latency and so increase TPF and reduce PPD.

The nvidia-settings command used in this tool for pcie bus usage
"nvidia-settings -t -q " << gpuId << "/GPUUtilization | tr ',' '\n' |grep PCIe|sed 's/[^0-9]//g'";

[Nathan_P]

I'm going to measure more projects from HFM overnight, a couple of results do not make conclusive results and a couple of the WU I've truned in today don't have any data from the old rig, 970 is staying consistent with previous results though.

More Linux testing Thursday, Linux isn't my thing - it took me an hour to get samba working yesterday.

[hiigaran]

While more data is being gathered, I'd like to sidetrack a little. Though my original intention was to find a solution to PCI-e capacity issues on a motherboards, these initial results heavily suggest that it may be impossible to use consumer-grade hardware to create a single, or even dual system with high GPU capacity. At least, not without a severe reduction in output. However, since 3.0 x8 equivalent bandwidth seems to be a viable option, I'd like to completely forget about any possibility of splitters, and instead look at the possibility of x16 risers.

Now I am fully aware of their existence. However, every flexible x16 riser I have seen to far is a bulky ribbon cable with limited length. Comparing this to the flexible x1 risers using a USB3.0 cable, which offers much more freedom, and length. Has anyone ever attempted a cross between the two? A x16 capacity riser with the flexibility of the x1s? If not, what's the longest x16 riser anyone has seen?

[bruce]

I was wondering that myself.

Two thoughts:

* A motherboard with 20 lanes can drive a x16 slot and a x1 slot. If you want to use the x1 slot, you're limited to the x1 bandwidth.

* Will the bandwidth of your extended x16 (V-whatever) slot be limited by the bandwidth of USB?

[Nathan_P]

While more data is being gathered, I'd like to sidetrack a little. Though my original intention was to find a solution to PCI-e capacity issues on a motherboards, these initial results heavily suggest that it may be impossible to use consumer-grade hardware to create a single, or even dual system with high GPU capacity. At least, not without a severe reduction in output. However, since 3.0 x8 equivalent bandwidth seems to be a viable option, I'd like to completely forget about any possibility of splitters, and instead look at the possibility of x16 risers.

Now I am fully aware of their existence. However, every flexible x16 riser I have seen to far is a bulky ribbon cable with limited length. Comparing this to the flexible x1 risers using a USB3.0 cable, which offers much more freedom, and length. Has anyone ever attempted a cross between the two? A x16 capacity riser with the flexibility of the x1s? If not, what's the longest x16 riser anyone has seen?

The best consumer, (lga 1150/1151) boards will have 4 x16 slots that run at x8, preferably PCIe3 to avoid any loss - even then you are limited to top end offerings, I'm running all these tests on an Asus Z87-WS. 2011 gives you more flexibility but I haven't researched that area. The boards with the most slots and expansion potential that I know of are the Z9PE-D8 and newer Z10PE-D8, even then the 7 x16 slots will only run 2 at x16 and 5 at x8.

From memory the longest x16 riser I have seen is about 6 inches, IIRC something to do with signal loss if they go any longer

[bruce]

In some cases, PCIe x1 is going to work but it's probably wrong to recommend it for folks with fast GPUs.

Environment tested:
GPU: GT 740 (Fermi)
Linux
FAHCore_21 (Project 11414)
PCIe v2.0

Results:
x16 utilization: 14%
TPF: 48:36
PPD: 11092

x1 utilization: 19%
TPF: 48:47
PPD: 11039

In this particular case, the 1x riser keeps the GPU at 100%. at least during the routine analysis portions of the WU. It might or might not show different results during startup/finish/checkpointing/etc. but that's a small portion of the run.

Environment tested (as above except):
GPU: GTX 980 (Maxwell)
FAHCore_18 (Project 9160)

Results:
x16 utilization: 1%
GPU utilization: 99-100%
TPF: 2:26
PPD: 138877

x1 utilization:
TPF:
PPD:

[hiigaran]

The splitter? Wait, did you actually get your hands on one? Or is this something else?

[bruce]

I have two different brands which convert an x1 PCIe slot to USB port and convert a USB connection to an x16 slot ... with an added power connection.

Installing them was less than straightforward but I was able to fix all the problems I encountered and got my second GPU to run over an x1 connection. I tried it with the GPU to winch my monitor was attached and so far have been unable to do that. (The next thing I'm going to try is reenabling my Monitor to the iGPU first, then moving attaching the GPU -- sort of like using the GPU in PhysX mode)

Next on the agenda is to connect a full-height GPU to a half-height system -- thought that might also be a good application for a x16 riser cable.

[hiigaran]

I have two different brands which convert an x1 PCIe slot to USB port and convert a USB connection to an x16 slot ... with an added power connection.

Did you mean one brand that is x1 to USB, and another brand that is USB to x16, or did you mean two brands of the same thing? If the latter, is that x1 -> USB -> x16, and if so, isn't that just the flexible risers that are used in bitcoin farming?

Or is it something completely different, and indicative of my lack of sleep?

[bruce]

Each one is x1 > USB > x16

See the picture (more or less) in the 3rd post on page 1 of this topic and a lot of discussion in between.

[hiigaran]

Right, so it's a riser. This is where my confusion was stemming from.

Moving on, looking at your initial data for the 980 bandwidth, it shows only 1% usage of x16. Based on that it sounds like x1 would work with only 16% of max bandwidth. With wildly varying data, could it be that different projects have different bandwidth requirements?