HPZ600 & Geforce Titan XP Pascal - Linux

[Kevin_Burton]

I've been running FAH for a month or two now on my old HPZ600 Workstation with my Quadro P2000 GPU. Along with the cpu i was getting around 300,000 PPD. I was a little disappointed with the figures to say the HP has 2 Xeon 6 Core processors in it along with the GPU. I was seeing other folders getting over a million PPD with the faster graphics cards so I bought a Titan XP Pascal for a lot of money. I knew I was pushing it for this system because it only has a 650W power supply and these cards can draw upto 300W peak. Sure enough, when I started folding with the Titan the system instantly rebooted. Damn I thought... wasted my money. I was thinking of adding an extra PSU but that would have meant more clutter.
Anyhow I solved the problem to a degree by simply limiting the power consumption of the Titan to 175W with this command: sudo nvidia-smi -i 1 -pl 175. This works like a dream with the CPU's, and both GPU's all folding. The Titan XP is still performing at between 96 - 98% even on the lower power setting and returning approx. 1.3 million PPD with the whole system generating around 1.6 million points per day and on generation 2 PCI shots too. Happy Days.

[PantherX]

Welcome to the F@H Forum Kevin_Burton,

If you want maximum PPD, then you may consider to stop folding on the CPU and Quadro P2000. Thus, you can have additional wattage for your Titan XP Pascal. Once you have the correct wattage, if you have additional wattage to use without pushing the system, you can either:
1) Resume folding on the Quadro P2000 and/or power limit it.
2) Resume folding on the CPUs and/or reduce the number of CPUs from 12 CPUs to 6 or less to ensure that the consumed wattage is within reason.

If you need further assistance, please post your log file. If you require guidance, here's a link: viewtopic.php?f=24&t=26036

[Kevin_Burton]

Thanks for the advice PantherX.
You’ve given me some ideas. I might just limit the processing to the GPU’s. I don’t seem to get much bang for the buck with CPU and both running flat out is probably consuming around 200W. I think my first experiment will be to find out at what point the Titan panics the system into reboot. I can simply keep increasing the power until it falls over and then back it off a little. I’d like to keep the P2000 running because that on its own raises about 10 times the PPD of the CPUs. I don’t personally think it’s a good idea dropping the CPU and then cranking up the GPU till it breaks. I need some CPU for other things. The only problem running both GPUs is the screen refresh slowing up and I’ve not been able to get the “when idle” feature working on my Mint system, instead having to manually pause it whilst using the machine.

Thanks again

Kevin

[foldy]

You can use 2x molex power connectors with a pcie 6pin adapter. This way you can put this connector to the original pcie 6pin into the Titan XP. Then GPU is driven by 2 different power lanes and you get more power to the GPU. Also be sure that you also have a 8pin connector into Titan XP, which can be created by pcie 6pin to pcie 8pin adapter but use that only for the 6pin pcie coming from power supply, not the 2x molex to 6 pin
https://www.newegg.com/p/0X6-04RT-0P300 ... -_-Product
https://www.newegg.com/en-labs-10cm-4-i ... -_-Product

See also
https://h30434.www3.hp.com/t5/Business- ... -p/5064832

[PantherX]

Regarding the On idle feature, F@H uses the OS setting, thus, you can figure out what Mint considers as idle and then once that's setup on the OS level, the client should be able to use it. I am not familiar enough with Linux to help you out. However, for the GPU, you may want to disable hardware acceleration at that would be one of the main contributing factor to the screen lag that you're encountering.

[Kevin_Burton]

Thank you again for the time and trouble. I’ll be looking into the power connections shortly but regarding “On Idle”. This I suspect is a bug in the Linux client. I can’t seem to make it work and yet if I fire up the boinc client, that seems to do exactly what you expect... in that leaving the machine idle for a few minutes it kicks in and carries on.

[JimboPalmer]

https://www.techpowerup.com/gpu-specs/q ... 2000.c2931

Quadro P2000 75 Watts

https://www.techpowerup.com/gpu-specs/titan-xp.c2948

Titan XP 250 Watts

https://support.hp.com/us-en/document/c01709709#AbT0

Power supply 650 watts

No idea which CPU, but the TDP is either 95 watts or 80 Watts. either way the Quardo P2000 uses less power than the CPU.

[MeeLee]

Hope you've played around with the overclocking as well (VRAM and GPU).
It's perfectly possible to stay in boost, by power capping and overclocking; doing nearly the same job with 1/3rd less energy and heat.

If it's only used for GPU folding, you can save some additional watts, by reducing the memory to a minimum of 4GB, 2 RAM slots if you can; and disable cores on your CPU in BIOS.
You really only need 2 CPU cores for feeding 2 GPUs.
Not only does it save power, it also allows the 2 active cores to run in boost a lot longer.
And you really need full boost on the titan (the titan might actually increase in PPD when doing this).

It'll also keep the case cooler, which will help maintain higher boost frequencies on the GPUs.
Keep fan speeds as high as you can tolerate (100% if you can).

[Kevin_Burton]

MeeLee thank you for the suggestions.

I am messing with the fan speeds etc but I think I need to have a script for sudo nvidia-smi -i 1 -pl175. I have this command in the startup and it’s set to run with no password. I’d prefer this to be up and running before login. Have you any idea the best way to accomplish this. I’ve written plenty of batch files in dos but this scripting is not in my league. I don’t seem to learn as quick as a I did 40 years ago. Any help would be appreciated.

Thank you

Kevin

[MeeLee]

I haven't done it myself, but you could make a terminal script in your text editor of choice.
Save it in /etc/init.d/nameofscript.sh

make the script executable by doing:
chmod +x nameofscript.sh

then do:
sudo update-rc.d /etc/init.d/nameofscript.sh defaults

this will allow it to be executed at boot.

you can set the fan speeds, but overclocking still has to be done via the GUI.
It's also safe to set the upper GPU frequency limit:
sudo nvidia-smi -i 0 -lgc 300,1995

The 1995 being your max safe GPU frequency.
it'll allow a well tuned and overclocked GPU to run at or below 1995Mhz on core 22, but not go any higher in a situation where Core 21 is being used.

[Kevin_Burton]

Thanks again for input MeeLee
I've still to look into the script but I have been playing with other things. First off I was playing with the "Coolbits" switch for fan speeds, etc. I did however notice that given I have had to reduce power consumption to 160w, the system would run for awhile and then reboot without warning of course if I manually increased the fan speed; 60% was keeping the temp below 80. It wouldn't have it so ive left it to auto until I can couple up another psu in the near future. Likewise with the overclocking. This is where the extra WATTS will come in. The GPU is running at just about 100% at a clock speed roughly 1500Mhz. I am not going to be able to boost above that until I get the extra power I mentioned.

I tried a number of things to allow the system to be cooler including pulling a PCIe RAID controller that was sitting between the Quadro P2000 and the GTX Titan XP. Oddly enough upon removal of that card I noticed that the PCIe bandwidth had dropped to virtually zero and making a big impact on my PPD. I put the RAID Controller back in, even though right now its not needed and the bandwidth went back to normal.

I've setting on 160W for the Titan until i get another power supply. Cranking it up doesn't seem to improve PPD... but it does crash the Z600 to rebooting.
I am very much looking forward to trying this thing at full power, but to be fair, near doubling the power to the 300W max of the GPU is not going to double my PPD. I think it has to be sensible in that I suspect that taking the GPU to its max and using near double the electricity is only going to give me maybe a 10 or 15% advantage.

I'll keep you posted about the script and the reults from the added power supply.

Thanks again.

K

[Kevin_Burton]

BTW... i didn't mention. The Titan XP and P2000 are returning close to 1.75m PPD. Im quite happy with that on reduced power.

K

[MeeLee]

As a correction to my above post, in a script, you can set the power target of Nvidia GPUs. But you can't set the fan speed, or overclock.
That you have to do in the GUI.

My recommendation is to keep the fans as high as possible (100% if you can), and keep the temps below 80C on a Core 22. 75C preferred.

You can also overclock the GPU, at the same time you're capping the power, by about 5-10% depending on what model.

80C seems rather high. If there's a possibility you can add a case fan, to cool the back plate of the GPU for additional cooling, the 2 or so watts a fan uses, is a good investment in trying to keep GPU running at higher frequencies;as Nvidia drivers scale the GPU frequency in relation to temperatures.
I run mine at 45-65C with the fan curve set to 100%.
But personally, I'd feel more comfortable running any GPU (especially older GPUs) at below 75C rather than 80C.
If you can't increase fan speeds any more, you can still reduce power consumption further, and compensate with a slightly higher overclock. Whatever still runs stable.

You'll need to run your system at those power settings, and record the power usage.
Then, find a PSU with double the power handling.
Eg: if your PC pulls 500W at the wall, you'll need a 1000W PSU.
It could allow you to add an additional GPU, or a higher power GPU, in case you're upgrading.
You will want to load your PSU with a continuous load, between 30 to 80% of it's rated capacity.
It's not good to run a PSU at over 80% of it's power handling, as it'll not have any spare headroom, and things like GPU or CPU freezes will become more common.

Now is a very bad time to buy any PC hardware.
In a few weeks, the first ships will arrive from China, resupplying the stock on major websites like Amazon or Newegg; stock which has been depleted due to the lack of flights, due to Corona.

[HaloJones]

You can overclock with a script.

e.g.

nvidia-settings -c :0 -a '/GPUMemoryTransferRateOffset[3]=200' - this will increase your memory speed by 200MHz
nvidia-settings -c :0 -a '/GPUGraphicsClockOffset[3]=50' - this will increase your gpu core speed by 50MHz

fan speed ought to be possible but I can't find the right syntax yet.

on pre10xx cards, you could also overvolt, eg:

nvidia-settings -c :0 -a '[gpu:1]/GPUOverVoltageOffset=72000'

nvidia-settings is very badly documented IMO meaning you're often just googling for hours to find a forum post that then works for you.

[MeeLee]

You can overclock with a script.

e.g.

nvidia-settings -c :0 -a '/GPUMemoryTransferRateOffset[3]=200' - this will increase your memory speed by 200MHz
nvidia-settings -c :0 -a '/GPUGraphicsClockOffset[3]=50' - this will increase your gpu core speed by 50MHz

fan speed ought to be possible but I can't find the right syntax yet.

on pre10xx cards, you could also overvolt, eg:

nvidia-settings -c :0 -a '[gpu:1]/GPUOverVoltageOffset=72000'

nvidia-settings is very badly documented IMO meaning you're often just googling for hours to find a forum post that then works for you.

I've tried those, but never got them to work in terminal.
Fan speed also has a command like that..
Been asking for ages on the Nvidia website to make terminal commands like the 'nvidia-smi -pl' command.