Old PC upgrade suggestions - Updated to RTX2060

[PantherX]

For some reason, CUDA isn't being detected which isn't ideal:
14:58:42: CUDA: Not detected: cuInit() returned 101

From what I have read, this is caused by missing OpenCL package:
14:58:42: OpenCL: Not detected: clGetPlatformIDs() returned -1001
and the solution is:
sudo apt-get install ocl-icd-opencl-dev

[MeeLee]

Power supply Pure Power L8 530W has max 340 watts on 12V1 and max 240 watts on 12V2. So if you substract 130 watts for CPU and MB it is 400 watts left for GPU. So it can drive one big GPU but not two big GPUs.

Unless you lower power consumption on the new big GPUs.
Like, it would run 2x RTX 2080 Super just fine at 145W per GPU.
For 2080Ti, you need about 200Watts per card minimum, or else performance drops too much.
Those you probably won't be able to pair on that PSU.

[agent71]

Yes I have that package already. My gut feeling is this is something to do with it being a VM and that I need to set a flag somewhere to spoof the OS into believing it's not a VM.

nvidia-smi errors with

Code: Select all

reports Unable to determine the device handle for GPU

and clinfo returns 0.

Although host OS fully recognises the GPU as RTX2060 I can't get the cuda or Nvidia drivers to work.

In ESXi that appears to be setting hypervisor.cpuid.v0 = "FALSE" but cannot see how to do this in Proxmox. I reckon if you do that your home and dry.

But I'm throwing in the towel!

To be honest this became an interesting, but ultimately pointless, exercise to see if I could get it working rather than it being the solution I really required. Had this of worked it would still have been overly complex for my original requirement. Still slightly disappointed I couldn't use my time to document how to do this for everyone. None the less will document what I did in the event others with a need can pick up and run with it.

Have learnt a few things though....

VMWare ESXi is so easy to use. Absolute childs play but very strict on it's hardware support so a real pain in a mixed environment. But it's meant for enterprise so not an issue as fit for purpose.
Proxmox supports wider hardware out the box but my god the UI is poor and management is a strange mix of basic UI and full command line Linux - very odd. The UI is definitely designed by a programmer!

If I had a requirement to build a virtualised environment with GPU pass through I'd go VMWare ESXi but would base hardware decisions on ESXi support. Even then you are into custom ISOs so not for beginners. But if your hardware is supported it looks pretty simple.
Mint has video support issues in its install ISO with proxmox. Fixed by forcing the VM to use VMWare console rather than default for proxmox.

Enable motherboard for Intel VT-d or AMD IOMMU
This as varied for Intel and AMD as it is for each motherboard manufacturer. Google your friend here rather than a static link from me but it's mostly config in your north bridge and cpu settings.

Installation
Download and install Proxmox
https://www.proxmox.com/en/

Once installed on the Proxmox hypervisor open a shell follow the following:
https://pve.proxmox.com/wiki/Pci_passthrough

Create VM for Linux install and assign PCI adpater for GPU
If using Mint 19.3 ensure console for VM set to VMWare otherwise you get black screen/flashing cursor top left after grub menu.

After OS installed install Nvidia drivers
https://linuxconfig.org/how-to-install- ... aver-linux

Install CUDA

Code: Select all

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

Useful reading
https://blog.quindorian.org/2018/03/bui ... ough.html/
https://bbs.archlinux.org/viewtopic.php?id=162768

I'm off to rebuild with Mint and basic Samba shares!

[agent71]

Machine back up and running and found Zotac P106-90's popping up on eBay for £40-45. My understanding is they are GTX1060 3GB without video ports. Too good a price to ignore and see if they'll work - especially at 75W.

In preparation for it and ensuring system ok I spent the evening installing TIG stack (Telegraf, InfluxDB and Grafana) to monitor the machine. Enabled nvidia-smi plugin for Telegraf too so I can check on GPU.





Ignore the 100% CPU - that's because BOINC kicks in at 23:00.

[MeeLee]

Not all motherboards recognize those mining GPUs. Just letting you know.
They're a great deal for compute workload per dollar.
They also might work well on a PCIE 3.0 x1 slot in Linux running Core 22.

[agent71]

Only slot I have free is PCIE 2.0 x16 which with another GPU in the PCIE 3.0 slot will drop the 2.0 slot to x4. We'll see and I'm fully aware it may not work. Also read that Nvidia have removed support for those cards in later drivers on Windows (have to assume Linux too) and I'm using the latest.

We'll see but fingers crossed and worst case I'll just sell on.

[MeeLee]

It's not the driver, it's the BIOS that might not recognize the GPU.

[agent71]

Unfortunately a slightly more simple issue with the card... Dual slot GPUs just won't physically fit with the motherboard. The card fouls a number of USB and fan connections that sit proud. Ah well. Worth a try!

[MeeLee]

Unfortunately a slightly more simple issue with the card... Dual slot GPUs just won't physically fit with the motherboard. The card fouls a number of USB and fan connections that sit proud. Ah well. Worth a try!

In that case, use a PCIE riser, to physically locate the GPU somewhere else inside the chassis.

[agent71]

Yes I'm looking into riser cables now. Case is small though so not a lot of room to position card elsewhere, Corsair Carbide 240 Air https://www.corsair.com/uk/en/Categorie ... 9011070-WW

I may have a Fractal Design R2 case in the loft from 10-15 years ago. Assuming I never sold it or threw it out I can rebuild in that.

[agent71]

Code: Select all

@folding01:~$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation Device 1e89 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 10f8 (rev a1)
01:00.2 USB controller: NVIDIA Corporation Device 1ad8 (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1ad9 (rev a1)
02:00.0 3D controller: NVIDIA Corporation GP106 [P106-090] (rev a1)
@folding01:~$ 

Sweet... that was easy. No changes to anything required. Just installed PCIE riser cable and P106 and added GPU slot in FAHControl.
https://www.amazon.co.uk/gp/product/B07 ... UTF8&psc=1


Est PPD seems high on P106 vs RTX2060 although I am throttling the RTX to 125W.



Need to work on installation though...

[agent71]

Hmmm.... interesting thing.

Not long after FAHControl reporting 1.2M PPD on P109 and TPF of 1m 54 seconds it dropped to 240K and 5m 45 seconds and ETA increased to over 8 hours.

Any ideas where to look to understand why? I don't think PSU an issue as nvidia-smi is reporting power as expected to each card. ie. 125W and 75W.

[HaloJones]

The initial estimated ppd is often wildly inaccurate. especially on a new card. the only thing it has to go on is the same units on the other card so it will guess high for a while.

[agent71]

Ok thanks for info. Will let it settle for a few days and see where it ends up. Cheers

[MeeLee]

The first 3% is inaccurate. After 5% of unstopped activity, you'll have a more accurate number of that WU.
You may have an occasional fast or slow WU as your first WU.
Once you've looked at PPDs on separate WUs, you have a better understanding on how fast the GPUs are.

You also can set the Folding Power slider to 'full', so your PC won't intermittently halt the folding process, which can cause lower PPD numbers.
Go to advanced and there's another option to use slightly more system resources.
The setting under the red rectangle (I snipped it off the inets):


Unless you are running an open bench, with fans to 100%, setting the RTX 2060, and the 106 (1060 alternative) to those power settings might be limiting your PPD.
If you're running them in a case, you might want to increase the wattage of the 2060 to 130-135W, and the 106 you'll have to play around with it, but I would estimate it to run better at 80-85W; and increase your fan speed to max on both (or as high as they can go without being obnoxious loud). Lower temps, mean higher performance.
I also would look at overclocking the GPU, as you'll sacrifice performance by capping power to those GPUs.

The 106 GPU you own, should be able to net 330k PPD (for the 3GB) and I believe it was around 600k PPD for the 6GB,but I'm not 100% sure about it on core 21 WUs.
The 2060 seems like it's well configured.

You might get a small PPD penalty on the 106, if it's plugged into an x1 slot.
If your primary GPU is running at x16 speed, and the secondary is running at x1 speed, it would be better to run the primary at x8 and the secondary at x8 or x4 speeds.