Can't use CUDA core

[thomas2000]

F@H isn't using CUDA on my RTX 2060 and log only says:

Code: Select all

08:30:35:WU01:FS01:0x22:Platform 3: CUDA
08:30:35:WU01:FS01:0x22:  cuda-device 0 specified
08:30:41:WU01:FS01:0x22:Attempting to create CUDA context:
08:30:41:WU01:FS01:0x22:  Configuring platform CUDA
08:30:41:WU01:FS01:0x22:Failed to create CUDA context:
08:30:41:WU01:FS01:0x22:Error loading CUDA module: CUDA_ERROR_FILE_NOT_FOUND (301)

How to find out what file is it? Driver reinstall using DDU didn't help.



i've installed CUDA tooolkit too, but no luck.



Any suggestions, please?

Full log:

Code: Select all

*********************** Log Started 2020-10-11T10:47:54Z ***********************
10:47:54:Trying to access database...
10:47:54:Successfully acquired database lock
10:47:54:Read GPUs.txt
10:47:54:Enabled folding slot 00: READY cpu:11
10:47:54:Enabled folding slot 01: READY gpu:0:TU106 [Geforce RTX 2060]
10:47:54:****************************** FAHClient ******************************
10:47:54:        Version: 7.6.13
10:47:54:         Author: Joseph Coffland <[email protected]>
10:47:54:      Copyright: 2020 foldingathome.org
10:47:54:       Homepage: https://foldingathome.org/
10:47:54:           Date: Apr 27 2020
10:47:54:           Time: 21:21:01
10:47:54:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
10:47:54:         Branch: master
10:47:54:       Compiler: Visual C++ 2008
10:47:54:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:47:54:       Platform: win32 10
10:47:54:           Bits: 32
10:47:54:           Mode: Release
10:47:54:           Args: --open-web-control
10:47:54:         Config: C:\Users\romus\AppData\Roaming\FAHClient\config.xml
10:47:54:******************************** CBang ********************************
10:47:54:           Date: Apr 24 2020
10:47:54:           Time: 17:07:55
10:47:54:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
10:47:54:         Branch: master
10:47:54:       Compiler: Visual C++ 2008
10:47:54:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:47:54:       Platform: win32 10
10:47:54:           Bits: 32
10:47:54:           Mode: Release
10:47:54:******************************* System ********************************
10:47:54:            CPU: AMD Ryzen 5 2600 Six-Core Processor
10:47:54:         CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
10:47:54:           CPUs: 12
10:47:54:         Memory: 15.95GiB
10:47:54:    Free Memory: 6.21GiB
10:47:54:        Threads: WINDOWS_THREADS
10:47:54:     OS Version: 6.2
10:47:54:    Has Battery: false
10:47:54:     On Battery: false
10:47:54:     UTC Offset: 1
10:47:54:            PID: 11960
10:47:54:            CWD: C:\Users\romus\AppData\Roaming\FAHClient
10:47:54:  Win32 Service: false
10:47:54:             OS: Windows 10 Enterprise
10:47:54:        OS Arch: AMD64
10:47:54:           GPUs: 1
10:47:54:          GPU 0: Bus:7 Slot:0 Func:0 NVIDIA:7 TU106 [Geforce RTX 2060]
10:47:54:  CUDA Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:7.5 Driver:11.1
10:47:54:OpenCL Device 0: Platform:0 Device:0 Bus:7 Slot:0 Compute:1.2 Driver:456.71
10:47:54:******************************* libFAH ********************************
10:47:54:           Date: Apr 15 2020
10:47:54:           Time: 14:53:14
10:47:54:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
10:47:54:         Branch: master
10:47:54:       Compiler: Visual C++ 2008
10:47:54:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:47:54:       Platform: win32 10
10:47:54:           Bits: 32
10:47:54:           Mode: Release
10:47:54:***********************************************************************
10:47:54:<config>
10:47:54:  <!-- Folding Slot Configuration -->
10:47:54:  <cuda-index v='0'/>
10:47:54:  <gpu-index v='1'/>
10:47:54:  <opencl-index v='0'/>
10:47:54:
10:47:54:  <!-- Slot Control -->
10:47:54:  <power v='full'/>
10:47:54:
10:47:54:  <!-- User Information -->
10:47:54:  <passkey v='*****'/>
10:47:54:  <team v='223518'/>
10:47:54:  <user v='romusking'/>
10:47:54:
10:47:54:  <!-- Folding Slots -->
10:47:54:  <slot id='0' type='CPU'/>
10:47:54:  <slot id='1' type='GPU'/>
10:47:54:</config>
10:47:54:WU02:FS01:Starting
10:47:54:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\romus\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 11960 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:47:54:WU02:FS01:Started FahCore on PID 2888
10:47:54:WU02:FS01:Core PID:2820
10:47:54:WU02:FS01:FahCore 0x22 started
10:47:54:WU00:FS00:Starting
10:47:54:WARNING:WU00:FS00:AS lowered CPUs from 11 to 10
10:47:54:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\romus\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx2-256/a8-0.0.8/Core_a8.fah/FahCore_a8.exe -dir 00 -suffix 01 -version 706 -lifeline 11960 -checkpoint 15 -np 10
10:47:54:WU00:FS00:Started FahCore on PID 10880
10:47:54:WU00:FS00:Core PID:14136
10:47:54:WU00:FS00:FahCore 0xa8 started
10:47:54:WU02:FS01:0x22:*********************** Log Started 2020-10-11T10:47:54Z ***********************
10:47:54:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
10:47:54:WU02:FS01:0x22:       Core: Core22
10:47:54:WU02:FS01:0x22:       Type: 0x22
10:47:54:WU02:FS01:0x22:    Version: 0.0.13
10:47:54:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
10:47:54:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
10:47:54:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
10:47:54:WU02:FS01:0x22:       Date: Sep 19 2020
10:47:54:WU02:FS01:0x22:       Time: 02:35:58
10:47:54:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
10:47:54:WU02:FS01:0x22:     Branch: core22-0.0.13
10:47:54:WU02:FS01:0x22:   Compiler: Visual C++ 2015
10:47:55:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
10:47:55:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
10:47:55:WU02:FS01:0x22:   Platform: win32 10
10:47:55:WU02:FS01:0x22:       Bits: 64
10:47:55:WU02:FS01:0x22:       Mode: Release
10:47:55:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
10:47:55:WU02:FS01:0x22:             <[email protected]>
10:47:55:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 2888 -checkpoint 15
10:47:55:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
10:47:55:WU02:FS01:0x22:             0 -gpu 0
10:47:55:WU02:FS01:0x22:************************************ libFAH ************************************
10:47:55:WU02:FS01:0x22:       Date: Sep 7 2020
10:47:55:WU02:FS01:0x22:       Time: 19:09:56
10:47:55:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
10:47:55:WU02:FS01:0x22:     Branch: HEAD
10:47:55:WU02:FS01:0x22:   Compiler: Visual C++ 2015
10:47:55:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
10:47:55:WU02:FS01:0x22:   Platform: win32 10
10:47:55:WU02:FS01:0x22:       Bits: 64
10:47:55:WU02:FS01:0x22:       Mode: Release
10:47:55:WU02:FS01:0x22:************************************ CBang *************************************
10:47:55:WU02:FS01:0x22:       Date: Sep 7 2020
10:47:55:WU02:FS01:0x22:       Time: 19:08:30
10:47:55:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
10:47:55:WU02:FS01:0x22:     Branch: HEAD
10:47:55:WU02:FS01:0x22:   Compiler: Visual C++ 2015
10:47:55:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
10:47:55:WU02:FS01:0x22:   Platform: win32 10
10:47:55:WU02:FS01:0x22:       Bits: 64
10:47:55:WU02:FS01:0x22:       Mode: Release
10:47:55:WU02:FS01:0x22:************************************ System ************************************
10:47:55:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
10:47:55:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
10:47:55:WU02:FS01:0x22:       CPUs: 12
10:47:55:WU02:FS01:0x22:     Memory: 15.95GiB
10:47:55:WU02:FS01:0x22:Free Memory: 6.18GiB
10:47:55:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
10:47:55:WU02:FS01:0x22: OS Version: 6.2
10:47:55:WU02:FS01:0x22:Has Battery: false
10:47:55:WU02:FS01:0x22: On Battery: false
10:47:55:WU02:FS01:0x22: UTC Offset: 1
10:47:55:WU02:FS01:0x22:        PID: 2820
10:47:55:WU02:FS01:0x22:        CWD: C:\Users\romus\AppData\Roaming\FAHClient\work
10:47:55:WU02:FS01:0x22:************************************ OpenMM ************************************
10:47:55:WU02:FS01:0x22:   Revision: 189320d0
10:47:55:WU02:FS01:0x22:********************************************************************************
10:47:55:WU02:FS01:0x22:Project: 14907 (Run 384, Clone 2, Gen 45)
10:47:55:WU02:FS01:0x22:Unit: 0x0000004181d59d695f5260284faa8db8
10:47:55:WU02:FS01:0x22:Digital signatures verified
10:47:55:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
10:47:55:WU02:FS01:0x22:Version 0.0.13
10:47:55:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
10:47:55:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
10:47:55:WU02:FS01:0x22:  XTC frame write interval: 50000 steps (5%) [20 total]
10:47:55:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
10:47:55:WU02:FS01:0x22:There are 4 platforms available.
10:47:55:WU02:FS01:0x22:Platform 0: Reference
10:47:55:WU02:FS01:0x22:Platform 1: CPU
10:47:55:WU00:FS00:0xa8:*********************** Log Started 2020-10-11T10:47:54Z ***********************
10:47:55:WU02:FS01:0x22:Platform 2: OpenCL
10:47:55:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
10:47:55:WU02:FS01:0x22:  opencl-device 0 specified
10:47:55:WU00:FS00:0xa8:       Core: Gromacs
10:47:55:WU02:FS01:0x22:Platform 3: CUDA
10:47:55:WU00:FS00:0xa8:       Type: 0xa8
10:47:55:WU02:FS01:0x22:  cuda-device 0 specified
10:47:55:WU00:FS00:0xa8:    Version: 0.0.8
10:47:55:WU00:FS00:0xa8:     Author: Joseph Coffland <[email protected]>
10:47:55:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
10:47:55:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
10:47:55:WU00:FS00:0xa8:       Date: Sep 25 2020
10:47:55:WU00:FS00:0xa8:       Time: 08:28:00
10:47:55:WU00:FS00:0xa8:   Revision: d4f41a5b7a748fdb0ff538746b195ca3a8075aae
10:47:55:WU00:FS00:0xa8:     Branch: master
10:47:55:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
10:47:55:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Ob3
10:47:55:WU00:FS00:0xa8:             /Zc:throwingNew /MT
10:47:55:WU00:FS00:0xa8:   Platform: win32 10
10:47:55:WU00:FS00:0xa8:       Bits: 64
10:47:55:WU00:FS00:0xa8:       Mode: Release
10:47:55:WU00:FS00:0xa8:       SIMD: avx2_256
10:47:55:WU00:FS00:0xa8:     OpenMP: ON
10:47:55:WU00:FS00:0xa8:       CUDA: OFF
10:47:55:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 10880 -checkpoint 15 -np
10:47:55:WU00:FS00:0xa8:             10
10:47:55:WU00:FS00:0xa8:************************************ libFAH ************************************
10:47:55:WU00:FS00:0xa8:       Date: Sep 25 2020
10:47:55:WU00:FS00:0xa8:       Time: 07:23:00
10:47:55:WU00:FS00:0xa8:   Revision: d4f41a5b7a748fdb0ff538746b195ca3a8075aae
10:47:55:WU00:FS00:0xa8:     Branch: master
10:47:55:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
10:47:55:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Ob3
10:47:55:WU00:FS00:0xa8:             /Zc:throwingNew /MT
10:47:55:WU00:FS00:0xa8:   Platform: win32 10
10:47:55:WU00:FS00:0xa8:       Bits: 64
10:47:55:WU00:FS00:0xa8:       Mode: Release
10:47:55:WU00:FS00:0xa8:************************************ CBang *************************************
10:47:55:WU00:FS00:0xa8:       Date: Sep 25 2020
10:47:55:WU00:FS00:0xa8:       Time: 07:22:40
10:47:55:WU00:FS00:0xa8:   Revision: d4f41a5b7a748fdb0ff538746b195ca3a8075aae
10:47:55:WU00:FS00:0xa8:     Branch: master
10:47:55:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
10:47:55:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Ob3
10:47:55:WU00:FS00:0xa8:             /Zc:throwingNew /MT
10:47:55:WU00:FS00:0xa8:   Platform: win32 10
10:47:55:WU00:FS00:0xa8:       Bits: 64
10:47:55:WU00:FS00:0xa8:       Mode: Release
10:47:55:WU00:FS00:0xa8:************************************ System ************************************
10:47:55:WU00:FS00:0xa8:        CPU: AMD Ryzen 5 2600 Six-Core Processor
10:47:55:WU00:FS00:0xa8:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
10:47:55:WU00:FS00:0xa8:       CPUs: 12
10:47:55:WU00:FS00:0xa8:     Memory: 15.95GiB
10:47:55:WU00:FS00:0xa8:Free Memory: 6.14GiB
10:47:55:WU00:FS00:0xa8:    Threads: WINDOWS_THREADS
10:47:55:WU00:FS00:0xa8: OS Version: 6.2
10:47:55:WU00:FS00:0xa8:Has Battery: false
10:47:55:WU00:FS00:0xa8: On Battery: false
10:47:55:WU00:FS00:0xa8: UTC Offset: 1
10:47:55:WU00:FS00:0xa8:        PID: 14136
10:47:55:WU00:FS00:0xa8:        CWD: C:\Users\romus\AppData\Roaming\FAHClient\work
10:47:55:WU00:FS00:0xa8:********************************************************************************
10:47:55:WU00:FS00:0xa8:Project: 16810 (Run 29, Clone 26, Gen 85)
10:47:55:WU00:FS00:0xa8:Unit: 0x00000060b2aec48a5eff27e689cc3726
10:47:55:WU00:FS00:0xa8:Digital signatures verified
10:47:55:WU00:FS00:0xa8:Calling: mdrun -c frame85.gro -s frame85.tpr -x frame85.xtc -cpi state.cpt -cpt 15 -nt 10 -ntmpi 1
10:47:55:WU00:FS00:0xa8:Steps: first=42500000 total=43000000
10:47:56:WU00:FS00:0xa8:Completed 122462 out of 500000 steps (24%)
10:47:59:5:127.0.0.1:New Web session
10:47:59:6:127.0.0.1:New Web session
10:48:12:WU02:FS01:0x22:Attempting to create CUDA context:
10:48:12:WU02:FS01:0x22:  Configuring platform CUDA
10:48:13:WU02:FS01:0x22:Failed to create CUDA context:
10:48:13:WU02:FS01:0x22:Error loading CUDA module: CUDA_ERROR_FILE_NOT_FOUND (301)
10:48:13:WU02:FS01:0x22:Attempting to create OpenCL context:
10:48:13:WU02:FS01:0x22:  Configuring platform OpenCL
10:48:19:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0

[foldy]

Try to uninstall all cuda sdk toolkit because they currently may interfer with FAH.

[thomas2000]

Try to uninstall all cuda sdk toolkit because they currently may interfer with FAH.

That worked like a charm

16:35:55:WU01:FS01:0x22:Attempting to create CUDA context:
16:35:55:WU01:FS01:0x22: Configuring platform CUDA
16:36:07:WU01:FS01:0x22: Using CUDA and gpu 0

I look forward to 15% higher PPD

Thank you very much!

[PantherX]

Welcome to the F@H Forum thomas2000,

FYI, CUDA can provide between 15% to 100% more PPD depending on the type of simulation being done. Traditional Projects would see 15% to 35% improvement while Moonshot (free-energy) can see anywhere between 50% to 100% improvement.

[thomas2000]

Hi PantherX,

Nice to be here.

I've seen up to ~70% uplift already on some projects, so that's great. I love eficiency

[YoScot]

Try to uninstall all cuda sdk toolkit because they currently may interfer with FAH.

I'll try to debug / look around in the logs but is there a particular DLL it's having trouble loading ? I would like to keep the sdk around for other work.
thanks

[YoScot]

I found the following in science.log -- the interesting line was:

Code: Select all

     nvcc fatal:    Cannot find compiler 'cl.exe' in PATH

for some reason that line wasn't in the app's control log -- perhaps it goes to stderr?
i was able to add the compiler path to system envs and everything worked:
e.g.

Code: Select all

c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64

Code: Select all

Attempting to create CUDA context:
  Configuring platform CUDA
    Setting CUDA platform DeviceIndex to 0
    Disabling UseBlockingSync on CUDA platform
    Setting platform precision to mixed
    Setting DisablePmeStream to 1    
nvcc fatal   : Cannot find compiler 'cl.exe' in PATH
Failed to create CUDA context:
Error loading CUDA module: CUDA_ERROR_FILE_NOT_FOUND (301)

[PantherX]

Please note that this is a known bug that the developers are aware off and working with Nvidia to resolve. There's no ETA but it hasn't been forgotten. Let's hope that it can be resolved soon

[florinandrei]

I also see this bug when CUDA 10.0 is installed. I've uninstalled CUDA and now FAH can actually use CUDA, but it's not an optimal solution. Hopefully, the bug will get fixed soon so we can keep CUDA installed on systems running FAH.

[bruce]

NVidia's backward compatibility is being used which can support a variety of hardware platforms, allowing the FAHCore to work in a variety of environments. Very few people have CUDA installed separately and their installation is customized to work (only) with their hardware species. The FAHCore downloads everything it needs but it also requires that we deprecate Fermi hardware. Apparently the runtime code is incorporated into the OS kernel.

If you have an incompatible installation which is installed specifically to match your hardware, you will somehow need to manage the Path and whatever other environmental variables that differentiate the FAHCore environment from your custom CUDA environment. You can use OpenCL.

See also viewtopic.php?f=61&t=36721