Page 1 of 1

NVIDIA QUADRO 2000 suddenly stopped folding!

Posted: Sun May 17, 2020 6:42 pm
by neil_en_suite
Hello,
first of all sorry if this post is similar to any other, I have been looking through the posts but not found a solution thats worked yet.

I have an old CAD station with a NVIDIA QUADRO 2000 (GF 106GL)
Up until a week ago, the card was folding well. I gave the machine periods of rest also to ensure I avoided any overheating.
I have had a windows update recently but the card continued to fold, then suddenly one day it stopped.

I have taken a snap of the log file, when the fold process starts up and you will see some error messages but I am not sure what they mean.
Unfortunately I dont have any only image account to link images to but needless to say on both the browser interface and the client interface the folding status looks as if paused.
- On the browser the indicator is yellow, with the status bar at 0%; work Unit(PRCG) = 0 ; work Unit(eta) = 0
- on the client under the GPU; The folding slot says Ready; The work ques says 'download'; all the info under the Selected work Unit pane, wither says 0 or unknown.

- I have re-installed the windows driver and it installed the same as before however the only difference I see is the driver says QUADRO 2000 with a "D" on the end
- I have downloaded and installed the NVIDIA driver mgmt software which I believe also updates the driver.
- I have uninstalled the FAH client, restarted then installed the latest FAH client
- I followed another forum post that suggested to delete the folder assigned tot he GPU within the FAH work folder within program files. Unfortunately there was no folder to delete.

None of these steps have worked.

I notice within the FAH work directory, there is only a folder for the CPU "01"; occasionally another folder appears and then disappears. this must be the client attempting to create a job for the GPU.

Not sure what else to do, so any suggestions or advice would be gratefully received.
Thanks in advance anyway for any help.


Below is the log file, from when the client first starts up. Near the bottom are entries with errors.

Code: Select all

  *********************** Log Started 2020-05-16T09:09:04Z ***********************
09:09:04:Trying to access database...
09:09:04:Successfully acquired database lock
09:09:04:Read GPUs.txt
09:09:05:Enabled folding slot 00: READY cpu:7
09:09:05:Enabled folding slot 01: READY gpu:0:GF106GL [Quadro 2000]
09:09:05:****************************** FAHClient ******************************
09:09:05:        Version: 7.6.13
09:09:05:         Author: Joseph Coffland <[email protected]>
09:09:05:      Copyright: 2020 foldingathome.org
09:09:05:       Homepage: https://foldingathome.org/
09:09:05:           Date: Apr 27 2020
09:09:05:           Time: 21:21:01
09:09:05:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
09:09:05:         Branch: master
09:09:05:       Compiler: Visual C++ 2008
09:09:05:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:09:05:       Platform: win32 10
09:09:05:           Bits: 32
09:09:05:           Mode: Release
09:09:05:           Args: --open-web-control
09:09:05:         Config: C:\Users\neil_\AppData\Roaming\FAHClient\config.xml
09:09:05:******************************** CBang ********************************
09:09:05:           Date: Apr 24 2020
09:09:05:           Time: 17:07:55
09:09:05:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
09:09:05:         Branch: master
09:09:05:       Compiler: Visual C++ 2008
09:09:05:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:09:05:       Platform: win32 10
09:09:05:           Bits: 32
09:09:05:           Mode: Release
09:09:05:******************************* System ********************************
09:09:05:            CPU: Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
09:09:05:         CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
09:09:05:           CPUs: 8
09:09:05:         Memory: 19.92GiB
09:09:05:    Free Memory: 17.20GiB
09:09:05:        Threads: WINDOWS_THREADS
09:09:05:     OS Version: 6.2
09:09:05:    Has Battery: false
09:09:05:     On Battery: false
09:09:05:     UTC Offset: 1
09:09:05:            PID: 1420
09:09:05:            CWD: C:\Users\neil_\AppData\Roaming\FAHClient
09:09:05:  Win32 Service: false
09:09:05:             OS: Windows 10 Enterprise
09:09:05:        OS Arch: AMD64
09:09:05:           GPUs: 1
09:09:05:          GPU 0: Bus:4 Slot:0 Func:0 NVIDIA:1 GF106GL [Quadro 2000]
09:09:05:  CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:2.1 Driver:8.0
09:09:05:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.1 Driver:377.83
09:09:05:******************************* libFAH ********************************
09:09:05:           Date: Apr 15 2020
09:09:05:           Time: 14:53:14
09:09:05:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
09:09:05:         Branch: master
09:09:05:       Compiler: Visual C++ 2008
09:09:05:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:09:05:       Platform: win32 10
09:09:05:           Bits: 32
09:09:05:           Mode: Release
09:09:05:***********************************************************************
09:09:05:<config>
09:09:05:  <!-- Network -->
09:09:05:  <proxy v=':8080'/>
09:09:05:
09:09:05:  <!-- Slot Control -->
09:09:05:  <power v='full'/>
09:09:05:
09:09:05:  <!-- User Information -->
09:09:05:  <passkey v='*****'/>
09:09:05:  <team v='262280'/>
09:09:05:  <user v='neil_en_suite'/>
09:09:05:
09:09:05:  <!-- Folding Slots -->
09:09:05:  <slot id='0' type='CPU'/>
09:09:05:  <slot id='1' type='GPU'/>
09:09:05:</config>
09:09:05:WU01:FS00:Starting
09:09:05:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\neil_\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 01 -suffix 01 -version 706 -lifeline 1420 -checkpoint 15 -np 7
09:09:05:WU01:FS00:Started FahCore on PID 13268
09:09:05:WU01:FS00:Core PID:3476
09:09:05:WU01:FS00:FahCore 0xa7 started
09:09:06:WU01:FS00:0xa7:*********************** Log Started 2020-05-16T09:09:05Z ***********************
09:09:06:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
09:09:06:WU01:FS00:0xa7:       Type: 0xa7
09:09:06:WU01:FS00:0xa7:       Core: Gromacs
09:09:06:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 706 -lifeline 13268 -checkpoint 15 -np
09:09:06:WU01:FS00:0xa7:             7
09:09:06:WU01:FS00:0xa7:************************************ CBang *************************************
09:09:06:WU01:FS00:0xa7:       Date: Oct 26 2019
09:09:06:WU01:FS00:0xa7:       Time: 01:38:25
09:09:06:WU01:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
09:09:06:WU01:FS00:0xa7:     Branch: master
09:09:06:WU01:FS00:0xa7:   Compiler: Visual C++ 2008
09:09:06:WU01:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:09:06:WU01:FS00:0xa7:   Platform: win32 10
09:09:06:WU01:FS00:0xa7:       Bits: 64
09:09:06:WU01:FS00:0xa7:       Mode: Release
09:09:06:WU01:FS00:0xa7:************************************ System ************************************
09:09:06:WU01:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
09:09:06:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
09:09:06:WU01:FS00:0xa7:       CPUs: 8
09:09:06:WU01:FS00:0xa7:     Memory: 19.92GiB
09:09:06:WU01:FS00:0xa7:Free Memory: 17.19GiB
09:09:06:WU00:FS01:Connecting to assign1.foldingathome.org:80
09:09:06:WU01:FS00:0xa7:    Threads: WINDOWS_THREADS
09:09:06:WU01:FS00:0xa7: OS Version: 6.2
09:09:06:WU01:FS00:0xa7:Has Battery: false
09:09:06:WU01:FS00:0xa7: On Battery: false
09:09:06:WU01:FS00:0xa7: UTC Offset: 1
09:09:06:WU01:FS00:0xa7:        PID: 3476
09:09:06:WU01:FS00:0xa7:        CWD: C:\Users\neil_\AppData\Roaming\FAHClient\work
09:09:06:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
09:09:06:WU01:FS00:0xa7:    Version: 0.0.18
09:09:06:WU01:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
09:09:06:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
09:09:06:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
09:09:06:WU01:FS00:0xa7:       Date: Oct 26 2019
09:09:06:WU01:FS00:0xa7:       Time: 01:52:30
09:09:06:WU01:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
09:09:06:WU01:FS00:0xa7:     Branch: master
09:09:06:WU01:FS00:0xa7:   Compiler: Visual C++ 2008
09:09:06:WU01:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:09:06:WU01:FS00:0xa7:   Platform: win32 10
09:09:06:WU01:FS00:0xa7:       Bits: 64
09:09:06:WU01:FS00:0xa7:       Mode: Release
09:09:06:WU01:FS00:0xa7:************************************ Build *************************************
09:09:06:WU01:FS00:0xa7:       SIMD: avx_256
09:09:06:WU01:FS00:0xa7:********************************************************************************
09:09:06:WU01:FS00:0xa7:Project: 16801 (Run 13, Clone 955, Gen 23)
09:09:06:WU01:FS00:0xa7:Unit: 0x0000001e82ed0b915e959440c5e34827
09:09:06:WU01:FS00:0xa7:Digital signatures verified
09:09:06:WU01:FS00:0xa7:Reducing thread count from 7 to 6 to avoid domain decomposition by a prime number > 3
09:09:06:WU01:FS00:0xa7:Calling: mdrun -s frame23.tpr -o frame23.trr -cpi state.cpt -cpt 15 -nt 6
09:09:06:WU01:FS00:0xa7:Steps: first=11500000 total=500000
09:09:07:WU01:FS00:0xa7:Completed 154517 out of 500000 steps (30%)
09:09:08:WU00:FS01:Assigned to work server 192.0.2.1
09:09:08:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GF106GL [Quadro 2000] from 192.0.2.1
09:09:08:WU00:FS01:Connecting to 192.0.2.1:8080
09:09:08:4:127.0.0.1:New Web session
09:09:29:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
09:09:29:WU00:FS01:Connecting to 192.0.2.1:80
09:09:42:WU01:FS00:0xa7:Completed 155000 out of 500000 steps (31%)
09:09:50:ERROR:WU00:FS01:Exception: Failed to connect to 192.0.2.1:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
09:09:50:WU00:FS01:Connecting to assign1.foldingathome.org:80
09:09:50:WU00:FS01:Assigned to work server 192.0.2.1
09:09:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GF106GL [Quadro 2000] from 192.0.2.1
09:09:50:WU00:FS01:Connecting to 192.0.2.1:8080
09:10:11:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
09:10:11:WU00:FS01:Connecting to 192.0.2.1:80
09:10:33:ERROR:WU00:FS01:Exception: Failed to connect to 192.0.2.1:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
09:10:50:WU00:FS01:Connecting to assign1.foldingathome.org:80
09:10:50:WU00:FS01:Assigned to work server 192.0.2.1
09:10:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GF106GL [Quadro 2000] from 192.0.2.1
09:10:50:WU00:FS01:Connecting to 192.0.2.1:8080

Re: NVIDIA QUADRO 2000 suddenly stopped folding!

Posted: Sun May 17, 2020 7:15 pm
by Joe_H
This response - viewtopic.php?f=74&t=35220&p=334019#p334019, and the topic it is in applies. I will bring your post to his attention, the Quadro 2000 may have been misclassified during the changes mentioned.

Re: NVIDIA QUADRO 2000 suddenly stopped folding!

Posted: Sun May 17, 2020 7:23 pm
by neil_en_suite
Thank you. I will have a read.

Re: NVIDIA QUADRO 2000 suddenly stopped folding!

Posted: Sun May 17, 2020 10:23 pm
by FireFox-89
Also since it is an old card I would be tempted to strip the card down and clean it out along with some new thermal paste during its downtime.

Re: NVIDIA QUADRO 2000 suddenly stopped folding!

Posted: Sun May 17, 2020 10:48 pm
by bruce
Fixed.
:oops: