3 GPUS say "Failed" (red background) after a few retries.

Moderators: Site Moderators, FAHC Science Team

Post Reply
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

3 GPUS say "Failed" (red background) after a few retries.

Post by Rwolf01 »

CPU slot is working, but all my real action is from the GPUs. I tried uninstalling FAH, including deleting all data, and then reinstalling, but it still does it.

I have 3 other workstations running the same windows, drivers and GPUs and they work. Only one of my workstations is messed up. It happens to be the one with Dual Xeon processors. (Dell T5610. My other workstations are Dell T3610s) The GPUs are Zotac P106-090 cards.

Note: This system used to work. I was offline for a few weeks during the heatwave, and just decided to startup. I'm all caught up on windows updates on all 4 machines.

Filtering any of my GPU slots for warnings & errors gives:

Code: Select all

*********************** Log Started 2020-09-15T07:22:51Z ***********************
07:23:49:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:20:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:48:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:25:13:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:25:47:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:07:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:24:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:49:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:27:14:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:27:32:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
Does anyone have suggestions for how I can get these GPUs productive again?
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

The (almost) complete log for my first GPU slot is:

Note: I had to chop one retry cycle out of the middle because of the 60k character limit.

Code: Select all

*********************** Log Started 2020-09-15T07:22:51Z ***********************
07:23:15:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:23:15:WU01:FS01:Assigned to work server 18.188.125.154
07:23:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:23:15:WU01:FS01:Connecting to 18.188.125.154:8080
07:23:16:WU01:FS01:Downloading 6.14MiB
07:23:22:WU01:FS01:Download 66.19%
07:23:24:WU01:FS01:Download complete
07:23:24:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3165 clone:15 gen:3 core:0x22 unit:0x0000000a12bc7d9a5f597cb670aca406
07:23:27:WU01:FS01:Starting
07:23:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:23:27:WU01:FS01:Started FahCore on PID 13088
07:23:27:WU01:FS01:Core PID:2060
07:23:27:WU01:FS01:FahCore 0x22 started
07:23:28:WU01:FS01:0x22:*********************** Log Started 2020-09-15T07:23:27Z ***********************
07:23:28:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:23:28:WU01:FS01:0x22:       Core: Core22
07:23:28:WU01:FS01:0x22:       Type: 0x22
07:23:28:WU01:FS01:0x22:    Version: 0.0.11
07:23:28:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:23:28:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:23:28:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU01:FS01:0x22:       Time: 19:49:16
07:23:28:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:28:WU01:FS01:0x22:     Branch: core22-0.0.11
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:28:WU01:FS01:0x22:             <[email protected]>
07:23:28:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 13088 -checkpoint 15
07:23:28:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:23:28:WU01:FS01:0x22:             1 -gpu 1
07:23:28:WU01:FS01:0x22:************************************ libFAH ************************************
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU01:FS01:0x22:       Time: 19:47:12
07:23:28:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:28:WU01:FS01:0x22:     Branch: HEAD
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:************************************ CBang *************************************
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU01:FS01:0x22:       Time: 19:46:11
07:23:28:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:28:WU01:FS01:0x22:     Branch: master
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:************************************ System ************************************
07:23:28:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:28:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:28:WU01:FS01:0x22:       CPUs: 32
07:23:28:WU01:FS01:0x22:     Memory: 31.95GiB
07:23:28:WU01:FS01:0x22:Free Memory: 28.68GiB
07:23:28:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:23:28:WU01:FS01:0x22: OS Version: 6.2
07:23:28:WU01:FS01:0x22:Has Battery: false
07:23:28:WU01:FS01:0x22: On Battery: false
07:23:28:WU01:FS01:0x22: UTC Offset: -7
07:23:28:WU01:FS01:0x22:        PID: 2060
07:23:28:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:28:WU01:FS01:0x22:********************************************************************************
07:23:28:WU01:FS01:0x22:Project: 13426 (Run 3165, Clone 15, Gen 3)
07:23:28:WU01:FS01:0x22:Unit: 0x0000000a12bc7d9a5f597cb670aca406
07:23:28:WU01:FS01:0x22:Reading tar file core.xml
07:23:28:WU01:FS01:0x22:Reading tar file integrator.xml
07:23:28:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:23:28:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:23:28:WU01:FS01:0x22:Digital signatures verified
07:23:28:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:23:28:WU01:FS01:0x22:Version 0.0.11
07:23:28:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:23:28:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:23:28:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:23:28:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:23:49:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:23:49:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:23:49:WU01:FS01:0x22:Saving result file science.log
07:23:49:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:23:49:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:23:49:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3165 clone:15 gen:3 core:0x22 unit:0x0000000a12bc7d9a5f597cb670aca406
07:23:49:WU01:FS01:Uploading 2.44KiB to 18.188.125.154
07:23:49:WU01:FS01:Connecting to 18.188.125.154:8080
07:23:50:WU01:FS01:Upload complete
07:23:50:WU01:FS01:Server responded WORK_ACK (400)
07:23:50:WU01:FS01:Cleaning up
07:23:50:WU03:FS01:Connecting to assign1.foldingathome.org:80
07:23:50:WU03:FS01:Assigned to work server 18.188.125.154
07:23:50:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:23:50:WU03:FS01:Connecting to 18.188.125.154:8080
07:23:51:WU03:FS01:Downloading 6.14MiB
07:23:57:WU03:FS01:Download 66.21%
07:23:59:WU03:FS01:Download complete
07:23:59:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13426 run:3583 clone:6 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbf48de1efa
07:23:59:WU03:FS01:Starting
07:23:59:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:23:59:WU03:FS01:Started FahCore on PID 11052
07:23:59:WU03:FS01:Core PID:11056
07:23:59:WU03:FS01:FahCore 0x22 started
07:23:59:WU03:FS01:0x22:*********************** Log Started 2020-09-15T07:23:59Z ***********************
07:23:59:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:23:59:WU03:FS01:0x22:       Core: Core22
07:23:59:WU03:FS01:0x22:       Type: 0x22
07:23:59:WU03:FS01:0x22:    Version: 0.0.11
07:23:59:WU03:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:23:59:WU03:FS01:0x22:  Copyright: 2020 foldingathome.org
07:23:59:WU03:FS01:0x22:   Homepage: https://foldingathome.org/
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:49:16
07:23:59:WU03:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:59:WU03:FS01:0x22:     Branch: core22-0.0.11
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:59:WU03:FS01:0x22:             <[email protected]>
07:23:59:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 11052 -checkpoint 15
07:23:59:WU03:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:23:59:WU03:FS01:0x22:             1 -gpu 1
07:23:59:WU03:FS01:0x22:************************************ libFAH ************************************
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:47:12
07:23:59:WU03:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:59:WU03:FS01:0x22:     Branch: HEAD
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:************************************ CBang *************************************
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:46:11
07:23:59:WU03:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:59:WU03:FS01:0x22:     Branch: master
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:************************************ System ************************************
07:23:59:WU03:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:59:WU03:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:59:WU03:FS01:0x22:       CPUs: 32
07:23:59:WU03:FS01:0x22:     Memory: 31.95GiB
07:23:59:WU03:FS01:0x22:Free Memory: 28.37GiB
07:23:59:WU03:FS01:0x22:    Threads: WINDOWS_THREADS
07:23:59:WU03:FS01:0x22: OS Version: 6.2
07:23:59:WU03:FS01:0x22:Has Battery: false
07:23:59:WU03:FS01:0x22: On Battery: false
07:24:00:WU03:FS01:0x22: UTC Offset: -7
07:24:00:WU03:FS01:0x22:        PID: 11056
07:24:00:WU03:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:00:WU03:FS01:0x22:********************************************************************************
07:24:00:WU03:FS01:0x22:Project: 13426 (Run 3583, Clone 6, Gen 2)
07:24:00:WU03:FS01:0x22:Unit: 0x0000000212bc7d9a5f597cbf48de1efa
07:24:00:WU03:FS01:0x22:Reading tar file core.xml
07:24:00:WU03:FS01:0x22:Reading tar file integrator.xml
07:24:00:WU03:FS01:0x22:Reading tar file state.xml.bz2
07:24:00:WU03:FS01:0x22:Reading tar file system.xml.bz2
07:24:00:WU03:FS01:0x22:Digital signatures verified
07:24:00:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:24:00:WU03:FS01:0x22:Version 0.0.11
07:24:00:WU03:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:24:00:WU03:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:24:00:WU03:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:24:00:WU03:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:24:20:WU03:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:24:20:WU03:FS01:0x22:Saving result file ..\logfile_01.txt
07:24:20:WU03:FS01:0x22:Saving result file science.log
07:24:20:WU03:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:20:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:20:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:13426 run:3583 clone:6 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbf48de1efa
07:24:20:WU03:FS01:Uploading 2.44KiB to 18.188.125.154
07:24:21:WU03:FS01:Connecting to 18.188.125.154:8080
07:24:21:WU03:FS01:Upload complete
07:24:21:WU04:FS01:Connecting to assign1.foldingathome.org:80
07:24:21:WU03:FS01:Server responded WORK_ACK (400)
07:24:21:WU03:FS01:Cleaning up
07:24:21:WU04:FS01:Assigned to work server 18.188.125.154
07:24:21:WU04:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:24:21:WU04:FS01:Connecting to 18.188.125.154:8080
07:24:22:WU04:FS01:Downloading 6.14MiB
07:24:26:WU04:FS01:Download complete
07:24:26:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:13426 run:3173 clone:9 gen:2 core:0x22 unit:0x0000000312bc7d9a5f597cb6b6949e31
07:24:26:WU04:FS01:Starting
07:24:26:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 04 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:24:26:WU04:FS01:Started FahCore on PID 12932
07:24:26:WU04:FS01:Core PID:7188
07:24:26:WU04:FS01:FahCore 0x22 started
07:24:27:WU04:FS01:0x22:*********************** Log Started 2020-09-15T07:24:26Z ***********************
07:24:27:WU04:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:24:27:WU04:FS01:0x22:       Core: Core22
07:24:27:WU04:FS01:0x22:       Type: 0x22
07:24:27:WU04:FS01:0x22:    Version: 0.0.11
07:24:27:WU04:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:24:27:WU04:FS01:0x22:  Copyright: 2020 foldingathome.org
07:24:27:WU04:FS01:0x22:   Homepage: https://foldingathome.org/
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:49:16
07:24:27:WU04:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:24:27:WU04:FS01:0x22:     Branch: core22-0.0.11
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:24:27:WU04:FS01:0x22:             <[email protected]>
07:24:27:WU04:FS01:0x22:       Args: -dir 04 -suffix 01 -version 706 -lifeline 12932 -checkpoint 15
07:24:27:WU04:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:24:27:WU04:FS01:0x22:             1 -gpu 1
07:24:27:WU04:FS01:0x22:************************************ libFAH ************************************
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:47:12
07:24:27:WU04:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:24:27:WU04:FS01:0x22:     Branch: HEAD
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:************************************ CBang *************************************
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:46:11
07:24:27:WU04:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:24:27:WU04:FS01:0x22:     Branch: master
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:************************************ System ************************************
07:24:27:WU04:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:24:27:WU04:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:24:27:WU04:FS01:0x22:       CPUs: 32
07:24:27:WU04:FS01:0x22:     Memory: 31.95GiB
07:24:27:WU04:FS01:0x22:Free Memory: 28.52GiB
07:24:27:WU04:FS01:0x22:    Threads: WINDOWS_THREADS
07:24:27:WU04:FS01:0x22: OS Version: 6.2
07:24:27:WU04:FS01:0x22:Has Battery: false
07:24:27:WU04:FS01:0x22: On Battery: false
07:24:27:WU04:FS01:0x22: UTC Offset: -7
07:24:27:WU04:FS01:0x22:        PID: 7188
07:24:27:WU04:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:27:WU04:FS01:0x22:********************************************************************************
07:24:27:WU04:FS01:0x22:Project: 13426 (Run 3173, Clone 9, Gen 2)
07:24:27:WU04:FS01:0x22:Unit: 0x0000000312bc7d9a5f597cb6b6949e31
07:24:27:WU04:FS01:0x22:Reading tar file core.xml
07:24:27:WU04:FS01:0x22:Reading tar file integrator.xml
07:24:27:WU04:FS01:0x22:Reading tar file state.xml.bz2
07:24:27:WU04:FS01:0x22:Reading tar file system.xml.bz2
07:24:27:WU04:FS01:0x22:Digital signatures verified
07:24:27:WU04:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:24:27:WU04:FS01:0x22:Version 0.0.11
07:24:27:WU04:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:24:27:WU04:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:24:27:WU04:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:24:27:WU04:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:24:48:WU04:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:24:48:WU04:FS01:0x22:Saving result file ..\logfile_01.txt
07:24:48:WU04:FS01:0x22:Saving result file science.log
07:24:48:WU04:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:48:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:48:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:13426 run:3173 clone:9 gen:2 core:0x22 unit:0x0000000312bc7d9a5f597cb6b6949e31
07:24:48:WU04:FS01:Uploading 2.44KiB to 18.188.125.154
07:24:48:WU04:FS01:Connecting to 18.188.125.154:8080
07:24:48:WU04:FS01:Upload complete
07:24:49:WU04:FS01:Server responded WORK_ACK (400)
07:24:49:WU04:FS01:Cleaning up
07:24:49:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:24:49:WU02:FS01:Assigned to work server 18.188.125.154
07:24:49:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:24:49:WU02:FS01:Connecting to 18.188.125.154:8080
07:24:50:WU02:FS01:Downloading 7.37MiB
07:24:54:WU02:FS01:Download complete

// -- One retry cycle deleted to make it fit....
 
07:26:04:WU02:FS01:Download complete
07:26:04:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:16920 run:25 clone:28 gen:125 core:0x22 unit:0x000000b60002894c5f2a13cd16f65293
07:26:04:WU02:FS01:Starting
07:26:04:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:26:04:WU02:FS01:Started FahCore on PID 9076
07:26:04:WU02:FS01:Core PID:10516
07:26:04:WU02:FS01:FahCore 0x22 started
07:26:04:WU02:FS01:0x22:*********************** Log Started 2020-09-15T07:26:04Z ***********************
07:26:04:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:26:04:WU02:FS01:0x22:       Core: Core22
07:26:04:WU02:FS01:0x22:       Type: 0x22
07:26:04:WU02:FS01:0x22:    Version: 0.0.11
07:26:04:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:26:04:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:26:04:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:26:04:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:04:WU02:FS01:0x22:       Time: 19:49:16
07:26:04:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:26:04:WU02:FS01:0x22:     Branch: core22-0.0.11
07:26:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:04:WU02:FS01:0x22:   Platform: win32 10
07:26:04:WU02:FS01:0x22:       Bits: 64
07:26:04:WU02:FS01:0x22:       Mode: Release
07:26:04:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:26:04:WU02:FS01:0x22:             <[email protected]>
07:26:04:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 9076 -checkpoint 15
07:26:04:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:26:04:WU02:FS01:0x22:             1 -gpu 1
07:26:04:WU02:FS01:0x22:************************************ libFAH ************************************
07:26:04:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:04:WU02:FS01:0x22:       Time: 19:47:12
07:26:04:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:26:04:WU02:FS01:0x22:     Branch: HEAD
07:26:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:04:WU02:FS01:0x22:   Platform: win32 10
07:26:04:WU02:FS01:0x22:       Bits: 64
07:26:04:WU02:FS01:0x22:       Mode: Release
07:26:04:WU02:FS01:0x22:************************************ CBang *************************************
07:26:04:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:04:WU02:FS01:0x22:       Time: 19:46:11
07:26:04:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:26:04:WU02:FS01:0x22:     Branch: master
07:26:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:04:WU02:FS01:0x22:   Platform: win32 10
07:26:04:WU02:FS01:0x22:       Bits: 64
07:26:04:WU02:FS01:0x22:       Mode: Release
07:26:04:WU02:FS01:0x22:************************************ System ************************************
07:26:04:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:26:04:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:26:04:WU02:FS01:0x22:       CPUs: 32
07:26:04:WU02:FS01:0x22:     Memory: 31.95GiB
07:26:04:WU02:FS01:0x22:Free Memory: 28.15GiB
07:26:04:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:26:04:WU02:FS01:0x22: OS Version: 6.2
07:26:04:WU02:FS01:0x22:Has Battery: false
07:26:04:WU02:FS01:0x22: On Battery: false
07:26:04:WU02:FS01:0x22: UTC Offset: -7
07:26:04:WU02:FS01:0x22:        PID: 10516
07:26:04:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:26:04:WU02:FS01:0x22:********************************************************************************
07:26:04:WU02:FS01:0x22:Project: 16920 (Run 25, Clone 28, Gen 125)
07:26:04:WU02:FS01:0x22:Unit: 0x000000b60002894c5f2a13cd16f65293
07:26:04:WU02:FS01:0x22:Reading tar file core.xml
07:26:04:WU02:FS01:0x22:Reading tar file integrator.xml
07:26:04:WU02:FS01:0x22:Reading tar file state.xml
07:26:04:WU02:FS01:0x22:Reading tar file system.xml
07:26:04:WU02:FS01:0x22:Digital signatures verified
07:26:04:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:26:04:WU02:FS01:0x22:Version 0.0.11
07:26:04:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:26:04:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:26:04:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:26:04:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:26:07:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:26:07:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:26:07:WU02:FS01:0x22:Saving result file science.log
07:26:07:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:26:07:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:07:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:16920 run:25 clone:28 gen:125 core:0x22 unit:0x000000b60002894c5f2a13cd16f65293
07:26:07:WU02:FS01:Uploading 10.00KiB to 155.247.166.220
07:26:07:WU02:FS01:Connecting to 155.247.166.220:8080
07:26:08:WU02:FS01:Upload complete
07:26:08:WU02:FS01:Server responded WORK_ACK (400)
07:26:08:WU02:FS01:Cleaning up
07:26:08:WU04:FS01:Connecting to assign1.foldingathome.org:80
07:26:08:WU04:FS01:Assigned to work server 18.188.125.154
07:26:08:WU04:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:26:08:WU04:FS01:Connecting to 18.188.125.154:8080
07:26:09:WU04:FS01:Downloading 4.95MiB
07:26:11:WU04:FS01:Download complete
07:26:11:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:13431 run:62 clone:3 gen:5 core:0x22 unit:0x0000000612bc7d9a5f5e3239dfe75415
07:26:11:WU04:FS01:Starting
07:26:11:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 04 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:26:11:WU04:FS01:Started FahCore on PID 9168
07:26:11:WU04:FS01:Core PID:8544
07:26:11:WU04:FS01:FahCore 0x22 started
07:26:12:WU04:FS01:0x22:*********************** Log Started 2020-09-15T07:26:11Z ***********************
07:26:12:WU04:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:26:12:WU04:FS01:0x22:       Core: Core22
07:26:12:WU04:FS01:0x22:       Type: 0x22
07:26:12:WU04:FS01:0x22:    Version: 0.0.11
07:26:12:WU04:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:26:12:WU04:FS01:0x22:  Copyright: 2020 foldingathome.org
07:26:12:WU04:FS01:0x22:   Homepage: https://foldingathome.org/
07:26:12:WU04:FS01:0x22:       Date: Jun 26 2020
07:26:12:WU04:FS01:0x22:       Time: 19:49:16
07:26:12:WU04:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:26:12:WU04:FS01:0x22:     Branch: core22-0.0.11
07:26:12:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:26:12:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:12:WU04:FS01:0x22:   Platform: win32 10
07:26:12:WU04:FS01:0x22:       Bits: 64
07:26:12:WU04:FS01:0x22:       Mode: Release
07:26:12:WU04:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:26:12:WU04:FS01:0x22:             <[email protected]>
07:26:12:WU04:FS01:0x22:       Args: -dir 04 -suffix 01 -version 706 -lifeline 9168 -checkpoint 15
07:26:12:WU04:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:26:12:WU04:FS01:0x22:             1 -gpu 1
07:26:12:WU04:FS01:0x22:************************************ libFAH ************************************
07:26:12:WU04:FS01:0x22:       Date: Jun 26 2020
07:26:12:WU04:FS01:0x22:       Time: 19:47:12
07:26:12:WU04:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:26:12:WU04:FS01:0x22:     Branch: HEAD
07:26:12:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:26:12:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:12:WU04:FS01:0x22:   Platform: win32 10
07:26:12:WU04:FS01:0x22:       Bits: 64
07:26:12:WU04:FS01:0x22:       Mode: Release
07:26:12:WU04:FS01:0x22:************************************ CBang *************************************
07:26:12:WU04:FS01:0x22:       Date: Jun 26 2020
07:26:12:WU04:FS01:0x22:       Time: 19:46:11
07:26:12:WU04:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:26:12:WU04:FS01:0x22:     Branch: master
07:26:12:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:26:12:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:12:WU04:FS01:0x22:   Platform: win32 10
07:26:12:WU04:FS01:0x22:       Bits: 64
07:26:12:WU04:FS01:0x22:       Mode: Release
07:26:12:WU04:FS01:0x22:************************************ System ************************************
07:26:12:WU04:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:26:12:WU04:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:26:12:WU04:FS01:0x22:       CPUs: 32
07:26:12:WU04:FS01:0x22:     Memory: 31.95GiB
07:26:12:WU04:FS01:0x22:Free Memory: 28.42GiB
07:26:12:WU04:FS01:0x22:    Threads: WINDOWS_THREADS
07:26:12:WU04:FS01:0x22: OS Version: 6.2
07:26:12:WU04:FS01:0x22:Has Battery: false
07:26:12:WU04:FS01:0x22: On Battery: false
07:26:12:WU04:FS01:0x22: UTC Offset: -7
07:26:12:WU04:FS01:0x22:        PID: 8544
07:26:12:WU04:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:26:12:WU04:FS01:0x22:********************************************************************************
07:26:12:WU04:FS01:0x22:Project: 13431 (Run 62, Clone 3, Gen 5)
07:26:12:WU04:FS01:0x22:Unit: 0x0000000612bc7d9a5f5e3239dfe75415
07:26:12:WU04:FS01:0x22:Reading tar file core.xml
07:26:12:WU04:FS01:0x22:Reading tar file integrator.xml.bz2
07:26:12:WU04:FS01:0x22:Reading tar file state.xml.bz2
07:26:12:WU04:FS01:0x22:Reading tar file system.xml.bz2
07:26:12:WU04:FS01:0x22:Digital signatures verified
07:26:12:WU04:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:26:12:WU04:FS01:0x22:Version 0.0.11
07:26:12:WU04:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:26:12:WU04:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:26:12:WU04:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:26:12:WU04:FS01:0x22:  Global context and integrator variables write interval: disabled
07:26:23:WU04:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:26:23:WU04:FS01:0x22:Saving result file ..\logfile_01.txt
07:26:23:WU04:FS01:0x22:Saving result file science.log
07:26:23:WU04:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:26:24:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:24:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:13431 run:62 clone:3 gen:5 core:0x22 unit:0x0000000612bc7d9a5f5e3239dfe75415
07:26:24:WU04:FS01:Uploading 2.44KiB to 18.188.125.154
07:26:24:WU04:FS01:Connecting to 18.188.125.154:8080
07:26:24:WU04:FS01:Upload complete
07:26:24:WU04:FS01:Server responded WORK_ACK (400)
07:26:24:WU04:FS01:Cleaning up
07:26:24:WU03:FS01:Connecting to assign1.foldingathome.org:80
07:26:24:WU03:FS01:Assigned to work server 66.170.111.50
07:26:24:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 66.170.111.50
07:26:24:WU03:FS01:Connecting to 66.170.111.50:8080
07:26:25:WU03:FS01:Downloading 10.20MiB
07:26:27:WU03:FS01:Download complete
07:26:27:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14487 run:0 clone:1775 gen:9 core:0x22 unit:0x0000001142aa6f325f45dea842425836
07:26:28:WU03:FS01:Starting
07:26:28:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:26:28:WU03:FS01:Started FahCore on PID 9760
07:26:28:WU03:FS01:Core PID:10536
07:26:28:WU03:FS01:FahCore 0x22 started
07:26:28:WU03:FS01:0x22:*********************** Log Started 2020-09-15T07:26:28Z ***********************
07:26:28:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:26:28:WU03:FS01:0x22:       Core: Core22
07:26:28:WU03:FS01:0x22:       Type: 0x22
07:26:28:WU03:FS01:0x22:    Version: 0.0.11
07:26:28:WU03:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:26:28:WU03:FS01:0x22:  Copyright: 2020 foldingathome.org
07:26:28:WU03:FS01:0x22:   Homepage: https://foldingathome.org/
07:26:28:WU03:FS01:0x22:       Date: Jun 26 2020
07:26:28:WU03:FS01:0x22:       Time: 19:49:16
07:26:28:WU03:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:26:28:WU03:FS01:0x22:     Branch: core22-0.0.11
07:26:28:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:26:28:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:28:WU03:FS01:0x22:   Platform: win32 10
07:26:28:WU03:FS01:0x22:       Bits: 64
07:26:28:WU03:FS01:0x22:       Mode: Release
07:26:28:WU03:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:26:28:WU03:FS01:0x22:             <[email protected]>
07:26:28:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 9760 -checkpoint 15
07:26:28:WU03:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:26:28:WU03:FS01:0x22:             1 -gpu 1
07:26:28:WU03:FS01:0x22:************************************ libFAH ************************************
07:26:28:WU03:FS01:0x22:       Date: Jun 26 2020
07:26:28:WU03:FS01:0x22:       Time: 19:47:12
07:26:28:WU03:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:26:28:WU03:FS01:0x22:     Branch: HEAD
07:26:28:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:26:28:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:28:WU03:FS01:0x22:   Platform: win32 10
07:26:28:WU03:FS01:0x22:       Bits: 64
07:26:28:WU03:FS01:0x22:       Mode: Release
07:26:28:WU03:FS01:0x22:************************************ CBang *************************************
07:26:28:WU03:FS01:0x22:       Date: Jun 26 2020
07:26:28:WU03:FS01:0x22:       Time: 19:46:11
07:26:28:WU03:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:26:28:WU03:FS01:0x22:     Branch: master
07:26:28:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:26:28:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:29:WU03:FS01:0x22:   Platform: win32 10
07:26:29:WU03:FS01:0x22:       Bits: 64
07:26:29:WU03:FS01:0x22:       Mode: Release
07:26:29:WU03:FS01:0x22:************************************ System ************************************
07:26:29:WU03:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:26:29:WU03:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:26:29:WU03:FS01:0x22:       CPUs: 32
07:26:29:WU03:FS01:0x22:     Memory: 31.95GiB
07:26:29:WU03:FS01:0x22:Free Memory: 28.10GiB
07:26:29:WU03:FS01:0x22:    Threads: WINDOWS_THREADS
07:26:29:WU03:FS01:0x22: OS Version: 6.2
07:26:29:WU03:FS01:0x22:Has Battery: false
07:26:29:WU03:FS01:0x22: On Battery: false
07:26:29:WU03:FS01:0x22: UTC Offset: -7
07:26:29:WU03:FS01:0x22:        PID: 10536
07:26:29:WU03:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:26:29:WU03:FS01:0x22:********************************************************************************
07:26:29:WU03:FS01:0x22:Project: 14487 (Run 0, Clone 1775, Gen 9)
07:26:29:WU03:FS01:0x22:Unit: 0x0000001142aa6f325f45dea842425836
07:26:29:WU03:FS01:0x22:Reading tar file core.xml
07:26:29:WU03:FS01:0x22:Reading tar file integrator.xml.bz2
07:26:29:WU03:FS01:0x22:Reading tar file state.xml.bz2
07:26:29:WU03:FS01:0x22:Reading tar file system.xml.bz2
07:26:29:WU03:FS01:0x22:Digital signatures verified
07:26:29:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:26:29:WU03:FS01:0x22:Version 0.0.11
07:26:29:WU03:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
07:26:29:WU03:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
07:26:29:WU03:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
07:26:29:WU03:FS01:0x22:  Global context and integrator variables write interval: disabled
07:26:49:WU03:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:26:49:WU03:FS01:0x22:Saving result file ..\logfile_01.txt
07:26:49:WU03:FS01:0x22:Saving result file science.log
07:26:49:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:26:49:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:14487 run:0 clone:1775 gen:9 core:0x22 unit:0x0000001142aa6f325f45dea842425836
07:26:49:WU03:FS01:Uploading 10.50KiB to 66.170.111.50
07:26:49:WU03:FS01:Connecting to 66.170.111.50:8080
07:26:49:WU03:FS01:Upload complete
07:26:49:WU03:FS01:Server responded WORK_ACK (400)
07:26:49:WU03:FS01:Cleaning up
07:26:50:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:26:50:WU02:FS01:Assigned to work server 66.170.111.50
07:26:50:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 66.170.111.50
07:26:50:WU02:FS01:Connecting to 66.170.111.50:8080
07:26:50:WU02:FS01:Downloading 10.19MiB
07:26:52:WU02:FS01:Download complete
07:26:52:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14489 run:0 clone:994 gen:20 core:0x22 unit:0x0000001a42aa6f325f45dea69f4d9280
07:26:52:WU02:FS01:Starting
07:26:52:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:26:52:WU02:FS01:Started FahCore on PID 4204
07:26:52:WU02:FS01:Core PID:10032
07:26:52:WU02:FS01:FahCore 0x22 started
07:26:53:WU02:FS01:0x22:*********************** Log Started 2020-09-15T07:26:52Z ***********************
07:26:53:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:26:53:WU02:FS01:0x22:       Core: Core22
07:26:53:WU02:FS01:0x22:       Type: 0x22
07:26:53:WU02:FS01:0x22:    Version: 0.0.11
07:26:53:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:26:53:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:26:53:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:26:53:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:53:WU02:FS01:0x22:       Time: 19:49:16
07:26:53:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:26:53:WU02:FS01:0x22:     Branch: core22-0.0.11
07:26:53:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:53:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:53:WU02:FS01:0x22:   Platform: win32 10
07:26:53:WU02:FS01:0x22:       Bits: 64
07:26:53:WU02:FS01:0x22:       Mode: Release
07:26:53:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:26:53:WU02:FS01:0x22:             <[email protected]>
07:26:53:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 4204 -checkpoint 15
07:26:53:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:26:53:WU02:FS01:0x22:             1 -gpu 1
07:26:53:WU02:FS01:0x22:************************************ libFAH ************************************
07:26:53:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:53:WU02:FS01:0x22:       Time: 19:47:12
07:26:53:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:26:53:WU02:FS01:0x22:     Branch: HEAD
07:26:53:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:53:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:53:WU02:FS01:0x22:   Platform: win32 10
07:26:53:WU02:FS01:0x22:       Bits: 64
07:26:53:WU02:FS01:0x22:       Mode: Release
07:26:53:WU02:FS01:0x22:************************************ CBang *************************************
07:26:53:WU02:FS01:0x22:       Date: Jun 26 2020
07:26:53:WU02:FS01:0x22:       Time: 19:46:11
07:26:53:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:26:53:WU02:FS01:0x22:     Branch: master
07:26:53:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:26:53:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:26:53:WU02:FS01:0x22:   Platform: win32 10
07:26:53:WU02:FS01:0x22:       Bits: 64
07:26:53:WU02:FS01:0x22:       Mode: Release
07:26:53:WU02:FS01:0x22:************************************ System ************************************
07:26:53:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:26:53:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:26:53:WU02:FS01:0x22:       CPUs: 32
07:26:53:WU02:FS01:0x22:     Memory: 31.95GiB
07:26:53:WU02:FS01:0x22:Free Memory: 28.04GiB
07:26:53:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:26:53:WU02:FS01:0x22: OS Version: 6.2
07:26:53:WU02:FS01:0x22:Has Battery: false
07:26:53:WU02:FS01:0x22: On Battery: false
07:26:53:WU02:FS01:0x22: UTC Offset: -7
07:26:53:WU02:FS01:0x22:        PID: 10032
07:26:53:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:26:53:WU02:FS01:0x22:********************************************************************************
07:26:53:WU02:FS01:0x22:Project: 14489 (Run 0, Clone 994, Gen 20)
07:26:53:WU02:FS01:0x22:Unit: 0x0000001a42aa6f325f45dea69f4d9280
07:26:53:WU02:FS01:0x22:Reading tar file core.xml
07:26:53:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:26:53:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:26:53:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:26:53:WU02:FS01:0x22:Digital signatures verified
07:26:53:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:26:53:WU02:FS01:0x22:Version 0.0.11
07:26:53:WU02:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
07:26:53:WU02:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
07:26:53:WU02:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
07:26:53:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:27:14:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:27:14:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:27:14:WU02:FS01:0x22:Saving result file science.log
07:27:14:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:27:14:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:27:14:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:14489 run:0 clone:994 gen:20 core:0x22 unit:0x0000001a42aa6f325f45dea69f4d9280
07:27:14:WU02:FS01:Uploading 10.50KiB to 66.170.111.50
07:27:14:WU02:FS01:Connecting to 66.170.111.50:8080
07:27:14:WU02:FS01:Upload complete
07:27:14:WU02:FS01:Server responded WORK_ACK (400)
07:27:14:WU02:FS01:Cleaning up
07:27:15:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:27:15:WU02:FS01:Assigned to work server 18.188.125.154
07:27:15:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:27:15:WU02:FS01:Connecting to 18.188.125.154:8080
07:27:16:WU02:FS01:Downloading 4.99MiB
07:27:18:WU02:FS01:Download complete
07:27:19:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13431 run:84 clone:4 gen:3 core:0x22 unit:0x0000000312bc7d9a5f5e32390c573eef
07:27:19:WU02:FS01:Starting
07:27:19:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:27:19:WU02:FS01:Started FahCore on PID 10048
07:27:19:WU02:FS01:Core PID:10760
07:27:19:WU02:FS01:FahCore 0x22 started
07:27:19:WU02:FS01:0x22:*********************** Log Started 2020-09-15T07:27:19Z ***********************
07:27:19:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:27:19:WU02:FS01:0x22:       Core: Core22
07:27:19:WU02:FS01:0x22:       Type: 0x22
07:27:19:WU02:FS01:0x22:    Version: 0.0.11
07:27:19:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:27:19:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:27:19:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:27:19:WU02:FS01:0x22:       Date: Jun 26 2020
07:27:19:WU02:FS01:0x22:       Time: 19:49:16
07:27:19:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:27:19:WU02:FS01:0x22:     Branch: core22-0.0.11
07:27:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:27:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:27:19:WU02:FS01:0x22:   Platform: win32 10
07:27:19:WU02:FS01:0x22:       Bits: 64
07:27:19:WU02:FS01:0x22:       Mode: Release
07:27:19:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:27:19:WU02:FS01:0x22:             <[email protected]>
07:27:19:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 10048 -checkpoint 15
07:27:19:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:27:19:WU02:FS01:0x22:             1 -gpu 1
07:27:19:WU02:FS01:0x22:************************************ libFAH ************************************
07:27:19:WU02:FS01:0x22:       Date: Jun 26 2020
07:27:19:WU02:FS01:0x22:       Time: 19:47:12
07:27:19:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:27:19:WU02:FS01:0x22:     Branch: HEAD
07:27:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:27:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:27:19:WU02:FS01:0x22:   Platform: win32 10
07:27:19:WU02:FS01:0x22:       Bits: 64
07:27:19:WU02:FS01:0x22:       Mode: Release
07:27:19:WU02:FS01:0x22:************************************ CBang *************************************
07:27:19:WU02:FS01:0x22:       Date: Jun 26 2020
07:27:19:WU02:FS01:0x22:       Time: 19:46:11
07:27:19:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:27:19:WU02:FS01:0x22:     Branch: master
07:27:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:27:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:27:19:WU02:FS01:0x22:   Platform: win32 10
07:27:19:WU02:FS01:0x22:       Bits: 64
07:27:19:WU02:FS01:0x22:       Mode: Release
07:27:19:WU02:FS01:0x22:************************************ System ************************************
07:27:19:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:27:19:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:27:19:WU02:FS01:0x22:       CPUs: 32
07:27:19:WU02:FS01:0x22:     Memory: 31.95GiB
07:27:19:WU02:FS01:0x22:Free Memory: 27.87GiB
07:27:19:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:27:19:WU02:FS01:0x22: OS Version: 6.2
07:27:19:WU02:FS01:0x22:Has Battery: false
07:27:19:WU02:FS01:0x22: On Battery: false
07:27:19:WU02:FS01:0x22: UTC Offset: -7
07:27:19:WU02:FS01:0x22:        PID: 10760
07:27:19:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:27:19:WU02:FS01:0x22:********************************************************************************
07:27:19:WU02:FS01:0x22:Project: 13431 (Run 84, Clone 4, Gen 3)
07:27:19:WU02:FS01:0x22:Unit: 0x0000000312bc7d9a5f5e32390c573eef
07:27:19:WU02:FS01:0x22:Reading tar file core.xml
07:27:19:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:27:19:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:27:19:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:27:19:WU02:FS01:0x22:Digital signatures verified
07:27:19:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:27:19:WU02:FS01:0x22:Version 0.0.11
07:27:19:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:27:19:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:27:19:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:27:19:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:27:31:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:27:31:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:27:31:WU02:FS01:0x22:Saving result file science.log
07:27:31:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:27:32:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:27:32:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13431 run:84 clone:4 gen:3 core:0x22 unit:0x0000000312bc7d9a5f5e32390c573eef
07:27:32:WU02:FS01:Uploading 2.44KiB to 18.188.125.154
07:27:32:WU02:FS01:Connecting to 18.188.125.154:8080
07:27:32:WU02:FS01:Upload complete
07:27:32:WU02:FS01:Server responded WORK_ACK (400)
07:27:32:WU02:FS01:Cleaning up
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by PantherX »

Can you please post the log file? Ensure you include the first 100 lines which will inform us of what the system configuration is and what the client settings are.

We have FahCore_22 version 0.0.12 in development that aims to provide sufficient information to debug this issue. While there's no ETA, I am guessing that is is very soon :)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

I am running the Nvidia Game Read Driver Version 451.67 on the 3 working machines. I was also running it on the machine that is failing.

I will now try upgrading the driver to 452.06 on the one failing machine, and hope I get lucky....
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

Still failing with the latest drivers. I also noticed that one of the jobs that failed had the same "PRCG" numer as a job which was running fine on the identical card in one of my other workstations....

So weird. The only difference I can see is the working system has a single E5-2650 v2 Xeon CPU, and the one that is failing has dual E5--2667 v2 Xeons.
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by PantherX »

Please note that without the logs (that contain the system information), it's very difficult to troubleshoot as we don't know what the client is doing. While we know what you're doing, that's part of the picture.

Here's what the initial section of the log looks like and you should have something similar:

Code: Select all

*********************** Log Started 2020-09-14T08:58:13Z ***********************
08:58:13:Trying to access database...
08:58:13:Successfully acquired database lock
08:58:13:Read GPUs.txt
08:58:14:Enabled folding slot 00: PAUSED cpu:4 (by user)
08:58:14:Enabled folding slot 01: PAUSED gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
08:58:14:****************************** FAHClient ******************************
08:58:14:        Version: 7.6.13
08:58:14:         Author: Joseph Coffland <[email protected]>
08:58:14:      Copyright: 2020 foldingathome.org
08:58:14:       Homepage: https://foldingathome.org/
08:58:14:           Date: Apr 27 2020
08:58:14:           Time: 21:21:01
08:58:14:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
08:58:14:         Branch: master
08:58:14:       Compiler: Visual C++ 2008
08:58:14:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
08:58:14:       Platform: win32 10
08:58:14:           Bits: 32
08:58:14:           Mode: Release
08:58:14:         Config: C:\Users\PantherX-H\AppData\Roaming\FAHClient\config.xml
08:58:14:******************************** CBang ********************************
08:58:14:           Date: Apr 24 2020
08:58:14:           Time: 17:07:55
08:58:14:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
08:58:14:         Branch: master
08:58:14:       Compiler: Visual C++ 2008
08:58:14:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
08:58:14:       Platform: win32 10
08:58:14:           Bits: 32
08:58:14:           Mode: Release
08:58:14:******************************* System ********************************
08:58:14:            CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
08:58:14:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
08:58:14:           CPUs: 8
08:58:14:         Memory: 31.94GiB
08:58:14:    Free Memory: 26.78GiB
08:58:14:        Threads: WINDOWS_THREADS
08:58:14:     OS Version: 6.2
08:58:14:    Has Battery: false
08:58:14:     On Battery: false
08:58:14:     UTC Offset: 12
08:58:14:            PID: 11352
08:58:14:            CWD: C:\Users\PantherX-H\AppData\Roaming\FAHClient
08:58:14:  Win32 Service: false
08:58:14:             OS: Windows 10 Enterprise
08:58:14:        OS Arch: AMD64
08:58:14:           GPUs: 1
08:58:14:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
08:58:14:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:11.0
08:58:14:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:452.6
08:58:14:******************************* libFAH ********************************
08:58:14:           Date: Apr 15 2020
08:58:14:           Time: 14:53:14
08:58:14:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
08:58:14:         Branch: master
08:58:14:       Compiler: Visual C++ 2008
08:58:14:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
08:58:14:       Platform: win32 10
08:58:14:           Bits: 32
08:58:14:           Mode: Release
08:58:14:***********************************************************************
08:58:15:<config>
08:58:15:  <!-- Error Handling -->
08:58:15:  <max-slot-errors v='20'/>
08:58:15:  <max-unit-errors v='20'/>
08:58:15:
08:58:15:  <!-- Folding Core -->
08:58:15:  <checkpoint v='30'/>
08:58:15:  <core-priority v='low'/>
08:58:15:
08:58:15:  <!-- Network -->
08:58:15:  <proxy v=':8080'/>
08:58:15:
08:58:15:  <!-- Slot Control -->
08:58:15:  <pause-on-start v='true'/>
08:58:15:  <power v='full'/>
08:58:15:
08:58:15:  <!-- User Information -->
08:58:15:  <passkey v='*****'/>
08:58:15:  <team v='69411'/>
08:58:15:  <user v='PantherX'/>
08:58:15:
08:58:15:  <!-- Work Unit Control -->
08:58:15:  <next-unit-percentage v='100'/>
08:58:15:
08:58:15:  <!-- Folding Slots -->
08:58:15:  <slot id='0' type='CPU'>
08:58:15:    <cpus v='4'/>
08:58:15:  </slot>
08:58:15:  <slot id='1' type='GPU'>
08:58:15:    <paused v='true'/>
08:58:15:  </slot>
08:58:15:</config>
Also, apart form the CPU difference, what else is different, hardware and software wise?
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
chabgood
Posts: 77
Joined: Wed Mar 29, 2017 7:59 pm
Hardware configuration: Core I7 7800
4 - 2080
3 - 2080 Super
32GB ram
Ubuntu OS.
All on an EK waterblock system
Location: Austin Tx.

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by chabgood »

Have you tried 1 GPU and see of that works?
Image
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

>> Please note that without the logs (that contain the system information), it's very difficult to troubleshoot

Um, I posted a log for one of the GPU slots above. (timestamp: Mon Sep 14, 2020 11:49 pm) Do you want me to grab it without filtering the slots? That will make it too large to post to the forum, but I'll post as much as will fit, starting from the beginning. If you want the whole thing, PM me an email address. (text file is 645kb)

Just to be clear, this log came from FAH Advanced control -> Log Tab, with all filters off. I clicked "clear", "refresh" and then "copy", then pasted it to a text file with notepad.

Code: Select all

*********************** Log Started 2020-09-15T07:22:51Z ***********************
07:22:51:Trying to access database...
07:22:51:Successfully acquired database lock
07:22:51:Downloading GPUs.txt from assign1.foldingathome.org:80
07:22:51:Connecting to assign1.foldingathome.org:80
07:22:51:Read GPUs.txt
07:22:51:Enabled folding slot 00: PAUSED cpu:28 (not configured)
07:22:52:Enabled folding slot 01: PAUSED gpu:0:GP106 [P106-090] (not configured)
07:22:52:Enabled folding slot 02: PAUSED gpu:1:GP106 [P106-090] (not configured)
07:22:52:Enabled folding slot 03: PAUSED gpu:2:GP106 [P106-090] (not configured)
07:22:52:****************************** FAHClient ******************************
07:22:52:        Version: 7.6.13
07:22:52:         Author: Joseph Coffland <[email protected]>
07:22:52:      Copyright: 2020 foldingathome.org
07:22:52:       Homepage: https://foldingathome.org/
07:22:52:           Date: Apr 27 2020
07:22:52:           Time: 21:21:01
07:22:52:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
07:22:52:         Branch: master
07:22:52:       Compiler: Visual C++ 2008
07:22:52:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:22:52:       Platform: win32 10
07:22:52:           Bits: 32
07:22:52:           Mode: Release
07:22:52:           Args: --open-web-control
07:22:52:******************************** CBang ********************************
07:22:52:           Date: Apr 24 2020
07:22:52:           Time: 17:07:55
07:22:52:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
07:22:52:         Branch: master
07:22:52:       Compiler: Visual C++ 2008
07:22:52:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:22:52:       Platform: win32 10
07:22:52:           Bits: 32
07:22:52:           Mode: Release
07:22:52:******************************* System ********************************
07:22:52:            CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:22:52:         CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:22:52:           CPUs: 32
07:22:52:         Memory: 31.95GiB
07:22:52:    Free Memory: 28.98GiB
07:22:52:        Threads: WINDOWS_THREADS
07:22:52:     OS Version: 6.2
07:22:52:    Has Battery: false
07:22:52:     On Battery: false
07:22:52:     UTC Offset: -7
07:22:52:            PID: 12000
07:22:52:            CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient
07:22:52:  Win32 Service: false
07:22:52:             OS: Windows 10 Enterprise
07:22:52:        OS Arch: AMD64
07:22:52:           GPUs: 3
07:22:52:          GPU 0: Bus:2 Slot:0 Func:0 NVIDIA:7 GP106 [P106-090]
07:22:52:          GPU 1: Bus:6 Slot:0 Func:0 NVIDIA:7 GP106 [P106-090]
07:22:52:          GPU 2: Bus:1 Slot:0 Func:0 NVIDIA:7 GP106 [P106-090]
07:22:52:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:11.0
07:22:52:  CUDA Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:6.1 Driver:11.0
07:22:52:  CUDA Device 2: Platform:0 Device:2 Bus:6 Slot:0 Compute:6.1 Driver:11.0
07:22:52:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:451.67
07:22:52:OpenCL Device 1: Platform:0 Device:1 Bus:2 Slot:0 Compute:1.2 Driver:451.67
07:22:52:OpenCL Device 2: Platform:0 Device:2 Bus:6 Slot:0 Compute:1.2 Driver:451.67
07:22:52:******************************* libFAH ********************************
07:22:52:           Date: Apr 15 2020
07:22:52:           Time: 14:53:14
07:22:52:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
07:22:52:         Branch: master
07:22:52:       Compiler: Visual C++ 2008
07:22:52:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:22:52:       Platform: win32 10
07:22:52:           Bits: 32
07:22:52:           Mode: Release
07:22:52:***********************************************************************
07:22:52:<config>
07:22:52:  <!-- Folding Slots -->
07:22:52:  <slot id='0' type='CPU'/>
07:22:52:  <slot id='1' type='GPU'/>
07:22:52:  <slot id='2' type='GPU'/>
07:22:52:  <slot id='3' type='GPU'/>
07:22:52:</config>
07:22:58:17:127.0.0.1:New Web session
07:23:14:Set client configured
07:23:14:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:23:15:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:23:15:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:23:15:WU00:FS00:Assigned to work server 150.136.14.110
07:23:15:WU00:FS00:Requesting new work unit for slot 00: READY cpu:28 from 150.136.14.110
07:23:15:WU00:FS00:Connecting to 150.136.14.110:8080
07:23:15:WU01:FS01:Assigned to work server 18.188.125.154
07:23:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:23:15:WU01:FS01:Connecting to 18.188.125.154:8080
07:23:15:WU02:FS02:Connecting to assign1.foldingathome.org:80
07:23:15:WU02:FS02:Assigned to work server 155.247.166.220
07:23:15:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:GP106 [P106-090] from 155.247.166.220
07:23:15:WU02:FS02:Connecting to 155.247.166.220:8080
07:23:15:WU00:FS00:Downloading 2.81MiB
07:23:15:WU03:FS03:Connecting to assign1.foldingathome.org:80
07:23:15:WU03:FS03:Assigned to work server 18.188.125.154
07:23:15:WU03:FS03:Requesting new work unit for slot 03: READY gpu:2:GP106 [P106-090] from 18.188.125.154
07:23:15:WU03:FS03:Connecting to 18.188.125.154:8080
07:23:16:WU02:FS02:Downloading 28.16MiB
07:23:16:WU01:FS01:Downloading 6.14MiB
07:23:16:WU03:FS03:Downloading 7.44MiB
07:23:19:WU00:FS00:Download complete
07:23:19:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16429 run:0 clone:599 gen:376 core:0xa7 unit:0x000001cf96880e6e5e924bf59a2c6f1a
07:23:19:WU00:FS00:Downloading core from http://cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah
07:23:19:WU00:FS00:Connecting to cores.foldingathome.org:80
07:23:19:WU00:FS00:FahCore a7: Downloading 6.71MiB
07:23:21:WU03:FS03:Download complete
07:23:22:WU03:FS03:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13434 run:165 clone:7 gen:1 core:0x22 unit:0x0000000112bc7d9a5f5e323f04735c18
07:23:22:WU03:FS03:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah
07:23:22:WU03:FS03:Connecting to cores.foldingathome.org:80
07:23:22:WU02:FS02:Download 17.53%
07:23:22:WU01:FS01:Download 66.19%
07:23:22:WU03:FS03:FahCore 22: Downloading 4.39MiB
07:23:24:WU01:FS01:Download complete
07:23:24:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3165 clone:15 gen:3 core:0x22 unit:0x0000000a12bc7d9a5f597cb670aca406
07:23:25:WU00:FS00:FahCore a7: 87.51%
07:23:26:WU03:FS03:FahCore 22: Download complete
07:23:26:WU03:FS03:Valid core signature
07:23:26:WU00:FS00:FahCore a7: Download complete
07:23:27:WU03:FS03:Unpacked 14.46MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe
07:23:27:WU03:FS03:Starting
07:23:27:WU03:FS03:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:23:27:WU03:FS03:Started FahCore on PID 9048
07:23:27:WU03:FS03:Core PID:10964
07:23:27:WU03:FS03:FahCore 0x22 started
07:23:27:WU00:FS00:Valid core signature
07:23:27:WU00:FS00:Unpacked 19.85MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe
07:23:27:WU00:FS00:Unpacked 2.64MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/libfftw3f-3.dll
07:23:27:WU00:FS00:Starting
07:23:27:WARNING:WU00:FS00:Changed SMP threads from 28 to 29 this can cause some work units to fail
07:23:27:WARNING:WU00:FS00:AS lowered CPUs from 29 to 28
07:23:27:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -np 28
07:23:27:WU00:FS00:Started FahCore on PID 11080
07:23:27:WU03:FS03:0x22:*********************** Log Started 2020-09-15T07:23:27Z ***********************
07:23:27:WU03:FS03:0x22:*************************** Core22 Folding@home Core ***************************
07:23:27:WU03:FS03:0x22:       Core: Core22
07:23:27:WU03:FS03:0x22:       Type: 0x22
07:23:27:WU03:FS03:0x22:    Version: 0.0.11
07:23:27:WU03:FS03:0x22:     Author: Joseph Coffland <[email protected]>
07:23:27:WU03:FS03:0x22:  Copyright: 2020 foldingathome.org
07:23:27:WU03:FS03:0x22:   Homepage: https://foldingathome.org/
07:23:27:WU03:FS03:0x22:       Date: Jun 26 2020
07:23:27:WU03:FS03:0x22:       Time: 19:49:16
07:23:27:WU03:FS03:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:27:WU03:FS03:0x22:     Branch: core22-0.0.11
07:23:27:WU03:FS03:0x22:   Compiler: Visual C++ 2015
07:23:27:WU03:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:27:WU03:FS03:0x22:   Platform: win32 10
07:23:27:WU03:FS03:0x22:       Bits: 64
07:23:27:WU03:FS03:0x22:       Mode: Release
07:23:27:WU03:FS03:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:27:WU03:FS03:0x22:             <[email protected]>
07:23:27:WU00:FS00:Core PID:11040
07:23:27:WU03:FS03:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 9048 -checkpoint 15
07:23:27:WU00:FS00:FahCore 0xa7 started
07:23:27:WU03:FS03:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:23:27:WU03:FS03:0x22:             0 -gpu 0
07:23:27:WU03:FS03:0x22:************************************ libFAH ************************************
07:23:27:WU03:FS03:0x22:       Date: Jun 26 2020
07:23:27:WU01:FS01:Starting
07:23:27:WU03:FS03:0x22:       Time: 19:47:12
07:23:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:23:27:WU03:FS03:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:27:WU01:FS01:Started FahCore on PID 13088
07:23:27:WU03:FS03:0x22:     Branch: HEAD
07:23:27:WU03:FS03:0x22:   Compiler: Visual C++ 2015
07:23:27:WU03:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:27:WU03:FS03:0x22:   Platform: win32 10
07:23:27:WU01:FS01:Core PID:2060
07:23:27:WU03:FS03:0x22:       Bits: 64
07:23:27:WU01:FS01:FahCore 0x22 started
07:23:27:WU03:FS03:0x22:       Mode: Release
07:23:27:WU03:FS03:0x22:************************************ CBang *************************************
07:23:27:WU03:FS03:0x22:       Date: Jun 26 2020
07:23:27:WU03:FS03:0x22:       Time: 19:46:11
07:23:27:WU03:FS03:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:27:WU03:FS03:0x22:     Branch: master
07:23:27:WU03:FS03:0x22:   Compiler: Visual C++ 2015
07:23:27:WU03:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:27:WU03:FS03:0x22:   Platform: win32 10
07:23:27:WU03:FS03:0x22:       Bits: 64
07:23:27:WU03:FS03:0x22:       Mode: Release
07:23:27:WU03:FS03:0x22:************************************ System ************************************
07:23:27:WU03:FS03:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:27:WU03:FS03:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:27:WU03:FS03:0x22:       CPUs: 32
07:23:27:WU03:FS03:0x22:     Memory: 31.95GiB
07:23:27:WU03:FS03:0x22:Free Memory: 28.67GiB
07:23:27:WU03:FS03:0x22:    Threads: WINDOWS_THREADS
07:23:27:WU03:FS03:0x22: OS Version: 6.2
07:23:27:WU03:FS03:0x22:Has Battery: false
07:23:27:WU03:FS03:0x22: On Battery: false
07:23:27:WU03:FS03:0x22: UTC Offset: -7
07:23:27:WU03:FS03:0x22:        PID: 10964
07:23:27:WU03:FS03:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:27:WU03:FS03:0x22:********************************************************************************
07:23:27:WU03:FS03:0x22:Project: 13434 (Run 165, Clone 7, Gen 1)
07:23:27:WU03:FS03:0x22:Unit: 0x0000000112bc7d9a5f5e323f04735c18
07:23:27:WU03:FS03:0x22:Reading tar file core.xml
07:23:27:WU03:FS03:0x22:Reading tar file integrator.xml.bz2
07:23:27:WU03:FS03:0x22:Reading tar file state.xml.bz2
07:23:27:WU03:FS03:0x22:Reading tar file system.xml.bz2
07:23:28:WU03:FS03:0x22:Digital signatures verified
07:23:28:WU03:FS03:0x22:Folding@home GPU Core22 Folding@home Core
07:23:28:WU03:FS03:0x22:Version 0.0.11
07:23:28:WU02:FS02:Download 24.86%
07:23:28:WU00:FS00:0xa7:*********************** Log Started 2020-09-15T07:23:27Z ***********************
07:23:28:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:23:28:WU00:FS00:0xa7:       Type: 0xa7
07:23:28:WU00:FS00:0xa7:       Core: Gromacs
07:23:28:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 11080 -checkpoint 15 -np
07:23:28:WU00:FS00:0xa7:             28
07:23:28:WU00:FS00:0xa7:************************************ CBang *************************************
07:23:28:WU00:FS00:0xa7:       Date: Nov 27 2019
07:23:28:WU00:FS00:0xa7:       Time: 03:40:09
07:23:28:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
07:23:28:WU00:FS00:0xa7:     Branch: master
07:23:28:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:23:28:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:23:28:WU00:FS00:0xa7:   Platform: win32 10
07:23:28:WU00:FS00:0xa7:       Bits: 64
07:23:28:WU00:FS00:0xa7:       Mode: Release
07:23:28:WU00:FS00:0xa7:************************************ System ************************************
07:23:28:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:28:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:28:WU00:FS00:0xa7:       CPUs: 32
07:23:28:WU00:FS00:0xa7:     Memory: 31.95GiB
07:23:28:WU00:FS00:0xa7:Free Memory: 28.70GiB
07:23:28:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
07:23:28:WU00:FS00:0xa7: OS Version: 6.2
07:23:28:WU00:FS00:0xa7:Has Battery: false
07:23:28:WU00:FS00:0xa7: On Battery: false
07:23:28:WU00:FS00:0xa7: UTC Offset: -7
07:23:28:WU00:FS00:0xa7:        PID: 11040
07:23:28:WU00:FS00:0xa7:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:28:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
07:23:28:WU00:FS00:0xa7:    Version: 0.0.19
07:23:28:WU00:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
07:23:28:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:23:28:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
07:23:28:WU00:FS00:0xa7:       Date: Nov 25 2019
07:23:28:WU00:FS00:0xa7:       Time: 17:12:41
07:23:28:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
07:23:28:WU00:FS00:0xa7:     Branch: master
07:23:28:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:23:28:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:23:28:WU00:FS00:0xa7:   Platform: win32 10
07:23:28:WU00:FS00:0xa7:       Bits: 64
07:23:28:WU00:FS00:0xa7:       Mode: Release
07:23:28:WU00:FS00:0xa7:************************************ Build *************************************
07:23:28:WU00:FS00:0xa7:       SIMD: avx_256
07:23:28:WU00:FS00:0xa7:********************************************************************************
07:23:28:WU00:FS00:0xa7:Project: 16429 (Run 0, Clone 599, Gen 376)
07:23:28:WU00:FS00:0xa7:Unit: 0x000001cf96880e6e5e924bf59a2c6f1a
07:23:28:WU00:FS00:0xa7:Reading tar file core.xml
07:23:28:WU00:FS00:0xa7:Reading tar file frame376.tpr
07:23:28:WU01:FS01:0x22:*********************** Log Started 2020-09-15T07:23:27Z ***********************
07:23:28:WU00:FS00:0xa7:Digital signatures verified
07:23:28:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:23:28:WU00:FS00:0xa7:Calling: mdrun -s frame376.tpr -o frame376.trr -x frame376.xtc -cpt 15 -nt 28
07:23:28:WU01:FS01:0x22:       Core: Core22
07:23:28:WU01:FS01:0x22:       Type: 0x22
07:23:28:WU01:FS01:0x22:    Version: 0.0.11
07:23:28:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:23:28:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:23:28:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU01:FS01:0x22:       Time: 19:49:16
07:23:28:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:28:WU01:FS01:0x22:     Branch: core22-0.0.11
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:28:WU01:FS01:0x22:             <[email protected]>
07:23:28:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 13088 -checkpoint 15
07:23:28:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:23:28:WU01:FS01:0x22:             1 -gpu 1
07:23:28:WU01:FS01:0x22:************************************ libFAH ************************************
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU01:FS01:0x22:       Time: 19:47:12
07:23:28:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:28:WU01:FS01:0x22:     Branch: HEAD
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:************************************ CBang *************************************
07:23:28:WU03:FS03:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:23:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:23:28:WU03:FS03:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:23:28:WU01:FS01:0x22:       Time: 19:46:11
07:23:28:WU03:FS03:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:23:28:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:28:WU03:FS03:0x22:  Global context and integrator variables write interval: disabled
07:23:28:WU01:FS01:0x22:     Branch: master
07:23:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:23:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:28:WU01:FS01:0x22:   Platform: win32 10
07:23:28:WU01:FS01:0x22:       Bits: 64
07:23:28:WU01:FS01:0x22:       Mode: Release
07:23:28:WU01:FS01:0x22:************************************ System ************************************
07:23:28:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:28:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:28:WU01:FS01:0x22:       CPUs: 32
07:23:28:WU01:FS01:0x22:     Memory: 31.95GiB
07:23:28:WU01:FS01:0x22:Free Memory: 28.68GiB
07:23:28:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:23:28:WU01:FS01:0x22: OS Version: 6.2
07:23:28:WU01:FS01:0x22:Has Battery: false
07:23:28:WU01:FS01:0x22: On Battery: false
07:23:28:WU01:FS01:0x22: UTC Offset: -7
07:23:28:WU01:FS01:0x22:        PID: 2060
07:23:28:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:28:WU01:FS01:0x22:********************************************************************************
07:23:28:WU01:FS01:0x22:Project: 13426 (Run 3165, Clone 15, Gen 3)
07:23:28:WU01:FS01:0x22:Unit: 0x0000000a12bc7d9a5f597cb670aca406
07:23:28:WU01:FS01:0x22:Reading tar file core.xml
07:23:28:WU01:FS01:0x22:Reading tar file integrator.xml
07:23:28:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:23:28:WU00:FS00:0xa7:Steps: first=188000000 total=500000
07:23:28:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:23:28:WU01:FS01:0x22:Digital signatures verified
07:23:28:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:23:28:WU01:FS01:0x22:Version 0.0.11
07:23:28:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:23:28:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:23:28:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:23:28:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:23:30:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
07:23:34:WU02:FS02:Download 39.28%
07:23:40:WU02:FS02:Download 60.58%
07:23:46:WU02:FS02:Download 84.55%
07:23:47:WU03:FS03:0x22:ERROR:exception: Error compiling kernel: 
07:23:47:WU03:FS03:0x22:Saving result file ..\logfile_01.txt
07:23:47:WU03:FS03:0x22:Saving result file science.log
07:23:47:WU03:FS03:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:23:47:WARNING:WU03:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:23:47:WU03:FS03:Sending unit results: id:03 state:SEND error:FAULTY project:13434 run:165 clone:7 gen:1 core:0x22 unit:0x0000000112bc7d9a5f5e323f04735c18
07:23:47:WU03:FS03:Uploading 2.45KiB to 18.188.125.154
07:23:47:WU03:FS03:Connecting to 18.188.125.154:8080
07:23:47:WU03:FS03:Upload complete
07:23:48:WU03:FS03:Server responded WORK_ACK (400)
07:23:48:WU03:FS03:Cleaning up
07:23:48:WU04:FS03:Connecting to assign1.foldingathome.org:80
07:23:48:WU04:FS03:Assigned to work server 129.213.157.105
07:23:48:WU04:FS03:Requesting new work unit for slot 03: READY gpu:2:GP106 [P106-090] from 129.213.157.105
07:23:48:WU04:FS03:Connecting to 129.213.157.105:8080
07:23:49:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:23:49:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:23:49:WU01:FS01:0x22:Saving result file science.log
07:23:49:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:23:49:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:23:49:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3165 clone:15 gen:3 core:0x22 unit:0x0000000a12bc7d9a5f597cb670aca406
07:23:49:WU01:FS01:Uploading 2.44KiB to 18.188.125.154
07:23:49:WU01:FS01:Connecting to 18.188.125.154:8080
07:23:50:WU01:FS01:Upload complete
07:23:50:WU01:FS01:Server responded WORK_ACK (400)
07:23:50:WU01:FS01:Cleaning up
07:23:50:WU03:FS01:Connecting to assign1.foldingathome.org:80
07:23:50:WU03:FS01:Assigned to work server 18.188.125.154
07:23:50:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:23:50:WU03:FS01:Connecting to 18.188.125.154:8080
07:23:50:WU02:FS02:Download complete
07:23:51:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:16918 run:160 clone:74 gen:22 core:0x22 unit:0x000000180002894c5f176170a831ae5b
07:23:51:WU02:FS02:Starting
07:23:51:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 2 -cuda-device 2 -gpu 2
07:23:51:WU02:FS02:Started FahCore on PID 8400
07:23:51:WU02:FS02:Core PID:12500
07:23:51:WU02:FS02:FahCore 0x22 started
07:23:51:WU03:FS01:Downloading 6.14MiB
07:23:51:WU02:FS02:0x22:*********************** Log Started 2020-09-15T07:23:51Z ***********************
07:23:51:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
07:23:51:WU02:FS02:0x22:       Core: Core22
07:23:51:WU02:FS02:0x22:       Type: 0x22
07:23:51:WU02:FS02:0x22:    Version: 0.0.11
07:23:51:WU02:FS02:0x22:     Author: Joseph Coffland <[email protected]>
07:23:51:WU02:FS02:0x22:  Copyright: 2020 foldingathome.org
07:23:51:WU02:FS02:0x22:   Homepage: https://foldingathome.org/
07:23:51:WU02:FS02:0x22:       Date: Jun 26 2020
07:23:51:WU02:FS02:0x22:       Time: 19:49:16
07:23:51:WU02:FS02:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:51:WU02:FS02:0x22:     Branch: core22-0.0.11
07:23:51:WU02:FS02:0x22:   Compiler: Visual C++ 2015
07:23:51:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:51:WU02:FS02:0x22:   Platform: win32 10
07:23:51:WU02:FS02:0x22:       Bits: 64
07:23:51:WU02:FS02:0x22:       Mode: Release
07:23:51:WU02:FS02:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:51:WU02:FS02:0x22:             <[email protected]>
07:23:51:WU02:FS02:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 8400 -checkpoint 15
07:23:51:WU02:FS02:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 2 -cuda-device
07:23:51:WU02:FS02:0x22:             2 -gpu 2
07:23:51:WU02:FS02:0x22:************************************ libFAH ************************************
07:23:51:WU02:FS02:0x22:       Date: Jun 26 2020
07:23:51:WU02:FS02:0x22:       Time: 19:47:12
07:23:51:WU02:FS02:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:51:WU02:FS02:0x22:     Branch: HEAD
07:23:51:WU02:FS02:0x22:   Compiler: Visual C++ 2015
07:23:51:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:51:WU02:FS02:0x22:   Platform: win32 10
07:23:51:WU02:FS02:0x22:       Bits: 64
07:23:51:WU02:FS02:0x22:       Mode: Release
07:23:51:WU02:FS02:0x22:************************************ CBang *************************************
07:23:51:WU02:FS02:0x22:       Date: Jun 26 2020
07:23:51:WU02:FS02:0x22:       Time: 19:46:11
07:23:51:WU02:FS02:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:51:WU02:FS02:0x22:     Branch: master
07:23:51:WU02:FS02:0x22:   Compiler: Visual C++ 2015
07:23:51:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:51:WU04:FS03:Downloading 10.63MiB
07:23:51:WU02:FS02:0x22:   Platform: win32 10
07:23:51:WU02:FS02:0x22:       Bits: 64
07:23:51:WU02:FS02:0x22:       Mode: Release
07:23:51:WU02:FS02:0x22:************************************ System ************************************
07:23:51:WU02:FS02:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:51:WU02:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:51:WU02:FS02:0x22:       CPUs: 32
07:23:51:WU02:FS02:0x22:     Memory: 31.95GiB
07:23:51:WU02:FS02:0x22:Free Memory: 28.59GiB
07:23:51:WU02:FS02:0x22:    Threads: WINDOWS_THREADS
07:23:51:WU02:FS02:0x22: OS Version: 6.2
07:23:51:WU02:FS02:0x22:Has Battery: false
07:23:51:WU02:FS02:0x22: On Battery: false
07:23:51:WU02:FS02:0x22: UTC Offset: -7
07:23:51:WU02:FS02:0x22:        PID: 12500
07:23:51:WU02:FS02:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:51:WU02:FS02:0x22:********************************************************************************
07:23:51:WU02:FS02:0x22:Project: 16918 (Run 160, Clone 74, Gen 22)
07:23:51:WU02:FS02:0x22:Unit: 0x000000180002894c5f176170a831ae5b
07:23:51:WU02:FS02:0x22:Reading tar file core.xml
07:23:51:WU02:FS02:0x22:Reading tar file integrator.xml
07:23:51:WU02:FS02:0x22:Reading tar file state.xml
07:23:51:WU02:FS02:0x22:Reading tar file system.xml
07:23:52:Saving configuration to config.xml
07:23:52:<config>
07:23:52:  <!-- Folding Slot Configuration -->
07:23:52:  <cause v='COVID_19'/>
07:23:52:
07:23:52:  <!-- Slot Control -->
07:23:52:  <power v='FULL'/>
07:23:52:
07:23:52:  <!-- User Information -->
07:23:52:  <passkey v='*****'/>
07:23:52:  <team v='749'/>
07:23:52:  <user v='Rwolf01'/>
07:23:52:
07:23:52:  <!-- Folding Slots -->
07:23:52:  <slot id='0' type='CPU'/>
07:23:52:  <slot id='1' type='GPU'/>
07:23:52:  <slot id='2' type='GPU'/>
07:23:52:  <slot id='3' type='GPU'/>
07:23:52:</config>
07:23:52:WU02:FS02:0x22:Digital signatures verified
07:23:52:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
07:23:52:WU02:FS02:0x22:Version 0.0.11
07:23:52:WU02:FS02:0x22:  Checkpoint write interval: 100000 steps (2%) [50 total]
07:23:52:WU02:FS02:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:23:52:WU02:FS02:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:23:52:WU02:FS02:0x22:  Global context and integrator variables write interval: disabled
07:23:57:WU04:FS03:Download 98.79%
07:23:57:WU03:FS01:Download 66.21%
07:23:57:WU04:FS03:Download complete
07:23:57:WU04:FS03:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:14904 run:225 clone:0 gen:15 core:0x22 unit:0x0000001581d59d695f4ec9e29511b37d
07:23:57:WU04:FS03:Starting
07:23:57:WU04:FS03:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 04 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:23:57:WU04:FS03:Started FahCore on PID 11000
07:23:57:WU04:FS03:Core PID:13040
07:23:57:WU04:FS03:FahCore 0x22 started
07:23:57:WU04:FS03:0x22:*********************** Log Started 2020-09-15T07:23:57Z ***********************
07:23:57:WU04:FS03:0x22:*************************** Core22 Folding@home Core ***************************
07:23:57:WU04:FS03:0x22:       Core: Core22
07:23:57:WU04:FS03:0x22:       Type: 0x22
07:23:57:WU04:FS03:0x22:    Version: 0.0.11
07:23:57:WU04:FS03:0x22:     Author: Joseph Coffland <[email protected]>
07:23:57:WU04:FS03:0x22:  Copyright: 2020 foldingathome.org
07:23:57:WU04:FS03:0x22:   Homepage: https://foldingathome.org/
07:23:57:WU04:FS03:0x22:       Date: Jun 26 2020
07:23:57:WU04:FS03:0x22:       Time: 19:49:16
07:23:57:WU04:FS03:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:57:WU04:FS03:0x22:     Branch: core22-0.0.11
07:23:57:WU04:FS03:0x22:   Compiler: Visual C++ 2015
07:23:57:WU04:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:57:WU04:FS03:0x22:   Platform: win32 10
07:23:57:WU04:FS03:0x22:       Bits: 64
07:23:57:WU04:FS03:0x22:       Mode: Release
07:23:57:WU04:FS03:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:57:WU04:FS03:0x22:             <[email protected]>
07:23:57:WU04:FS03:0x22:       Args: -dir 04 -suffix 01 -version 706 -lifeline 11000 -checkpoint 15
07:23:57:WU04:FS03:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:23:57:WU04:FS03:0x22:             0 -gpu 0
07:23:57:WU04:FS03:0x22:************************************ libFAH ************************************
07:23:57:WU04:FS03:0x22:       Date: Jun 26 2020
07:23:57:WU04:FS03:0x22:       Time: 19:47:12
07:23:57:WU04:FS03:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:57:WU04:FS03:0x22:     Branch: HEAD
07:23:57:WU04:FS03:0x22:   Compiler: Visual C++ 2015
07:23:57:WU04:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:57:WU04:FS03:0x22:   Platform: win32 10
07:23:57:WU04:FS03:0x22:       Bits: 64
07:23:57:WU04:FS03:0x22:       Mode: Release
07:23:57:WU04:FS03:0x22:************************************ CBang *************************************
07:23:57:WU04:FS03:0x22:       Date: Jun 26 2020
07:23:57:WU04:FS03:0x22:       Time: 19:46:11
07:23:57:WU04:FS03:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:57:WU04:FS03:0x22:     Branch: master
07:23:57:WU04:FS03:0x22:   Compiler: Visual C++ 2015
07:23:57:WU04:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:57:WU04:FS03:0x22:   Platform: win32 10
07:23:57:WU04:FS03:0x22:       Bits: 64
07:23:57:WU04:FS03:0x22:       Mode: Release
07:23:57:WU04:FS03:0x22:************************************ System ************************************
07:23:57:WU04:FS03:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:57:WU04:FS03:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:57:WU04:FS03:0x22:       CPUs: 32
07:23:57:WU04:FS03:0x22:     Memory: 31.95GiB
07:23:57:WU04:FS03:0x22:Free Memory: 28.52GiB
07:23:57:WU04:FS03:0x22:    Threads: WINDOWS_THREADS
07:23:57:WU04:FS03:0x22: OS Version: 6.2
07:23:57:WU04:FS03:0x22:Has Battery: false
07:23:57:WU04:FS03:0x22: On Battery: false
07:23:57:WU04:FS03:0x22: UTC Offset: -7
07:23:57:WU04:FS03:0x22:        PID: 13040
07:23:57:WU04:FS03:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:23:57:WU04:FS03:0x22:********************************************************************************
07:23:57:WU04:FS03:0x22:Project: 14904 (Run 225, Clone 0, Gen 15)
07:23:57:WU04:FS03:0x22:Unit: 0x0000001581d59d695f4ec9e29511b37d
07:23:57:WU04:FS03:0x22:Reading tar file core.xml
07:23:57:WU04:FS03:0x22:Reading tar file integrator.xml
07:23:57:WU04:FS03:0x22:Reading tar file state.xml
07:23:58:WU04:FS03:0x22:Reading tar file system.xml
07:23:59:WU03:FS01:Download complete
07:23:59:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13426 run:3583 clone:6 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbf48de1efa
07:23:59:WU03:FS01:Starting
07:23:59:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:23:59:WU03:FS01:Started FahCore on PID 11052
07:23:59:WU03:FS01:Core PID:11056
07:23:59:WU03:FS01:FahCore 0x22 started
07:23:59:WU04:FS03:0x22:Digital signatures verified
07:23:59:WU04:FS03:0x22:Folding@home GPU Core22 Folding@home Core
07:23:59:WU04:FS03:0x22:Version 0.0.11
07:23:59:WU03:FS01:0x22:*********************** Log Started 2020-09-15T07:23:59Z ***********************
07:23:59:WU03:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:23:59:WU03:FS01:0x22:       Core: Core22
07:23:59:WU03:FS01:0x22:       Type: 0x22
07:23:59:WU03:FS01:0x22:    Version: 0.0.11
07:23:59:WU03:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:23:59:WU03:FS01:0x22:  Copyright: 2020 foldingathome.org
07:23:59:WU03:FS01:0x22:   Homepage: https://foldingathome.org/
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:49:16
07:23:59:WU03:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:23:59:WU03:FS01:0x22:     Branch: core22-0.0.11
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:23:59:WU03:FS01:0x22:             <[email protected]>
07:23:59:WU03:FS01:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 11052 -checkpoint 15
07:23:59:WU03:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:23:59:WU03:FS01:0x22:             1 -gpu 1
07:23:59:WU03:FS01:0x22:************************************ libFAH ************************************
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:47:12
07:23:59:WU03:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:23:59:WU03:FS01:0x22:     Branch: HEAD
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:************************************ CBang *************************************
07:23:59:WU03:FS01:0x22:       Date: Jun 26 2020
07:23:59:WU03:FS01:0x22:       Time: 19:46:11
07:23:59:WU03:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:23:59:WU03:FS01:0x22:     Branch: master
07:23:59:WU03:FS01:0x22:   Compiler: Visual C++ 2015
07:23:59:WU03:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:23:59:WU03:FS01:0x22:   Platform: win32 10
07:23:59:WU03:FS01:0x22:       Bits: 64
07:23:59:WU03:FS01:0x22:       Mode: Release
07:23:59:WU03:FS01:0x22:************************************ System ************************************
07:23:59:WU03:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:23:59:WU03:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:23:59:WU03:FS01:0x22:       CPUs: 32
07:23:59:WU03:FS01:0x22:     Memory: 31.95GiB
07:23:59:WU03:FS01:0x22:Free Memory: 28.37GiB
07:23:59:WU03:FS01:0x22:    Threads: WINDOWS_THREADS
07:23:59:WU03:FS01:0x22: OS Version: 6.2
07:23:59:WU03:FS01:0x22:Has Battery: false
07:23:59:WU04:FS03:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
07:23:59:WU03:FS01:0x22: On Battery: false
07:23:59:WU04:FS03:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
07:24:00:WU03:FS01:0x22: UTC Offset: -7
07:24:00:WU04:FS03:0x22:  XTC frame write interval: 50000 steps (2.5%) [40 total]
07:24:00:WU03:FS01:0x22:        PID: 11056
07:24:00:WU04:FS03:0x22:  Global context and integrator variables write interval: disabled
07:24:00:WU03:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:00:WU03:FS01:0x22:********************************************************************************
07:24:00:WU03:FS01:0x22:Project: 13426 (Run 3583, Clone 6, Gen 2)
07:24:00:WU03:FS01:0x22:Unit: 0x0000000212bc7d9a5f597cbf48de1efa
07:24:00:WU03:FS01:0x22:Reading tar file core.xml
07:24:00:WU03:FS01:0x22:Reading tar file integrator.xml
07:24:00:WU03:FS01:0x22:Reading tar file state.xml.bz2
07:24:00:WU03:FS01:0x22:Reading tar file system.xml.bz2
07:24:00:WU03:FS01:0x22:Digital signatures verified
07:24:00:WU03:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:24:00:WU03:FS01:0x22:Version 0.0.11
07:24:00:WU03:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:24:00:WU03:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:24:00:WU03:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:24:00:WU03:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:24:00:WU02:FS02:0x22:ERROR:exception: Error compiling kernel: 
07:24:00:WU02:FS02:0x22:Saving result file ..\logfile_01.txt
07:24:00:WU02:FS02:0x22:Saving result file science.log
07:24:00:WU02:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:00:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:00:WU02:FS02:Sending unit results: id:02 state:SEND error:FAULTY project:16918 run:160 clone:74 gen:22 core:0x22 unit:0x000000180002894c5f176170a831ae5b
07:24:00:WU02:FS02:Uploading 10.00KiB to 155.247.166.220
07:24:00:WU02:FS02:Connecting to 155.247.166.220:8080
07:24:00:WU02:FS02:Upload complete
07:24:00:WU01:FS02:Connecting to assign1.foldingathome.org:80
07:24:00:WU02:FS02:Server responded WORK_ACK (400)
07:24:00:WU02:FS02:Cleaning up
07:24:01:WU01:FS02:Assigned to work server 66.170.111.50
07:24:01:WU01:FS02:Requesting new work unit for slot 02: READY gpu:1:GP106 [P106-090] from 66.170.111.50
07:24:01:WU01:FS02:Connecting to 66.170.111.50:8080
07:24:01:WU01:FS02:Downloading 11.16MiB
07:24:03:WU01:FS02:Download complete
07:24:03:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14485 run:0 clone:1407 gen:30 core:0x22 unit:0x0000002742aa6f325f45deaa3bd817dc
07:24:03:WU01:FS02:Starting
07:24:03:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 2 -cuda-device 2 -gpu 2
07:24:03:WU01:FS02:Started FahCore on PID 4556
07:24:03:WU01:FS02:Core PID:2112
07:24:03:WU01:FS02:FahCore 0x22 started
07:24:04:WU01:FS02:0x22:*********************** Log Started 2020-09-15T07:24:03Z ***********************
07:24:04:WU01:FS02:0x22:*************************** Core22 Folding@home Core ***************************
07:24:04:WU01:FS02:0x22:       Core: Core22
07:24:04:WU01:FS02:0x22:       Type: 0x22
07:24:04:WU01:FS02:0x22:    Version: 0.0.11
07:24:04:WU01:FS02:0x22:     Author: Joseph Coffland <[email protected]>
07:24:04:WU01:FS02:0x22:  Copyright: 2020 foldingathome.org
07:24:04:WU01:FS02:0x22:   Homepage: https://foldingathome.org/
07:24:04:WU01:FS02:0x22:       Date: Jun 26 2020
07:24:04:WU01:FS02:0x22:       Time: 19:49:16
07:24:04:WU01:FS02:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:24:04:WU01:FS02:0x22:     Branch: core22-0.0.11
07:24:04:WU01:FS02:0x22:   Compiler: Visual C++ 2015
07:24:04:WU01:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:04:WU01:FS02:0x22:   Platform: win32 10
07:24:04:WU01:FS02:0x22:       Bits: 64
07:24:04:WU01:FS02:0x22:       Mode: Release
07:24:04:WU01:FS02:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:24:04:WU01:FS02:0x22:             <[email protected]>
07:24:04:WU01:FS02:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4556 -checkpoint 15
07:24:04:WU01:FS02:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 2 -cuda-device
07:24:04:WU01:FS02:0x22:             2 -gpu 2
07:24:04:WU01:FS02:0x22:************************************ libFAH ************************************
07:24:04:WU01:FS02:0x22:       Date: Jun 26 2020
07:24:04:WU01:FS02:0x22:       Time: 19:47:12
07:24:04:WU01:FS02:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:24:04:WU01:FS02:0x22:     Branch: HEAD
07:24:04:WU01:FS02:0x22:   Compiler: Visual C++ 2015
07:24:04:WU01:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:04:WU01:FS02:0x22:   Platform: win32 10
07:24:04:WU01:FS02:0x22:       Bits: 64
07:24:04:WU01:FS02:0x22:       Mode: Release
07:24:04:WU01:FS02:0x22:************************************ CBang *************************************
07:24:04:WU01:FS02:0x22:       Date: Jun 26 2020
07:24:04:WU01:FS02:0x22:       Time: 19:46:11
07:24:04:WU01:FS02:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:24:04:WU01:FS02:0x22:     Branch: master
07:24:04:WU01:FS02:0x22:   Compiler: Visual C++ 2015
07:24:04:WU01:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:04:WU01:FS02:0x22:   Platform: win32 10
07:24:04:WU01:FS02:0x22:       Bits: 64
07:24:04:WU01:FS02:0x22:       Mode: Release
07:24:04:WU01:FS02:0x22:************************************ System ************************************
07:24:04:WU01:FS02:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:24:04:WU01:FS02:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:24:04:WU01:FS02:0x22:       CPUs: 32
07:24:04:WU01:FS02:0x22:     Memory: 31.95GiB
07:24:04:WU01:FS02:0x22:Free Memory: 28.09GiB
07:24:04:WU01:FS02:0x22:    Threads: WINDOWS_THREADS
07:24:04:WU01:FS02:0x22: OS Version: 6.2
07:24:04:WU01:FS02:0x22:Has Battery: false
07:24:04:WU01:FS02:0x22: On Battery: false
07:24:04:WU01:FS02:0x22: UTC Offset: -7
07:24:04:WU01:FS02:0x22:        PID: 2112
07:24:04:WU01:FS02:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:04:WU01:FS02:0x22:********************************************************************************
07:24:04:WU01:FS02:0x22:Project: 14485 (Run 0, Clone 1407, Gen 30)
07:24:04:WU01:FS02:0x22:Unit: 0x0000002742aa6f325f45deaa3bd817dc
07:24:04:WU01:FS02:0x22:Reading tar file core.xml
07:24:04:WU01:FS02:0x22:Reading tar file integrator.xml.bz2
07:24:04:WU01:FS02:0x22:Reading tar file state.xml.bz2
07:24:04:WU01:FS02:0x22:Reading tar file system.xml.bz2
07:24:04:WU01:FS02:0x22:Digital signatures verified
07:24:04:WU01:FS02:0x22:Folding@home GPU Core22 Folding@home Core
07:24:04:WU01:FS02:0x22:Version 0.0.11
07:24:04:WU01:FS02:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
07:24:04:WU01:FS02:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
07:24:04:WU01:FS02:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
07:24:04:WU01:FS02:0x22:  Global context and integrator variables write interval: disabled
07:24:16:WU04:FS03:0x22:ERROR:exception: Error compiling kernel: 
07:24:16:WU04:FS03:0x22:Saving result file ..\logfile_01.txt
07:24:16:WU04:FS03:0x22:Saving result file science.log
07:24:16:WU04:FS03:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:16:WARNING:WU04:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:16:WU04:FS03:Sending unit results: id:04 state:SEND error:FAULTY project:14904 run:225 clone:0 gen:15 core:0x22 unit:0x0000001581d59d695f4ec9e29511b37d
07:24:16:WU04:FS03:Uploading 2.42KiB to 129.213.157.105
07:24:16:WU04:FS03:Connecting to 129.213.157.105:8080
07:24:16:WU04:FS03:Upload complete
07:24:16:WU04:FS03:Server responded WORK_ACK (400)
07:24:16:WU04:FS03:Cleaning up
07:24:16:WU02:FS03:Connecting to assign1.foldingathome.org:80
07:24:17:WU02:FS03:Assigned to work server 18.188.125.154
07:24:17:WU02:FS03:Requesting new work unit for slot 03: READY gpu:2:GP106 [P106-090] from 18.188.125.154
07:24:17:WU02:FS03:Connecting to 18.188.125.154:8080
07:24:17:WU02:FS03:Downloading 4.98MiB
07:24:20:WU03:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:24:20:WU03:FS01:0x22:Saving result file ..\logfile_01.txt
07:24:20:WU03:FS01:0x22:Saving result file science.log
07:24:20:WU03:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:20:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:20:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:13426 run:3583 clone:6 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbf48de1efa
07:24:20:WU03:FS01:Uploading 2.44KiB to 18.188.125.154
07:24:21:WU03:FS01:Connecting to 18.188.125.154:8080
07:24:21:WU02:FS03:Download complete
07:24:21:WU02:FS03:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13433 run:223 clone:4 gen:2 core:0x22 unit:0x0000000212bc7d9a5f5e396a4a17dd37
07:24:21:WU02:FS03:Starting
07:24:21:WU02:FS03:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:24:21:WU02:FS03:Started FahCore on PID 9236
07:24:21:WU02:FS03:Core PID:9348
07:24:21:WU02:FS03:FahCore 0x22 started
07:24:21:WU03:FS01:Upload complete
07:24:21:WU04:FS01:Connecting to assign1.foldingathome.org:80
07:24:21:WU03:FS01:Server responded WORK_ACK (400)
07:24:21:WU03:FS01:Cleaning up
07:24:21:WU02:FS03:0x22:*********************** Log Started 2020-09-15T07:24:21Z ***********************
07:24:21:WU02:FS03:0x22:*************************** Core22 Folding@home Core ***************************
07:24:21:WU02:FS03:0x22:       Core: Core22
07:24:21:WU02:FS03:0x22:       Type: 0x22
07:24:21:WU02:FS03:0x22:    Version: 0.0.11
07:24:21:WU02:FS03:0x22:     Author: Joseph Coffland <[email protected]>
07:24:21:WU02:FS03:0x22:  Copyright: 2020 foldingathome.org
07:24:21:WU02:FS03:0x22:   Homepage: https://foldingathome.org/
07:24:21:WU02:FS03:0x22:       Date: Jun 26 2020
07:24:21:WU02:FS03:0x22:       Time: 19:49:16
07:24:21:WU02:FS03:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:24:21:WU02:FS03:0x22:     Branch: core22-0.0.11
07:24:21:WU02:FS03:0x22:   Compiler: Visual C++ 2015
07:24:21:WU02:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:21:WU02:FS03:0x22:   Platform: win32 10
07:24:21:WU02:FS03:0x22:       Bits: 64
07:24:21:WU02:FS03:0x22:       Mode: Release
07:24:21:WU02:FS03:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:24:21:WU02:FS03:0x22:             <[email protected]>
07:24:21:WU02:FS03:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 9236 -checkpoint 15
07:24:21:WU02:FS03:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:24:21:WU02:FS03:0x22:             0 -gpu 0
07:24:21:WU02:FS03:0x22:************************************ libFAH ************************************
07:24:21:WU02:FS03:0x22:       Date: Jun 26 2020
07:24:21:WU02:FS03:0x22:       Time: 19:47:12
07:24:21:WU02:FS03:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:24:21:WU02:FS03:0x22:     Branch: HEAD
07:24:21:WU02:FS03:0x22:   Compiler: Visual C++ 2015
07:24:21:WU02:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:21:WU02:FS03:0x22:   Platform: win32 10
07:24:21:WU02:FS03:0x22:       Bits: 64
07:24:21:WU02:FS03:0x22:       Mode: Release
07:24:21:WU02:FS03:0x22:************************************ CBang *************************************
07:24:21:WU02:FS03:0x22:       Date: Jun 26 2020
07:24:21:WU02:FS03:0x22:       Time: 19:46:11
07:24:21:WU02:FS03:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:24:21:WU02:FS03:0x22:     Branch: master
07:24:21:WU02:FS03:0x22:   Compiler: Visual C++ 2015
07:24:21:WU02:FS03:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:21:WU02:FS03:0x22:   Platform: win32 10
07:24:21:WU02:FS03:0x22:       Bits: 64
07:24:21:WU02:FS03:0x22:       Mode: Release
07:24:21:WU02:FS03:0x22:************************************ System ************************************
07:24:21:WU02:FS03:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:24:21:WU04:FS01:Assigned to work server 18.188.125.154
07:24:21:WU02:FS03:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:24:21:WU04:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:24:21:WU02:FS03:0x22:       CPUs: 32
07:24:21:WU04:FS01:Connecting to 18.188.125.154:8080
07:24:21:WU02:FS03:0x22:     Memory: 31.95GiB
07:24:21:WU02:FS03:0x22:Free Memory: 28.40GiB
07:24:21:WU02:FS03:0x22:    Threads: WINDOWS_THREADS
07:24:21:WU02:FS03:0x22: OS Version: 6.2
07:24:21:WU02:FS03:0x22:Has Battery: false
07:24:21:WU02:FS03:0x22: On Battery: false
07:24:21:WU02:FS03:0x22: UTC Offset: -7
07:24:21:WU02:FS03:0x22:        PID: 9348
07:24:21:WU02:FS03:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:21:WU02:FS03:0x22:********************************************************************************
07:24:21:WU02:FS03:0x22:Project: 13433 (Run 223, Clone 4, Gen 2)
07:24:21:WU02:FS03:0x22:Unit: 0x0000000212bc7d9a5f5e396a4a17dd37
07:24:21:WU02:FS03:0x22:Reading tar file core.xml
07:24:21:WU02:FS03:0x22:Reading tar file integrator.xml.bz2
07:24:21:WU02:FS03:0x22:Reading tar file state.xml.bz2
07:24:21:WU02:FS03:0x22:Reading tar file system.xml.bz2
07:24:21:WU02:FS03:0x22:Digital signatures verified
07:24:21:WU02:FS03:0x22:Folding@home GPU Core22 Folding@home Core
07:24:21:WU02:FS03:0x22:Version 0.0.11
07:24:21:WU02:FS03:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:24:21:WU02:FS03:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:24:21:WU02:FS03:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:24:21:WU02:FS03:0x22:  Global context and integrator variables write interval: disabled
07:24:22:WU04:FS01:Downloading 6.14MiB
07:24:26:WU01:FS02:0x22:ERROR:exception: Error compiling kernel: 
07:24:26:WU01:FS02:0x22:Saving result file ..\logfile_01.txt
07:24:26:WU01:FS02:0x22:Saving result file science.log
07:24:26:WU01:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:24:26:WU04:FS01:Download complete
07:24:26:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:13426 run:3173 clone:9 gen:2 core:0x22 unit:0x0000000312bc7d9a5f597cb6b6949e31
07:24:26:WU04:FS01:Starting
07:24:26:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 04 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
07:24:26:WU04:FS01:Started FahCore on PID 12932
07:24:26:WU04:FS01:Core PID:7188
07:24:26:WU04:FS01:FahCore 0x22 started
07:24:26:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:24:27:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:14485 run:0 clone:1407 gen:30 core:0x22 unit:0x0000002742aa6f325f45deaa3bd817dc
07:24:27:WU01:FS02:Uploading 10.50KiB to 66.170.111.50
07:24:27:WU01:FS02:Connecting to 66.170.111.50:8080
07:24:27:WU01:FS02:Upload complete
07:24:27:WU04:FS01:0x22:*********************** Log Started 2020-09-15T07:24:26Z ***********************
07:24:27:WU04:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:24:27:WU04:FS01:0x22:       Core: Core22
07:24:27:WU01:FS02:Server responded WORK_ACK (400)
07:24:27:WU04:FS01:0x22:       Type: 0x22
07:24:27:WU04:FS01:0x22:    Version: 0.0.11
07:24:27:WU01:FS02:Cleaning up
07:24:27:WU04:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:24:27:WU04:FS01:0x22:  Copyright: 2020 foldingathome.org
07:24:27:WU04:FS01:0x22:   Homepage: https://foldingathome.org/
07:24:27:WU03:FS02:Connecting to assign1.foldingathome.org:80
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:49:16
07:24:27:WU04:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:24:27:WU04:FS01:0x22:     Branch: core22-0.0.11
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:24:27:WU04:FS01:0x22:             <[email protected]>
07:24:27:WU04:FS01:0x22:       Args: -dir 04 -suffix 01 -version 706 -lifeline 12932 -checkpoint 15
07:24:27:WU04:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
07:24:27:WU04:FS01:0x22:             1 -gpu 1
07:24:27:WU04:FS01:0x22:************************************ libFAH ************************************
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:47:12
07:24:27:WU04:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:24:27:WU04:FS01:0x22:     Branch: HEAD
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:************************************ CBang *************************************
07:24:27:WU04:FS01:0x22:       Date: Jun 26 2020
07:24:27:WU04:FS01:0x22:       Time: 19:46:11
07:24:27:WU04:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:24:27:WU04:FS01:0x22:     Branch: master
07:24:27:WU04:FS01:0x22:   Compiler: Visual C++ 2015
07:24:27:WU04:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:24:27:WU04:FS01:0x22:   Platform: win32 10
07:24:27:WU04:FS01:0x22:       Bits: 64
07:24:27:WU04:FS01:0x22:       Mode: Release
07:24:27:WU04:FS01:0x22:************************************ System ************************************
07:24:27:WU04:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:24:27:WU04:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:24:27:WU04:FS01:0x22:       CPUs: 32
07:24:27:WU04:FS01:0x22:     Memory: 31.95GiB
07:24:27:WU04:FS01:0x22:Free Memory: 28.52GiB
07:24:27:WU04:FS01:0x22:    Threads: WINDOWS_THREADS
07:24:27:WU04:FS01:0x22: OS Version: 6.2
07:24:27:WU04:FS01:0x22:Has Battery: false
07:24:27:WU04:FS01:0x22: On Battery: false
07:24:27:WU04:FS01:0x22: UTC Offset: -7
07:24:27:WU04:FS01:0x22:        PID: 7188
07:24:27:WU04:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:24:27:WU04:FS01:0x22:********************************************************************************
07:24:27:WU04:FS01:0x22:Project: 13426 (Run 3173, Clone 9, Gen 2)
07:24:27:WU04:FS01:0x22:Unit: 0x0000000312bc7d9a5f597cb6b6949e31
07:24:27:WU04:FS01:0x22:Reading tar file core.xml
07:24:27:WU04:FS01:0x22:Reading tar file integrator.xml
07:24:27:WU04:FS01:0x22:Reading tar file state.xml.bz2
07:24:27:WU04:FS01:0x22:Reading tar file system.xml.bz2
07:24:27:WU04:FS01:0x22:Digital signatures verified
07:24:27:WU04:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:24:27:WU04:FS01:0x22:Version 0.0.11
07:24:27:WU04:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:24:27:WU04:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:24:27:WU04:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:24:27:WU04:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:24:27:WU03:FS02:Assigned to work server 18.188.125.154
07:24:27:WU03:FS02:Requesting new work unit for slot 02: READY gpu:1:GP106 [P106-090] from 18.188.125.154
07:24:27:WU03:FS02:Connecting to 18.188.125.154:8080
07:24:28:WU03:FS02:Downloading 7.59MiB
07:24:32:WU03:FS02:Download complete
07:24:32:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13434 run:8 clone:7 gen:2 core:0x22 unit:0x0000000312bc7d9a5f5e323b335b7ee7
07:24:32:WU03:FS02:Starting
07:24:32:WU03:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 12000 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 2 -cuda-device 2 -gpu 2
07:24:32:WU03:FS02:Started FahCore on PID 4400
07:24:32:WU03:FS02:Core PID:10468
07:24:32:WU03:FS02:FahCore 0x22 started
// truncated to fit.
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by PantherX »

Thanks, that initial section is what is important to us.

You're using a mining GPU (https://www.techpowerup.com/gpu-specs/p106-090.c2999) which is supported for folding and donors have used it (viewtopic.php?f=38&t=32223).

Can you run GPU-Z to ensure that you can properly detect the 3 GPUs and that it shows CUDA and OpenCL support?

Also, can you describe how your GPU is connected to the motherboard (using risers, etc.) and what kind of PSU you have.

For troubleshooting, I would enter <pause-on-start v='true'/> under the Expert option in FAHControl (AKA Advanced Control). Restart the client and then simply test with one GPU to ensure that the log is readable. Once the issue on 1 GPU is solved, try with 2 GPUs and repeat. I would suggest that you get the latest drivers from Nvidia and do a clean installation. In one case, I read that the drivers from the GPU vendor worked so give that a shot too.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

I have not tried removing GPUs since it started failing. It was working with multiple GPUs before and my 3 other workstations all have multiple GPUs of the same or similar type.

But it could be a HW issue. It's easy enough to try. (If nothing else, it will make the logs smaller :-)

I powered down and pulled 2 GPUs. So there is only one GPU left, but it is still a dual CPU workstation.
When I rebooted the FAH "Advanced Control" client was acting badly so I uninstalled FAH, including all data, rebooted and then reinstalled FAH.

The CPU slot took a few retries to get going, but now has a job downloaded and is chewing away at it. The sole GPU slot will download a job and appear to start working ("Running" in green) but then something happens and it starts downloading a new job before even finishing 1% of the old one. This repeats until it eventually flags the GPU slot as "Failed" in Red.

Here is the first part of the unfiltered log from that experiment. (still too large to post in ful)

Thanks, in advance, for any assistance you can give me in figuring this out.

Code: Select all

*********************** Log Started 2020-09-16T07:52:29Z ***********************
07:52:29:Trying to access database...
07:52:29:Successfully acquired database lock
07:52:29:Downloading GPUs.txt from assign1.foldingathome.org:80
07:52:29:Connecting to assign1.foldingathome.org:80
07:52:29:Read GPUs.txt
07:52:29:Enabled folding slot 00: PAUSED cpu:30 (not configured)
07:52:30:Enabled folding slot 01: PAUSED gpu:0:GP106 [P106-090] (not configured)
07:52:30:****************************** FAHClient ******************************
07:52:30:        Version: 7.6.13
07:52:30:         Author: Joseph Coffland <[email protected]>
07:52:30:      Copyright: 2020 foldingathome.org
07:52:30:       Homepage: https://foldingathome.org/
07:52:30:           Date: Apr 27 2020
07:52:30:           Time: 21:21:01
07:52:30:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
07:52:30:         Branch: master
07:52:30:       Compiler: Visual C++ 2008
07:52:30:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:52:30:       Platform: win32 10
07:52:30:           Bits: 32
07:52:30:           Mode: Release
07:52:30:           Args: --open-web-control
07:52:30:******************************** CBang ********************************
07:52:30:           Date: Apr 24 2020
07:52:30:           Time: 17:07:55
07:52:30:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
07:52:30:         Branch: master
07:52:30:       Compiler: Visual C++ 2008
07:52:30:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:52:30:       Platform: win32 10
07:52:30:           Bits: 32
07:52:30:           Mode: Release
07:52:30:******************************* System ********************************
07:52:30:            CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:52:30:         CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:52:30:           CPUs: 32
07:52:30:         Memory: 31.95GiB
07:52:30:    Free Memory: 29.00GiB
07:52:30:        Threads: WINDOWS_THREADS
07:52:30:     OS Version: 6.2
07:52:30:    Has Battery: false
07:52:30:     On Battery: false
07:52:30:     UTC Offset: -7
07:52:30:            PID: 12744
07:52:30:            CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient
07:52:30:  Win32 Service: false
07:52:30:             OS: Windows 10 Enterprise
07:52:30:        OS Arch: AMD64
07:52:30:           GPUs: 1
07:52:30:          GPU 0: Bus:6 Slot:0 Func:0 NVIDIA:7 GP106 [P106-090]
07:52:30:  CUDA Device 0: Platform:0 Device:0 Bus:6 Slot:0 Compute:6.1 Driver:11.0
07:52:30:OpenCL Device 0: Platform:0 Device:0 Bus:6 Slot:0 Compute:1.2 Driver:452.6
07:52:30:******************************* libFAH ********************************
07:52:30:           Date: Apr 15 2020
07:52:30:           Time: 14:53:14
07:52:30:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
07:52:30:         Branch: master
07:52:30:       Compiler: Visual C++ 2008
07:52:30:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:52:30:       Platform: win32 10
07:52:30:           Bits: 32
07:52:30:           Mode: Release
07:52:30:***********************************************************************
07:52:30:<config>
07:52:30:  <!-- Folding Slots -->
07:52:30:  <slot id='0' type='CPU'/>
07:52:30:  <slot id='1' type='GPU'/>
07:52:30:</config>
07:52:33:3:127.0.0.1:New Web session
07:52:50:Set client configured
07:52:50:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:52:50:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:52:51:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:52:51:WU00:FS00:Assigned to work server 143.89.243.111
07:52:51:WU00:FS00:Requesting new work unit for slot 00: READY cpu:30 from 143.89.243.111
07:52:51:WU00:FS00:Connecting to 143.89.243.111:8080
07:52:51:WU01:FS01:Assigned to work server 18.188.125.154
07:52:51:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:52:51:WU01:FS01:Connecting to 18.188.125.154:8080
07:52:51:ERROR:WU00:FS00:Exception: Server did not assign work unit
07:52:51:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:52:51:WU01:FS01:Downloading 6.15MiB
07:52:52:WU00:FS00:Assigned to work server 143.89.243.111
07:52:52:WU00:FS00:Requesting new work unit for slot 00: READY cpu:30 from 143.89.243.111
07:52:52:WU00:FS00:Connecting to 143.89.243.111:8080
07:52:52:ERROR:WU00:FS00:Exception: Server did not assign work unit
07:52:55:WU01:FS01:Download complete
07:52:55:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3503 clone:1 gen:3 core:0x22 unit:0x0000000712bc7d9a5f597cc1b9170394
07:52:55:WU01:FS01:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah
07:52:55:WU01:FS01:Connecting to cores.foldingathome.org:80
07:52:55:WU01:FS01:FahCore 22: Downloading 4.39MiB
07:52:57:WU01:FS01:FahCore 22: Download complete
07:52:57:WU01:FS01:Valid core signature
07:52:57:WU01:FS01:Unpacked 14.46MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe
07:52:57:WU01:FS01:Starting
07:52:57:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:52:57:WU01:FS01:Started FahCore on PID 1820
07:52:58:WU01:FS01:Core PID:2556
07:52:58:WU01:FS01:FahCore 0x22 started
07:52:58:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:52:58Z ***********************
07:52:58:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:52:58:WU01:FS01:0x22:       Core: Core22
07:52:58:WU01:FS01:0x22:       Type: 0x22
07:52:58:WU01:FS01:0x22:    Version: 0.0.11
07:52:58:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:52:58:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:52:58:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:52:58:WU01:FS01:0x22:       Date: Jun 26 2020
07:52:58:WU01:FS01:0x22:       Time: 19:49:16
07:52:58:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:52:58:WU01:FS01:0x22:     Branch: core22-0.0.11
07:52:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:52:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:52:58:WU01:FS01:0x22:   Platform: win32 10
07:52:58:WU01:FS01:0x22:       Bits: 64
07:52:58:WU01:FS01:0x22:       Mode: Release
07:52:58:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:52:58:WU01:FS01:0x22:             <[email protected]>
07:52:58:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 1820 -checkpoint 15
07:52:58:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:52:58:WU01:FS01:0x22:             0 -gpu 0
07:52:58:WU01:FS01:0x22:************************************ libFAH ************************************
07:52:58:WU01:FS01:0x22:       Date: Jun 26 2020
07:52:58:WU01:FS01:0x22:       Time: 19:47:12
07:52:58:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:52:58:WU01:FS01:0x22:     Branch: HEAD
07:52:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:52:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:52:58:WU01:FS01:0x22:   Platform: win32 10
07:52:58:WU01:FS01:0x22:       Bits: 64
07:52:58:WU01:FS01:0x22:       Mode: Release
07:52:58:WU01:FS01:0x22:************************************ CBang *************************************
07:52:58:WU01:FS01:0x22:       Date: Jun 26 2020
07:52:58:WU01:FS01:0x22:       Time: 19:46:11
07:52:58:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:52:58:WU01:FS01:0x22:     Branch: master
07:52:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:52:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:52:58:WU01:FS01:0x22:   Platform: win32 10
07:52:58:WU01:FS01:0x22:       Bits: 64
07:52:58:WU01:FS01:0x22:       Mode: Release
07:52:58:WU01:FS01:0x22:************************************ System ************************************
07:52:58:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:52:58:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:52:58:WU01:FS01:0x22:       CPUs: 32
07:52:58:WU01:FS01:0x22:     Memory: 31.95GiB
07:52:58:WU01:FS01:0x22:Free Memory: 28.72GiB
07:52:58:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:52:58:WU01:FS01:0x22: OS Version: 6.2
07:52:58:WU01:FS01:0x22:Has Battery: false
07:52:58:WU01:FS01:0x22: On Battery: false
07:52:58:WU01:FS01:0x22: UTC Offset: -7
07:52:58:WU01:FS01:0x22:        PID: 2556
07:52:58:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:52:58:WU01:FS01:0x22:********************************************************************************
07:52:58:WU01:FS01:0x22:Project: 13426 (Run 3503, Clone 1, Gen 3)
07:52:58:WU01:FS01:0x22:Unit: 0x0000000712bc7d9a5f597cc1b9170394
07:52:58:WU01:FS01:0x22:Reading tar file core.xml
07:52:58:WU01:FS01:0x22:Reading tar file integrator.xml
07:52:58:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:52:58:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:52:58:WU01:FS01:0x22:Digital signatures verified
07:52:58:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:52:58:WU01:FS01:0x22:Version 0.0.11
07:52:58:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:52:58:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:52:58:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:52:58:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:53:10:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:53:10:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:53:10:WU01:FS01:0x22:Saving result file science.log
07:53:10:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:53:10:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:53:10:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3503 clone:1 gen:3 core:0x22 unit:0x0000000712bc7d9a5f597cc1b9170394
07:53:10:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:53:10:WU01:FS01:Connecting to 18.188.125.154:8080
07:53:10:WU01:FS01:Upload complete
07:53:10:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:53:10:WU01:FS01:Server responded WORK_ACK (400)
07:53:11:WU01:FS01:Cleaning up
07:53:11:WU02:FS01:Assigned to work server 18.188.125.154
07:53:11:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:53:11:WU02:FS01:Connecting to 18.188.125.154:8080
07:53:11:WU02:FS01:Downloading 5.02MiB
07:53:14:WU02:FS01:Download complete
07:53:14:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13430 run:63 clone:0 gen:5 core:0x22 unit:0x0000000612bc7d9a5f5e396524aae324
07:53:14:WU02:FS01:Starting
07:53:14:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:53:14:WU02:FS01:Started FahCore on PID 12344
07:53:14:WU02:FS01:Core PID:12552
07:53:14:WU02:FS01:FahCore 0x22 started
07:53:15:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:53:14Z ***********************
07:53:15:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:53:15:WU02:FS01:0x22:       Core: Core22
07:53:15:WU02:FS01:0x22:       Type: 0x22
07:53:15:WU02:FS01:0x22:    Version: 0.0.11
07:53:15:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:53:15:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:53:15:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:53:15:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:15:WU02:FS01:0x22:       Time: 19:49:16
07:53:15:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:53:15:WU02:FS01:0x22:     Branch: core22-0.0.11
07:53:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:15:WU02:FS01:0x22:   Platform: win32 10
07:53:15:WU02:FS01:0x22:       Bits: 64
07:53:15:WU02:FS01:0x22:       Mode: Release
07:53:15:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:53:15:WU02:FS01:0x22:             <[email protected]>
07:53:15:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 12344 -checkpoint 15
07:53:15:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:53:15:WU02:FS01:0x22:             0 -gpu 0
07:53:15:WU02:FS01:0x22:************************************ libFAH ************************************
07:53:15:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:15:WU02:FS01:0x22:       Time: 19:47:12
07:53:15:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:53:15:WU02:FS01:0x22:     Branch: HEAD
07:53:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:15:WU02:FS01:0x22:   Platform: win32 10
07:53:15:WU02:FS01:0x22:       Bits: 64
07:53:15:WU02:FS01:0x22:       Mode: Release
07:53:15:WU02:FS01:0x22:************************************ CBang *************************************
07:53:15:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:15:WU02:FS01:0x22:       Time: 19:46:11
07:53:15:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:53:15:WU02:FS01:0x22:     Branch: master
07:53:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:15:WU02:FS01:0x22:   Platform: win32 10
07:53:15:WU02:FS01:0x22:       Bits: 64
07:53:15:WU02:FS01:0x22:       Mode: Release
07:53:15:WU02:FS01:0x22:************************************ System ************************************
07:53:15:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:53:15:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:53:15:WU02:FS01:0x22:       CPUs: 32
07:53:15:WU02:FS01:0x22:     Memory: 31.95GiB
07:53:15:WU02:FS01:0x22:Free Memory: 28.83GiB
07:53:15:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:53:15:WU02:FS01:0x22: OS Version: 6.2
07:53:15:WU02:FS01:0x22:Has Battery: false
07:53:15:WU02:FS01:0x22: On Battery: false
07:53:15:WU02:FS01:0x22: UTC Offset: -7
07:53:15:WU02:FS01:0x22:        PID: 12552
07:53:15:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:53:15:WU02:FS01:0x22:********************************************************************************
07:53:15:WU02:FS01:0x22:Project: 13430 (Run 63, Clone 0, Gen 5)
07:53:15:WU02:FS01:0x22:Unit: 0x0000000612bc7d9a5f5e396524aae324
07:53:15:WU02:FS01:0x22:Reading tar file core.xml
07:53:15:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:53:15:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:53:15:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:53:15:WU02:FS01:0x22:Digital signatures verified
07:53:15:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:53:15:WU02:FS01:0x22:Version 0.0.11
07:53:15:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:53:15:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:53:15:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:53:15:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:53:21:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:53:21:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:53:21:WU02:FS01:0x22:Saving result file science.log
07:53:21:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:53:21:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:53:21:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13430 run:63 clone:0 gen:5 core:0x22 unit:0x0000000612bc7d9a5f5e396524aae324
07:53:21:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:53:21:WU02:FS01:Connecting to 18.188.125.154:8080
07:53:22:WU02:FS01:Upload complete
07:53:22:WU02:FS01:Server responded WORK_ACK (400)
07:53:22:WU02:FS01:Cleaning up
07:53:22:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:53:22:WU01:FS01:Assigned to work server 18.188.125.154
07:53:22:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:53:22:WU01:FS01:Connecting to 18.188.125.154:8080
07:53:23:WU01:FS01:Downloading 6.14MiB
07:53:26:WU01:FS01:Download complete
07:53:26:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3533 clone:2 gen:5 core:0x22 unit:0x0000000812bc7d9a5f597cc00fd88f19
07:53:26:WU01:FS01:Starting
07:53:26:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:53:26:WU01:FS01:Started FahCore on PID 8968
07:53:26:WU01:FS01:Core PID:11672
07:53:26:WU01:FS01:FahCore 0x22 started
07:53:27:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:53:26Z ***********************
07:53:27:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:53:27:WU01:FS01:0x22:       Core: Core22
07:53:27:WU01:FS01:0x22:       Type: 0x22
07:53:27:WU01:FS01:0x22:    Version: 0.0.11
07:53:27:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:53:27:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:53:27:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:53:27:WU01:FS01:0x22:       Date: Jun 26 2020
07:53:27:WU01:FS01:0x22:       Time: 19:49:16
07:53:27:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:53:27:WU01:FS01:0x22:     Branch: core22-0.0.11
07:53:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:53:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:27:WU01:FS01:0x22:   Platform: win32 10
07:53:27:WU01:FS01:0x22:       Bits: 64
07:53:27:WU01:FS01:0x22:       Mode: Release
07:53:27:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:53:27:WU01:FS01:0x22:             <[email protected]>
07:53:27:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 8968 -checkpoint 15
07:53:27:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:53:27:WU01:FS01:0x22:             0 -gpu 0
07:53:27:WU01:FS01:0x22:************************************ libFAH ************************************
07:53:27:WU01:FS01:0x22:       Date: Jun 26 2020
07:53:27:WU01:FS01:0x22:       Time: 19:47:12
07:53:27:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:53:27:WU01:FS01:0x22:     Branch: HEAD
07:53:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:53:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:27:WU01:FS01:0x22:   Platform: win32 10
07:53:27:WU01:FS01:0x22:       Bits: 64
07:53:27:WU01:FS01:0x22:       Mode: Release
07:53:27:WU01:FS01:0x22:************************************ CBang *************************************
07:53:27:WU01:FS01:0x22:       Date: Jun 26 2020
07:53:27:WU01:FS01:0x22:       Time: 19:46:11
07:53:27:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:53:27:WU01:FS01:0x22:     Branch: master
07:53:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:53:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:27:WU01:FS01:0x22:   Platform: win32 10
07:53:27:WU01:FS01:0x22:       Bits: 64
07:53:27:WU01:FS01:0x22:       Mode: Release
07:53:27:WU01:FS01:0x22:************************************ System ************************************
07:53:27:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:53:27:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:53:27:WU01:FS01:0x22:       CPUs: 32
07:53:27:WU01:FS01:0x22:     Memory: 31.95GiB
07:53:27:WU01:FS01:0x22:Free Memory: 28.81GiB
07:53:27:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:53:27:WU01:FS01:0x22: OS Version: 6.2
07:53:27:WU01:FS01:0x22:Has Battery: false
07:53:27:WU01:FS01:0x22: On Battery: false
07:53:27:WU01:FS01:0x22: UTC Offset: -7
07:53:27:WU01:FS01:0x22:        PID: 11672
07:53:27:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:53:27:WU01:FS01:0x22:********************************************************************************
07:53:27:WU01:FS01:0x22:Project: 13426 (Run 3533, Clone 2, Gen 5)
07:53:27:WU01:FS01:0x22:Unit: 0x0000000812bc7d9a5f597cc00fd88f19
07:53:27:WU01:FS01:0x22:Reading tar file core.xml
07:53:27:WU01:FS01:0x22:Reading tar file integrator.xml
07:53:27:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:53:27:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:53:27:WU01:FS01:0x22:Digital signatures verified
07:53:27:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:53:27:WU01:FS01:0x22:Version 0.0.11
07:53:27:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:53:27:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:53:27:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:53:27:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:53:30:Saving configuration to config.xml
07:53:30:<config>
07:53:30:  <!-- Folding Slot Configuration -->
07:53:30:  <cause v='COVID_19'/>
07:53:30:
07:53:30:  <!-- Slot Control -->
07:53:30:  <power v='FULL'/>
07:53:30:
07:53:30:  <!-- User Information -->
07:53:30:  <passkey v='*****'/>
07:53:30:  <team v='749'/>
07:53:30:  <user v='Rwolf01'/>
07:53:30:
07:53:30:  <!-- Folding Slots -->
07:53:30:  <slot id='0' type='CPU'/>
07:53:30:  <slot id='1' type='GPU'/>
07:53:30:</config>
07:53:38:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:53:38:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:53:38:WU01:FS01:0x22:Saving result file science.log
07:53:38:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:53:38:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:53:38:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3533 clone:2 gen:5 core:0x22 unit:0x0000000812bc7d9a5f597cc00fd88f19
07:53:38:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:53:38:WU01:FS01:Connecting to 18.188.125.154:8080
07:53:39:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:53:39:WU01:FS01:Upload complete
07:53:39:WU01:FS01:Server responded WORK_ACK (400)
07:53:39:WU01:FS01:Cleaning up
07:53:39:WU02:FS01:Assigned to work server 18.188.125.154
07:53:39:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:53:39:WU02:FS01:Connecting to 18.188.125.154:8080
07:53:40:WU02:FS01:Downloading 5.04MiB
07:53:42:WU02:FS01:Download complete
07:53:43:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13431 run:161 clone:5 gen:4 core:0x22 unit:0x0000000612bc7d9a5f5e323af5520541
07:53:43:WU02:FS01:Starting
07:53:43:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:53:43:WU02:FS01:Started FahCore on PID 12448
07:53:43:WU02:FS01:Core PID:7100
07:53:43:WU02:FS01:FahCore 0x22 started
07:53:43:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:53:43Z ***********************
07:53:43:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:53:43:WU02:FS01:0x22:       Core: Core22
07:53:43:WU02:FS01:0x22:       Type: 0x22
07:53:43:WU02:FS01:0x22:    Version: 0.0.11
07:53:43:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:53:43:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:53:43:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:53:43:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:43:WU02:FS01:0x22:       Time: 19:49:16
07:53:43:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:53:43:WU02:FS01:0x22:     Branch: core22-0.0.11
07:53:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:43:WU02:FS01:0x22:   Platform: win32 10
07:53:43:WU02:FS01:0x22:       Bits: 64
07:53:43:WU02:FS01:0x22:       Mode: Release
07:53:43:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:53:43:WU02:FS01:0x22:             <[email protected]>
07:53:43:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 12448 -checkpoint 15
07:53:43:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:53:43:WU02:FS01:0x22:             0 -gpu 0
07:53:43:WU02:FS01:0x22:************************************ libFAH ************************************
07:53:43:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:43:WU02:FS01:0x22:       Time: 19:47:12
07:53:43:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:53:43:WU02:FS01:0x22:     Branch: HEAD
07:53:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:43:WU02:FS01:0x22:   Platform: win32 10
07:53:43:WU02:FS01:0x22:       Bits: 64
07:53:43:WU02:FS01:0x22:       Mode: Release
07:53:43:WU02:FS01:0x22:************************************ CBang *************************************
07:53:43:WU02:FS01:0x22:       Date: Jun 26 2020
07:53:43:WU02:FS01:0x22:       Time: 19:46:11
07:53:43:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:53:43:WU02:FS01:0x22:     Branch: master
07:53:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:53:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:53:43:WU02:FS01:0x22:   Platform: win32 10
07:53:43:WU02:FS01:0x22:       Bits: 64
07:53:43:WU02:FS01:0x22:       Mode: Release
07:53:43:WU02:FS01:0x22:************************************ System ************************************
07:53:43:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:53:43:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:53:43:WU02:FS01:0x22:       CPUs: 32
07:53:43:WU02:FS01:0x22:     Memory: 31.95GiB
07:53:43:WU02:FS01:0x22:Free Memory: 28.86GiB
07:53:43:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:53:43:WU02:FS01:0x22: OS Version: 6.2
07:53:43:WU02:FS01:0x22:Has Battery: false
07:53:43:WU02:FS01:0x22: On Battery: false
07:53:43:WU02:FS01:0x22: UTC Offset: -7
07:53:43:WU02:FS01:0x22:        PID: 7100
07:53:43:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:53:43:WU02:FS01:0x22:********************************************************************************
07:53:43:WU02:FS01:0x22:Project: 13431 (Run 161, Clone 5, Gen 4)
07:53:43:WU02:FS01:0x22:Unit: 0x0000000612bc7d9a5f5e323af5520541
07:53:43:WU02:FS01:0x22:Reading tar file core.xml
07:53:43:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:53:43:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:53:43:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:53:43:WU02:FS01:0x22:Digital signatures verified
07:53:43:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:53:43:WU02:FS01:0x22:Version 0.0.11
07:53:43:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:53:43:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:53:43:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:53:43:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:53:49:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:53:49:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:53:49:WU02:FS01:0x22:Saving result file science.log
07:53:49:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:53:50:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:53:50:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13431 run:161 clone:5 gen:4 core:0x22 unit:0x0000000612bc7d9a5f5e323af5520541
07:53:50:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:53:50:WU02:FS01:Connecting to 18.188.125.154:8080
07:53:50:WU02:FS01:Upload complete
07:53:50:WU02:FS01:Server responded WORK_ACK (400)
07:53:50:WU02:FS01:Cleaning up
07:53:50:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:53:50:WU01:FS01:Assigned to work server 18.188.125.154
07:53:50:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:53:50:WU01:FS01:Connecting to 18.188.125.154:8080
07:53:51:WU01:FS01:Downloading 6.13MiB
07:53:51:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:53:52:WU00:FS00:Assigned to work server 143.89.243.111
07:53:52:WU00:FS00:Requesting new work unit for slot 00: READY cpu:31 from 143.89.243.111
07:53:52:WU00:FS00:Connecting to 143.89.243.111:8080
07:53:52:ERROR:WU00:FS00:Exception: Server did not assign work unit
07:53:53:FS00:Paused
07:53:53:FS01:Paused
07:53:54:WU01:FS01:Download complete
07:53:54:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3677 clone:18 gen:1 core:0x22 unit:0x0000000112bc7d9a5f597cbe0be6b89a
07:54:01:FS00:Unpaused
07:54:31:Saving configuration to config.xml
07:54:31:<config>
07:54:31:  <!-- Folding Slot Configuration -->
07:54:31:  <cause v='COVID_19'/>
07:54:31:
07:54:31:  <!-- Slot Control -->
07:54:31:  <power v='FULL'/>
07:54:31:
07:54:31:  <!-- User Information -->
07:54:31:  <passkey v='*****'/>
07:54:31:  <team v='749'/>
07:54:31:  <user v='Rwolf01'/>
07:54:31:
07:54:31:  <!-- Folding Slots -->
07:54:31:  <slot id='0' type='CPU'/>
07:54:31:  <slot id='1' type='GPU'>
07:54:31:    <paused v='true'/>
07:54:31:  </slot>
07:54:31:</config>
07:54:35:FS00:Paused
07:54:42:FS00:Unpaused
07:54:52:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:54:52:WU00:FS00:Assigned to work server 150.136.14.110
07:54:52:WU00:FS00:Requesting new work unit for slot 00: READY cpu:31 from 150.136.14.110
07:54:52:WU00:FS00:Connecting to 150.136.14.110:8080
07:54:52:WU00:FS00:Downloading 2.18MiB
07:54:54:WU00:FS00:Download complete
07:54:54:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16423 run:249 clone:4 gen:75 core:0xa7 unit:0x0000005596880e6e5e8a8e7c5b732d72
07:54:54:WU00:FS00:Downloading core from http://cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah
07:54:54:WU00:FS00:Connecting to cores.foldingathome.org:80
07:54:54:WU00:FS00:FahCore a7: Downloading 6.71MiB
07:54:57:WU00:FS00:FahCore a7: Download complete
07:54:57:WU00:FS00:Valid core signature
07:54:57:WU00:FS00:Unpacked 19.85MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe
07:54:57:WU00:FS00:Unpacked 2.64MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/libfftw3f-3.dll
07:54:57:WU00:FS00:Starting
07:54:57:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -np 31
07:54:57:WU00:FS00:Started FahCore on PID 13768
07:54:58:WU00:FS00:Core PID:6788
07:54:58:WU00:FS00:FahCore 0xa7 started
07:54:58:WU00:FS00:0xa7:*********************** Log Started 2020-09-16T07:54:58Z ***********************
07:54:58:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:54:58:WU00:FS00:0xa7:       Type: 0xa7
07:54:58:WU00:FS00:0xa7:       Core: Gromacs
07:54:58:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 13768 -checkpoint 15 -np
07:54:58:WU00:FS00:0xa7:             31
07:54:58:WU00:FS00:0xa7:************************************ CBang *************************************
07:54:58:WU00:FS00:0xa7:       Date: Nov 27 2019
07:54:58:WU00:FS00:0xa7:       Time: 03:40:09
07:54:58:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
07:54:58:WU00:FS00:0xa7:     Branch: master
07:54:58:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:54:58:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:54:58:WU00:FS00:0xa7:   Platform: win32 10
07:54:58:WU00:FS00:0xa7:       Bits: 64
07:54:58:WU00:FS00:0xa7:       Mode: Release
07:54:58:WU00:FS00:0xa7:************************************ System ************************************
07:54:58:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:54:58:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:54:58:WU00:FS00:0xa7:       CPUs: 32
07:54:58:WU00:FS00:0xa7:     Memory: 31.95GiB
07:54:58:WU00:FS00:0xa7:Free Memory: 28.89GiB
07:54:58:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
07:54:58:WU00:FS00:0xa7: OS Version: 6.2
07:54:58:WU00:FS00:0xa7:Has Battery: false
07:54:58:WU00:FS00:0xa7: On Battery: false
07:54:58:WU00:FS00:0xa7: UTC Offset: -7
07:54:58:WU00:FS00:0xa7:        PID: 6788
07:54:58:WU00:FS00:0xa7:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:54:58:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
07:54:58:WU00:FS00:0xa7:    Version: 0.0.19
07:54:58:WU00:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
07:54:58:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:54:58:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
07:54:58:WU00:FS00:0xa7:       Date: Nov 25 2019
07:54:58:WU00:FS00:0xa7:       Time: 17:12:41
07:54:58:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
07:54:58:WU00:FS00:0xa7:     Branch: master
07:54:58:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:54:58:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:54:58:WU00:FS00:0xa7:   Platform: win32 10
07:54:58:WU00:FS00:0xa7:       Bits: 64
07:54:58:WU00:FS00:0xa7:       Mode: Release
07:54:58:WU00:FS00:0xa7:************************************ Build *************************************
07:54:58:WU00:FS00:0xa7:       SIMD: avx_256
07:54:58:WU00:FS00:0xa7:********************************************************************************
07:54:58:WU00:FS00:0xa7:Project: 16423 (Run 249, Clone 4, Gen 75)
07:54:58:WU00:FS00:0xa7:Unit: 0x0000005596880e6e5e8a8e7c5b732d72
07:54:58:WU00:FS00:0xa7:Reading tar file core.xml
07:54:58:WU00:FS00:0xa7:Reading tar file frame75.tpr
07:54:58:WU00:FS00:0xa7:Digital signatures verified
07:54:58:WU00:FS00:0xa7:Reducing thread count from 31 to 30 to avoid domain decomposition by a prime number > 3
07:54:58:WU00:FS00:0xa7:Calling: mdrun -s frame75.tpr -o frame75.trr -x frame75.xtc -cpt 15 -nt 30
07:54:58:WU00:FS00:0xa7:Steps: first=18750000 total=250000
07:55:00:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
07:55:06:FS01:Unpaused
07:55:06:WU01:FS01:Starting
07:55:06:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:55:06:WU01:FS01:Started FahCore on PID 4628
07:55:06:WU01:FS01:Core PID:12988
07:55:06:WU01:FS01:FahCore 0x22 started
07:55:06:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:55:06Z ***********************
07:55:06:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:55:06:WU01:FS01:0x22:       Core: Core22
07:55:06:WU01:FS01:0x22:       Type: 0x22
07:55:06:WU01:FS01:0x22:    Version: 0.0.11
07:55:06:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:55:06:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:55:06:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:55:06:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:06:WU01:FS01:0x22:       Time: 19:49:16
07:55:06:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:55:06:WU01:FS01:0x22:     Branch: core22-0.0.11
07:55:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:06:WU01:FS01:0x22:   Platform: win32 10
07:55:06:WU01:FS01:0x22:       Bits: 64
07:55:06:WU01:FS01:0x22:       Mode: Release
07:55:06:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:55:06:WU01:FS01:0x22:             <[email protected]>
07:55:06:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4628 -checkpoint 15
07:55:06:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:55:06:WU01:FS01:0x22:             0 -gpu 0
07:55:06:WU01:FS01:0x22:************************************ libFAH ************************************
07:55:06:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:06:WU01:FS01:0x22:       Time: 19:47:12
07:55:06:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:55:06:WU01:FS01:0x22:     Branch: HEAD
07:55:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:06:WU01:FS01:0x22:   Platform: win32 10
07:55:06:WU01:FS01:0x22:       Bits: 64
07:55:06:WU01:FS01:0x22:       Mode: Release
07:55:06:WU01:FS01:0x22:************************************ CBang *************************************
07:55:06:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:06:WU01:FS01:0x22:       Time: 19:46:11
07:55:06:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:55:06:WU01:FS01:0x22:     Branch: master
07:55:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:06:WU01:FS01:0x22:   Platform: win32 10
07:55:06:WU01:FS01:0x22:       Bits: 64
07:55:06:WU01:FS01:0x22:       Mode: Release
07:55:06:WU01:FS01:0x22:************************************ System ************************************
07:55:06:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:55:06:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:55:06:WU01:FS01:0x22:       CPUs: 32
07:55:06:WU01:FS01:0x22:     Memory: 31.95GiB
07:55:06:WU01:FS01:0x22:Free Memory: 28.68GiB
07:55:06:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:55:06:WU01:FS01:0x22: OS Version: 6.2
07:55:06:WU01:FS01:0x22:Has Battery: false
07:55:06:WU01:FS01:0x22: On Battery: false
07:55:06:WU01:FS01:0x22: UTC Offset: -7
07:55:06:WU01:FS01:0x22:        PID: 12988
07:55:06:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:55:06:WU01:FS01:0x22:********************************************************************************
07:55:06:WU01:FS01:0x22:Project: 13426 (Run 3677, Clone 18, Gen 1)
07:55:06:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5f597cbe0be6b89a
07:55:06:WU01:FS01:0x22:Reading tar file core.xml
07:55:06:WU01:FS01:0x22:Reading tar file integrator.xml
07:55:06:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:55:06:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:55:06:WU01:FS01:0x22:Digital signatures verified
07:55:06:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:55:06:WU01:FS01:0x22:Version 0.0.11
07:55:06:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:55:06:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:55:06:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:55:06:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:55:23:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
07:55:28:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:55:28:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:55:28:WU01:FS01:0x22:Saving result file science.log
07:55:28:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:55:28:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:55:28:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3677 clone:18 gen:1 core:0x22 unit:0x0000000112bc7d9a5f597cbe0be6b89a
07:55:28:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:55:28:WU01:FS01:Connecting to 18.188.125.154:8080
07:55:29:WU01:FS01:Upload complete
07:55:29:WU01:FS01:Server responded WORK_ACK (400)
07:55:29:WU01:FS01:Cleaning up
07:55:29:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:55:29:WU01:FS01:Assigned to work server 18.188.125.154
07:55:29:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:55:29:WU01:FS01:Connecting to 18.188.125.154:8080
07:55:30:WU01:FS01:Downloading 6.13MiB
07:55:32:Saving configuration to config.xml
07:55:32:<config>
07:55:32:  <!-- Folding Slot Configuration -->
07:55:32:  <cause v='COVID_19'/>
07:55:32:
07:55:32:  <!-- Slot Control -->
07:55:32:  <power v='FULL'/>
07:55:32:
07:55:32:  <!-- User Information -->
07:55:32:  <passkey v='*****'/>
07:55:32:  <team v='749'/>
07:55:32:  <user v='Rwolf01'/>
07:55:32:
07:55:32:  <!-- Folding Slots -->
07:55:32:  <slot id='0' type='CPU'/>
07:55:32:  <slot id='1' type='GPU'/>
07:55:32:</config>
07:55:33:WU01:FS01:Download complete
07:55:33:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3636 clone:25 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbe240928a6
07:55:33:WU01:FS01:Starting
07:55:33:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:55:33:WU01:FS01:Started FahCore on PID 14060
07:55:33:WU01:FS01:Core PID:7964
07:55:33:WU01:FS01:FahCore 0x22 started
07:55:33:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:55:33Z ***********************
07:55:33:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:55:33:WU01:FS01:0x22:       Core: Core22
07:55:33:WU01:FS01:0x22:       Type: 0x22
07:55:33:WU01:FS01:0x22:    Version: 0.0.11
07:55:33:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:55:33:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:55:33:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:55:33:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:33:WU01:FS01:0x22:       Time: 19:49:16
07:55:33:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:55:33:WU01:FS01:0x22:     Branch: core22-0.0.11
07:55:33:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:33:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:33:WU01:FS01:0x22:   Platform: win32 10
07:55:33:WU01:FS01:0x22:       Bits: 64
07:55:33:WU01:FS01:0x22:       Mode: Release
07:55:33:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:55:33:WU01:FS01:0x22:             <[email protected]>
07:55:33:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14060 -checkpoint 15
07:55:33:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:55:33:WU01:FS01:0x22:             0 -gpu 0
07:55:33:WU01:FS01:0x22:************************************ libFAH ************************************
07:55:33:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:33:WU01:FS01:0x22:       Time: 19:47:12
07:55:33:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:55:33:WU01:FS01:0x22:     Branch: HEAD
07:55:33:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:33:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:33:WU01:FS01:0x22:   Platform: win32 10
07:55:33:WU01:FS01:0x22:       Bits: 64
07:55:33:WU01:FS01:0x22:       Mode: Release
07:55:33:WU01:FS01:0x22:************************************ CBang *************************************
07:55:33:WU01:FS01:0x22:       Date: Jun 26 2020
07:55:33:WU01:FS01:0x22:       Time: 19:46:11
07:55:33:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:55:33:WU01:FS01:0x22:     Branch: master
07:55:33:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:55:33:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:55:33:WU01:FS01:0x22:   Platform: win32 10
07:55:33:WU01:FS01:0x22:       Bits: 64
07:55:33:WU01:FS01:0x22:       Mode: Release
07:55:33:WU01:FS01:0x22:************************************ System ************************************
07:55:33:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:55:33:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:55:33:WU01:FS01:0x22:       CPUs: 32
07:55:33:WU01:FS01:0x22:     Memory: 31.95GiB
07:55:33:WU01:FS01:0x22:Free Memory: 28.67GiB
07:55:34:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:55:34:WU01:FS01:0x22: OS Version: 6.2
07:55:34:WU01:FS01:0x22:Has Battery: false
07:55:34:WU01:FS01:0x22: On Battery: false
07:55:34:WU01:FS01:0x22: UTC Offset: -7
07:55:34:WU01:FS01:0x22:        PID: 7964
07:55:34:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:55:34:WU01:FS01:0x22:********************************************************************************
07:55:34:WU01:FS01:0x22:Project: 13426 (Run 3636, Clone 25, Gen 2)
07:55:34:WU01:FS01:0x22:Unit: 0x0000000212bc7d9a5f597cbe240928a6
07:55:34:WU01:FS01:0x22:Reading tar file core.xml
07:55:34:WU01:FS01:0x22:Reading tar file integrator.xml
07:55:34:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:55:34:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:55:34:WU01:FS01:0x22:Digital signatures verified
07:55:34:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:55:34:WU01:FS01:0x22:Version 0.0.11
07:55:34:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:55:34:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:55:34:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:55:34:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:55:46:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
07:55:54:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:55:54:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:55:54:WU01:FS01:0x22:Saving result file science.log
07:55:54:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:55:54:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:55:55:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3636 clone:25 gen:2 core:0x22 unit:0x0000000212bc7d9a5f597cbe240928a6
07:55:55:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:55:55:WU01:FS01:Connecting to 18.188.125.154:8080
07:55:55:WU01:FS01:Upload complete
07:55:55:WU01:FS01:Server responded WORK_ACK (400)
07:55:55:WU01:FS01:Cleaning up
07:55:55:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:55:55:WU02:FS01:Assigned to work server 18.188.125.154
07:55:55:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:55:55:WU02:FS01:Connecting to 18.188.125.154:8080
07:55:56:WU02:FS01:Downloading 6.14MiB
07:55:59:WU02:FS01:Download complete
07:55:59:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13426 run:3319 clone:26 gen:5 core:0x22 unit:0x0000000612bc7d9a5f597cb33e33a7e9
07:55:59:WU02:FS01:Starting
07:55:59:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:55:59:WU02:FS01:Started FahCore on PID 13036
07:55:59:WU02:FS01:Core PID:13520
07:55:59:WU02:FS01:FahCore 0x22 started
07:56:00:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:55:59Z ***********************
07:56:00:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:56:00:WU02:FS01:0x22:       Core: Core22
07:56:00:WU02:FS01:0x22:       Type: 0x22
07:56:00:WU02:FS01:0x22:    Version: 0.0.11
07:56:00:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:56:00:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:56:00:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:56:00:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:00:WU02:FS01:0x22:       Time: 19:49:16
07:56:00:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:56:00:WU02:FS01:0x22:     Branch: core22-0.0.11
07:56:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:00:WU02:FS01:0x22:   Platform: win32 10
07:56:00:WU02:FS01:0x22:       Bits: 64
07:56:00:WU02:FS01:0x22:       Mode: Release
07:56:00:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:56:00:WU02:FS01:0x22:             <[email protected]>
07:56:00:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 13036 -checkpoint 15
07:56:00:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:56:00:WU02:FS01:0x22:             0 -gpu 0
07:56:00:WU02:FS01:0x22:************************************ libFAH ************************************
07:56:00:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:00:WU02:FS01:0x22:       Time: 19:47:12
07:56:00:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:56:00:WU02:FS01:0x22:     Branch: HEAD
07:56:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:00:WU02:FS01:0x22:   Platform: win32 10
07:56:00:WU02:FS01:0x22:       Bits: 64
07:56:00:WU02:FS01:0x22:       Mode: Release
07:56:00:WU02:FS01:0x22:************************************ CBang *************************************
07:56:00:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:00:WU02:FS01:0x22:       Time: 19:46:11
07:56:00:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:56:00:WU02:FS01:0x22:     Branch: master
07:56:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:00:WU02:FS01:0x22:   Platform: win32 10
07:56:00:WU02:FS01:0x22:       Bits: 64
07:56:00:WU02:FS01:0x22:       Mode: Release
07:56:00:WU02:FS01:0x22:************************************ System ************************************
07:56:00:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:56:00:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:56:00:WU02:FS01:0x22:       CPUs: 32
07:56:00:WU02:FS01:0x22:     Memory: 31.95GiB
07:56:00:WU02:FS01:0x22:Free Memory: 28.68GiB
07:56:00:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:56:00:WU02:FS01:0x22: OS Version: 6.2
07:56:00:WU02:FS01:0x22:Has Battery: false
07:56:00:WU02:FS01:0x22: On Battery: false
07:56:00:WU02:FS01:0x22: UTC Offset: -7
07:56:00:WU02:FS01:0x22:        PID: 13520
07:56:00:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:56:00:WU02:FS01:0x22:********************************************************************************
07:56:00:WU02:FS01:0x22:Project: 13426 (Run 3319, Clone 26, Gen 5)
07:56:00:WU02:FS01:0x22:Unit: 0x0000000612bc7d9a5f597cb33e33a7e9
07:56:00:WU02:FS01:0x22:Reading tar file core.xml
07:56:00:WU02:FS01:0x22:Reading tar file integrator.xml
07:56:00:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:56:00:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:56:00:WU02:FS01:0x22:Digital signatures verified
07:56:00:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:56:00:WU02:FS01:0x22:Version 0.0.11
07:56:00:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:56:00:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:56:00:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:56:00:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:56:09:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
07:56:21:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:56:21:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:56:21:WU02:FS01:0x22:Saving result file science.log
07:56:21:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:56:21:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:56:21:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13426 run:3319 clone:26 gen:5 core:0x22 unit:0x0000000612bc7d9a5f597cb33e33a7e9
07:56:21:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:56:21:WU02:FS01:Connecting to 18.188.125.154:8080
07:56:21:WU02:FS01:Upload complete
07:56:21:WU02:FS01:Server responded WORK_ACK (400)
07:56:21:WU02:FS01:Cleaning up
07:56:21:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:56:22:WU01:FS01:Assigned to work server 18.188.125.154
07:56:22:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:56:22:WU01:FS01:Connecting to 18.188.125.154:8080
07:56:22:WU01:FS01:Downloading 7.97MiB
07:56:27:WU01:FS01:Download complete
07:56:27:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13435 run:29 clone:5 gen:4 core:0x22 unit:0x0000000512bc7d9a5f5e323e33150ff6
07:56:27:WU01:FS01:Starting
07:56:27:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:56:27:WU01:FS01:Started FahCore on PID 10728
07:56:27:WU01:FS01:Core PID:10952
07:56:27:WU01:FS01:FahCore 0x22 started
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

Rest of that log...

Code: Select all

07:56:28:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:56:27Z ***********************
07:56:28:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:56:28:WU01:FS01:0x22:       Core: Core22
07:56:28:WU01:FS01:0x22:       Type: 0x22
07:56:28:WU01:FS01:0x22:    Version: 0.0.11
07:56:28:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:56:28:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:56:28:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:56:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:56:28:WU01:FS01:0x22:       Time: 19:49:16
07:56:28:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:56:28:WU01:FS01:0x22:     Branch: core22-0.0.11
07:56:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:56:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:28:WU01:FS01:0x22:   Platform: win32 10
07:56:28:WU01:FS01:0x22:       Bits: 64
07:56:28:WU01:FS01:0x22:       Mode: Release
07:56:28:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:56:28:WU01:FS01:0x22:             <[email protected]>
07:56:28:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10728 -checkpoint 15
07:56:28:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:56:28:WU01:FS01:0x22:             0 -gpu 0
07:56:28:WU01:FS01:0x22:************************************ libFAH ************************************
07:56:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:56:28:WU01:FS01:0x22:       Time: 19:47:12
07:56:28:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:56:28:WU01:FS01:0x22:     Branch: HEAD
07:56:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:56:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:28:WU01:FS01:0x22:   Platform: win32 10
07:56:28:WU01:FS01:0x22:       Bits: 64
07:56:28:WU01:FS01:0x22:       Mode: Release
07:56:28:WU01:FS01:0x22:************************************ CBang *************************************
07:56:28:WU01:FS01:0x22:       Date: Jun 26 2020
07:56:28:WU01:FS01:0x22:       Time: 19:46:11
07:56:28:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:56:28:WU01:FS01:0x22:     Branch: master
07:56:28:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:56:28:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:28:WU01:FS01:0x22:   Platform: win32 10
07:56:28:WU01:FS01:0x22:       Bits: 64
07:56:28:WU01:FS01:0x22:       Mode: Release
07:56:28:WU01:FS01:0x22:************************************ System ************************************
07:56:28:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:56:28:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:56:28:WU01:FS01:0x22:       CPUs: 32
07:56:28:WU01:FS01:0x22:     Memory: 31.95GiB
07:56:28:WU01:FS01:0x22:Free Memory: 28.69GiB
07:56:28:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:56:28:WU01:FS01:0x22: OS Version: 6.2
07:56:28:WU01:FS01:0x22:Has Battery: false
07:56:28:WU01:FS01:0x22: On Battery: false
07:56:28:WU01:FS01:0x22: UTC Offset: -7
07:56:28:WU01:FS01:0x22:        PID: 10952
07:56:28:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:56:28:WU01:FS01:0x22:********************************************************************************
07:56:28:WU01:FS01:0x22:Project: 13435 (Run 29, Clone 5, Gen 4)
07:56:28:WU01:FS01:0x22:Unit: 0x0000000512bc7d9a5f5e323e33150ff6
07:56:28:WU01:FS01:0x22:Reading tar file core.xml
07:56:28:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
07:56:28:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:56:28:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:56:28:WU01:FS01:0x22:Digital signatures verified
07:56:28:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:56:28:WU01:FS01:0x22:Version 0.0.11
07:56:28:WU01:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:56:28:WU01:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:56:28:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:56:28:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
07:56:31:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
07:56:47:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:56:47:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:56:47:WU01:FS01:0x22:Saving result file science.log
07:56:47:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:56:47:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:56:47:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13435 run:29 clone:5 gen:4 core:0x22 unit:0x0000000512bc7d9a5f5e323e33150ff6
07:56:47:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:56:47:WU01:FS01:Connecting to 18.188.125.154:8080
07:56:48:WU01:FS01:Upload complete
07:56:48:WU01:FS01:Server responded WORK_ACK (400)
07:56:48:WU01:FS01:Cleaning up
07:56:48:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:56:48:WU02:FS01:Assigned to work server 18.188.125.154
07:56:48:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:56:48:WU02:FS01:Connecting to 18.188.125.154:8080
07:56:48:WU02:FS01:Downloading 5.03MiB
07:56:51:WU02:FS01:Download complete
07:56:51:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13433 run:182 clone:6 gen:3 core:0x22 unit:0x0000000312bc7d9a5f5e323d42f594fc
07:56:51:WU02:FS01:Starting
07:56:51:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:56:51:WU02:FS01:Started FahCore on PID 7636
07:56:51:WU02:FS01:Core PID:13608
07:56:51:WU02:FS01:FahCore 0x22 started
07:56:52:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:56:51Z ***********************
07:56:52:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:56:52:WU02:FS01:0x22:       Core: Core22
07:56:52:WU02:FS01:0x22:       Type: 0x22
07:56:52:WU02:FS01:0x22:    Version: 0.0.11
07:56:52:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:56:52:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:56:52:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:56:52:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:52:WU02:FS01:0x22:       Time: 19:49:16
07:56:52:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:56:52:WU02:FS01:0x22:     Branch: core22-0.0.11
07:56:52:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:52:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:52:WU02:FS01:0x22:   Platform: win32 10
07:56:52:WU02:FS01:0x22:       Bits: 64
07:56:52:WU02:FS01:0x22:       Mode: Release
07:56:52:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:56:52:WU02:FS01:0x22:             <[email protected]>
07:56:52:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 7636 -checkpoint 15
07:56:52:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:56:52:WU02:FS01:0x22:             0 -gpu 0
07:56:52:WU02:FS01:0x22:************************************ libFAH ************************************
07:56:52:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:52:WU02:FS01:0x22:       Time: 19:47:12
07:56:52:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:56:52:WU02:FS01:0x22:     Branch: HEAD
07:56:52:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:52:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:52:WU02:FS01:0x22:   Platform: win32 10
07:56:52:WU02:FS01:0x22:       Bits: 64
07:56:52:WU02:FS01:0x22:       Mode: Release
07:56:52:WU02:FS01:0x22:************************************ CBang *************************************
07:56:52:WU02:FS01:0x22:       Date: Jun 26 2020
07:56:52:WU02:FS01:0x22:       Time: 19:46:11
07:56:52:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:56:52:WU02:FS01:0x22:     Branch: master
07:56:52:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:56:52:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:56:52:WU02:FS01:0x22:   Platform: win32 10
07:56:52:WU02:FS01:0x22:       Bits: 64
07:56:52:WU02:FS01:0x22:       Mode: Release
07:56:52:WU02:FS01:0x22:************************************ System ************************************
07:56:52:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:56:52:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:56:52:WU02:FS01:0x22:       CPUs: 32
07:56:52:WU02:FS01:0x22:     Memory: 31.95GiB
07:
56:52:WU02:FS01:0x22:Free Memory: 28.71GiB
07:56:52:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:56:52:WU02:FS01:0x22: OS Version: 6.2
07:56:52:WU02:FS01:0x22:Has Battery: false
07:56:52:WU02:FS01:0x22: On Battery: false
07:56:52:WU02:FS01:0x22: UTC Offset: -7
07:56:52:WU02:FS01:0x22:        PID: 13608
07:56:52:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:56:52:WU02:FS01:0x22:********************************************************************************
07:56:52:WU02:FS01:0x22:Project: 13433 (Run 182, Clone 6, Gen 3)
07:56:52:WU02:FS01:0x22:Unit: 0x0000000312bc7d9a5f5e323d42f594fc
07:56:52:WU02:FS01:0x22:Reading tar file core.xml
07:56:52:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:56:52:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:56:52:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:56:52:WU02:FS01:0x22:Digital signatures verified
07:56:52:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:56:52:WU02:FS01:0x22:Version 0.0.11
07:56:52:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:56:52:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:56:52:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:56:52:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:56:54:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
07:57:03:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:57:03:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:57:03:WU02:FS01:0x22:Saving result file science.log
07:57:03:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:57:04:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:57:04:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13433 run:182 clone:6 gen:3 core:0x22 unit:0x0000000312bc7d9a5f5e323d42f594fc
07:57:04:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:57:04:WU02:FS01:Connecting to 18.188.125.154:8080
07:57:04:WU02:FS01:Upload complete
07:57:04:WU02:FS01:Server responded WORK_ACK (400)
07:57:04:WU02:FS01:Cleaning up
07:57:04:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:57:04:WU01:FS01:Assigned to work server 18.188.125.154
07:57:04:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:57:04:WU01:FS01:Connecting to 18.188.125.154:8080
07:57:05:WU01:FS01:Downloading 6.13MiB
07:57:08:WU01:FS01:Download complete
07:57:08:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13426 run:3104 clone:18 gen:5 core:0x22 unit:0x0000000712bc7d9a5f597cb7540612f6
07:57:08:WU01:FS01:Starting
07:57:08:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:57:08:WU01:FS01:Started FahCore on PID 10332
07:57:09:WU01:FS01:Core PID:3464
07:57:09:WU01:FS01:FahCore 0x22 started
07:57:09:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:57:09Z ***********************
07:57:09:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:57:09:WU01:FS01:0x22:       Core: Core22
07:57:09:WU01:FS01:0x22:       Type: 0x22
07:57:09:WU01:FS01:0x22:    Version: 0.0.11
07:57:09:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:57:09:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:57:09:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:57:09:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:09:WU01:FS01:0x22:       Time: 19:49:16
07:57:09:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:57:09:WU01:FS01:0x22:     Branch: core22-0.0.11
07:57:09:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:09:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:09:WU01:FS01:0x22:   Platform: win32 10
07:57:09:WU01:FS01:0x22:       Bits: 64
07:57:09:WU01:FS01:0x22:       Mode: Release
07:57:09:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:57:09:WU01:FS01:0x22:             <[email protected]>
07:57:09:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10332 -checkpoint 15
07:57:09:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:57:09:WU01:FS01:0x22:             0 -gpu 0
07:57:09:WU01:FS01:0x22:************************************ libFAH ************************************
07:57:09:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:09:WU01:FS01:0x22:       Time: 19:47:12
07:57:09:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:57:09:WU01:FS01:0x22:     Branch: HEAD
07:57:09:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:09:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:09:WU01:FS01:0x22:   Platform: win32 10
07:57:09:WU01:FS01:0x22:       Bits: 64
07:57:09:WU01:FS01:0x22:       Mode: Release
07:57:09:WU01:FS01:0x22:************************************ CBang *************************************
07:57:09:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:09:WU01:FS01:0x22:       Time: 19:46:11
07:57:09:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:57:09:WU01:FS01:0x22:     Branch: master
07:57:09:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:09:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:09:WU01:FS01:0x22:   Platform: win32 10
07:57:09:WU01:FS01:0x22:       Bits: 64
07:57:09:WU01:FS01:0x22:       Mode: Release
07:57:09:WU01:FS01:0x22:************************************ System ************************************
07:57:09:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:57:09:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:57:09:WU01:FS01:0x22:       CPUs: 32
07:57:09:WU01:FS01:0x22:     Memory: 31.95GiB
07:57:09:WU01:FS01:0x22:Free Memory: 28.71GiB
07:57:09:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:57:09:WU01:FS01:0x22: OS Version: 6.2
07:57:09:WU01:FS01:0x22:Has Battery: false
07:57:09:WU01:FS01:0x22: On Battery: false
07:57:09:WU01:FS01:0x22: UTC Offset: -7
07:57:09:WU01:FS01:0x22:        PID: 3464
07:57:09:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:57:09:WU01:FS01:0x22:********************************************************************************
07:57:09:WU01:FS01:0x22:Project: 13426 (Run 3104, Clone 18, Gen 5)
07:57:09:WU01:FS01:0x22:Unit: 0x0000000712bc7d9a5f597cb7540612f6
07:57:09:WU01:FS01:0x22:Reading tar file core.xml
07:57:09:WU01:FS01:0x22:Reading tar file integrator.xml
07:57:09:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:57:09:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:57:09:WU01:FS01:0x22:Digital signatures verified
07:57:09:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:57:09:WU01:FS01:0x22:Version 0.0.11
07:57:09:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
07:57:09:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
07:57:09:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
07:57:09:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
07:57:16:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
07:57:29:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:57:29:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:57:29:WU01:FS01:0x22:Saving result file science.log
07:57:29:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:57:29:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:57:29:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13426 run:3104 clone:18 gen:5 core:0x22 unit:0x0000000712bc7d9a5f597cb7540612f6
07:57:29:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:57:29:WU01:FS01:Connecting to 18.188.125.154:8080
07:57:30:WU01:FS01:Upload complete
07:57:30:WU01:FS01:Server responded WORK_ACK (400)
07:57:30:WU01:FS01:Cleaning up
07:57:30:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:57:30:WU02:FS01:Assigned to work server 18.188.125.154
07:57:30:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:57:30:WU02:FS01:Connecting to 18.188.125.154:8080
07:57:31:WU02:FS01:Downloading 7.20MiB
07:57:34:WU02:FS01:Download complete
07:57:34:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13435 run:56 clone:5 gen:5 core:0x22 unit:0x0000000812bc7d9a5f5e324214b9ae98
07:57:34:WU02:FS01:Starting
07:57:34:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:57:34:WU02:FS01:Started FahCore on PID 7272
07:57:34:WU02:FS01:Core PID:1160
07:57:34:WU02:FS01:FahCore 0x22 started
07:57:35:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:57:34Z ***********************
07:57:35:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:57:35:WU02:FS01:0x22:       Core: Core22
07:57:35:WU02:FS01:0x22:       Type: 0x22
07:57:35:WU02:FS01:0x22:    Version: 0.0.11
07:57:35:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:57:35:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:57:35:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:57:35:WU02:FS01:0x22:       Date: Jun 26 2020
07:57:35:WU02:FS01:0x22:       Time: 19:49:16
07:57:35:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:57:35:WU02:FS01:0x22:     Branch: core22-0.0.11
07:57:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:57:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:35:WU02:FS01:0x22:   Platform: win32 10
07:57:35:WU02:FS01:0x22:       Bits: 64
07:57:35:WU02:FS01:0x22:       Mode: Release
07:57:35:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:57:35:WU02:FS01:0x22:             <[email protected]>
07:57:35:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 7272 -checkpoint 15
07:57:35:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:57:35:WU02:FS01:0x22:             0 -gpu 0
07:57:35:WU02:FS01:0x22:************************************ libFAH ************************************
07:57:35:WU02:FS01:0x22:       Date: Jun 26 2020
07:57:35:WU02:FS01:0x22:       Time: 19:47:12
07:57:35:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:57:35:WU02:FS01:0x22:     Branch: HEAD
07:57:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:57:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:35:WU02:FS01:0x22:   Platform: win32 10
07:57:35:WU02:FS01:0x22:       Bits: 64
07:57:35:WU02:FS01:0x22:       Mode: Release
07:57:35:WU02:FS01:0x22:************************************ CBang *************************************
07:57:35:WU02:FS01:0x22:       Date: Jun 26 2020
07:57:35:WU02:FS01:0x22:       Time: 19:46:11
07:57:35:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:57:35:WU02:FS01:0x22:     Branch: master
07:57:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:57:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:35:WU02:FS01:0x22:   Platform: win32 10
07:57:35:WU02:FS01:0x22:       Bits: 64
07:57:35:WU02:FS01:0x22:       Mode: Release
07:57:35:WU02:FS01:0x22:************************************ System ************************************
07:57:35:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:57:35:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:57:35:WU02:FS01:0x22:       CPUs: 32
07:57:35:WU02:FS01:0x22:     Memory: 31.95GiB
07:57:35:WU02:FS01:0x22:Free Memory: 28.72GiB
07:57:35:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:57:35:WU02:FS01:0x22: OS Version: 6.2
07:57:35:WU02:FS01:0x22:Has Battery: false
07:57:35:WU02:FS01:0x22: On Battery: false
07:57:35:WU02:FS01:0x22: UTC Offset: -7
07:57:35:WU02:FS01:0x22:        PID: 1160
07:57:35:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:57:35:WU02:FS01:0x22:********************************************************************************
07:57:35:WU02:FS01:0x22:Project: 13435 (Run 56, Clone 5, Gen 5)
07:57:35:WU02:FS01:0x22:Unit: 0x0000000812bc7d9a5f5e324214b9ae98
07:57:35:WU02:FS01:0x22:Reading tar file core.xml
07:57:35:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:57:35:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:57:35:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:57:35:WU02:FS01:0x22:Digital signatures verified
07:57:35:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:57:35:WU02:FS01:0x22:Version 0.0.11
07:57:35:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:57:35:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:57:35:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:57:35:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:57:38:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
07:57:53:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:57:53:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:57:53:WU02:FS01:0x22:Saving result file science.log
07:57:53:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:57:53:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:57:53:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13435 run:56 clone:5 gen:5 core:0x22 unit:0x0000000812bc7d9a5f5e324214b9ae98
07:57:53:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:57:53:WU02:FS01:Connecting to 18.188.125.154:8080
07:57:53:WU02:FS01:Upload complete
07:57:53:WU02:FS01:Server responded WORK_ACK (400)
07:57:53:WU02:FS01:Cleaning up
07:57:53:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:57:54:WU01:FS01:Assigned to work server 18.188.125.154
07:57:54:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:57:54:WU01:FS01:Connecting to 18.188.125.154:8080
07:57:54:WU01:FS01:Downloading 7.23MiB
07:57:58:WU01:FS01:Download complete
07:57:58:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13434 run:93 clone:1 gen:0 core:0x22 unit:0x0000000112bc7d9a5f5e323f75afef71
07:57:58:WU01:FS01:Starting
07:57:58:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:57:58:WU01:FS01:Started FahCore on PID 11664
07:57:58:WU01:FS01:Core PID:9228
07:57:58:WU01:FS01:FahCore 0x22 started
07:57:59:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:57:58Z ***********************
07:57:59:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:57:59:WU01:FS01:0x22:       Core: Core22
07:57:59:WU01:FS01:0x22:       Type: 0x22
07:57:59:WU01:FS01:0x22:    Version: 0.0.11
07:57:59:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:57:59:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:57:59:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:57:59:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:59:WU01:FS01:0x22:       Time: 19:49:16
07:57:59:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:57:59:WU01:FS01:0x22:     Branch: core22-0.0.11
07:57:59:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:59:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:59:WU01:FS01:0x22:   Platform: win32 10
07:57:59:WU01:FS01:0x22:       Bits: 64
07:57:59:WU01:FS01:0x22:       Mode: Release
07:57:59:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:57:59:WU01:FS01:0x22:             <[email protected]>
07:57:59:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 11664 -checkpoint 15
07:57:59:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:57:59:WU01:FS01:0x22:             0 -gpu 0
07:57:59:WU01:FS01:0x22:************************************ libFAH ************************************
07:57:59:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:59:WU01:FS01:0x22:       Time: 19:47:12
07:57:59:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:57:59:WU01:FS01:0x22:     Branch: HEAD
07:57:59:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:59:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:59:WU01:FS01:0x22:   Platform: win32 10
07:57:59:WU01:FS01:0x22:       Bits: 64
07:57:59:WU01:FS01:0x22:       Mode: Release
07:57:59:WU01:FS01:0x22:************************************ CBang *************************************
07:57:59:WU01:FS01:0x22:       Date: Jun 26 2020
07:57:59:WU01:FS01:0x22:       Time: 19:46:11
07:57:59:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:57:59:WU01:FS01:0x22:     Branch: master
07:57:59:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:57:59:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:57:59:WU01:FS01:0x22:   Platform: win32 10
07:57:59:WU01:FS01:0x22:       Bits: 64
07:57:59:WU01:FS01:0x22:       Mode: Release
07:57:59:WU01:FS01:0x22:************************************ System ************************************
07:57:59:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:57:59:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:57:59:WU01:FS01:0x22:       CPUs: 32
07:57:59:WU01:FS01:0x22:     Memory: 31.95GiB
07:57:59:WU01:FS01:0x22:Free Memory: 28.74GiB
07:57:59:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:57:59:WU01:FS01:0x22: OS Version: 6.2
07:57:59:WU01:FS01:0x22:Has Battery: false
07:57:59:WU01:FS01:0x22: On Battery: false
07:57:59:WU01:FS01:0x22: UTC Offset: -7
07:57:59:WU01:FS01:0x22:        PID: 9228
07:57:59:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:57:59:WU01:FS01:0x22:********************************************************************************
07:57:59:WU01:FS01:0x22:Project: 13434 (Run 93, Clone 1, Gen 0)
07:57:59:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5f5e323f75afef71
07:57:59:WU01:FS01:0x22:Reading tar file core.xml
07:57:59:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
07:57:59:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:57:59:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:57:59:WU01:FS01:0x22:Digital signatures verified
07:57:59:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:57:59:WU01:FS01:0x22:Version 0.0.11
07:57:59:WU01:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:57:59:WU01:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:57:59:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:57:59:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
07:58:00:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
07:58:16:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:58:16:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:58:16:WU01:FS01:0x22:Saving result file science.log
07:58:16:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:58:16:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:58:16:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13434 run:93 clone:1 gen:0 core:0x22 unit:0x0000000112bc7d9a5f5e323f75afef71
07:58:16:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:58:16:WU01:FS01:Connecting to 18.188.125.154:8080
07:58:16:WU01:FS01:Upload complete
07:58:16:WU01:FS01:Server responded WORK_ACK (400)
07:58:16:WU01:FS01:Cleaning up
07:58:17:WU01:FS01:Connecting to assign1.foldingathome.org:80
07:58:17:WU01:FS01:Assigned to work server 18.188.125.154
07:58:17:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:58:17:WU01:FS01:Connecting to 18.188.125.154:8080
07:58:18:WU01:FS01:Downloading 7.43MiB
07:58:22:WU01:FS01:Download complete
07:58:22:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13436 run:112 clone:1 gen:3 core:0x22 unit:0x0000000612bc7d9a5f5e32475c03228d
07:58:22:WU01:FS01:Starting
07:58:22:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:58:22:WU01:FS01:Started FahCore on PID 12672
07:58:22:WU01:FS01:Core PID:6796
07:58:22:WU01:FS01:FahCore 0x22 started
07:58:22:WU01:FS01:0x22:*********************** Log Started 2020-09-16T07:58:22Z ***********************
07:58:22:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:58:22:WU01:FS01:0x22:       Core: Core22
07:58:22:WU01:FS01:0x22:       Type: 0x22
07:58:22:WU01:FS01:0x22:    Version: 0.0.11
07:58:22:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:58:22:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
07:58:22:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
07:58:22:WU01:FS01:0x22:       Date: Jun 26 2020
07:58:22:WU01:FS01:0x22:       Time: 19:49:16
07:58:22:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:58:22:WU01:FS01:0x22:     Branch: core22-0.0.11
07:58:22:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:58:22:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:22:WU01:FS01:0x22:   Platform: win32 10
07:58:22:WU01:FS01:0x22:       Bits: 64
07:58:22:WU01:FS01:0x22:       Mode: Release
07:58:22:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:58:22:WU01:FS01:0x22:             <[email protected]>
07:58:22:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 12672 -checkpoint 15
07:58:22:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:58:22:WU01:FS01:0x22:             0 -gpu 0
07:58:22:WU01:FS01:0x22:************************************ libFAH ************************************
07:58:22:WU01:FS01:0x22:       Date: Jun 26 2020
07:58:22:WU01:FS01:0x22:       Time: 19:47:12
07:58:22:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:58:22:WU01:FS01:0x22:     Branch: HEAD
07:58:22:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:58:22:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:22:WU01:FS01:0x22:   Platform: win32 10
07:58:22:WU01:FS01:0x22:       Bits: 64
07:58:22:WU01:FS01:0x22:       Mode: Release
07:58:22:WU01:FS01:0x22:************************************ CBang *************************************
07:58:22:WU01:FS01:0x22:       Date: Jun 26 2020
07:58:22:WU01:FS01:0x22:       Time: 19:46:11
07:58:22:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:58:22:WU01:FS01:0x22:     Branch: master
07:58:22:WU01:FS01:0x22:   Compiler: Visual C++ 2015
07:58:22:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:22:WU01:FS01:0x22:   Platform: win32 10
07:58:22:WU01:FS01:0x22:       Bits: 64
07:58:22:WU01:FS01:0x22:       Mode: Release
07:58:22:WU01:FS01:0x22:************************************ System ************************************
07:58:22:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:58:22:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:58:22:WU01:FS01:0x22:       CPUs: 32
07:58:22:WU01:FS01:0x22:     Memory: 31.95GiB
07:58:22:WU01:FS01:0x22:Free Memory: 28.75GiB
07:58:22:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
07:58:22:WU01:FS01:0x22: OS Version: 6.2
07:58:22:WU01:FS01:0x22:Has Battery: false
07:58:22:WU01:FS01:0x22: On Battery: false
07:58:22:WU01:FS01:0x22: UTC Offset: -7
07:58:22:WU01:FS01:0x22:        PID: 6796
07:58:22:WU01:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:58:22:WU01:FS01:0x22:********************************************************************************
07:58:22:WU01:FS01:0x22:Project: 13436 (Run 112, Clone 1, Gen 3)
07:58:22:WU01:FS01:0x22:Unit: 0x0000000612bc7d9a5f5e32475c03228d
07:58:22:WU01:FS01:0x22:Reading tar file core.xml
07:58:22:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
07:58:22:WU01:FS01:0x22:Reading tar file state.xml.bz2
07:58:22:WU01:FS01:0x22:Reading tar file system.xml.bz2
07:58:22:WU01:FS01:0x22:Digital signatures verified
07:58:22:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:58:22:WU01:FS01:0x22:Version 0.0.11
07:58:22:WU01:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:58:22:WU01:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:58:22:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:58:22:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
07:58:23:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
07:58:40:WU01:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:58:40:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
07:58:40:WU01:FS01:0x22:Saving result file science.log
07:58:40:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:58:41:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:58:41:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13436 run:112 clone:1 gen:3 core:0x22 unit:0x0000000612bc7d9a5f5e32475c03228d
07:58:41:WU01:FS01:Uploading 2.43KiB to 18.188.125.154
07:58:41:WU01:FS01:Connecting to 18.188.125.154:8080
07:58:41:WU01:FS01:Upload complete
07:58:41:WU01:FS01:Server responded WORK_ACK (400)
07:58:41:WU01:FS01:Cleaning up
07:58:41:WU02:FS01:Connecting to assign1.foldingathome.org:80
07:58:41:WU02:FS01:Assigned to work server 18.188.125.154
07:58:41:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [P106-090] from 18.188.125.154
07:58:41:WU02:FS01:Connecting to 18.188.125.154:8080
07:58:42:WU02:FS01:Downloading 7.87MiB
07:58:45:WU00:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
07:58:46:WU02:FS01:Download complete
07:58:46:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13435 run:130 clone:4 gen:4 core:0x22 unit:0x0000000412bc7d9a5f5e32432bfe49e5
07:58:46:WU02:FS01:Starting
07:58:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Rwolf01\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12744 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:58:46:WU02:FS01:Started FahCore on PID 5932
07:58:46:WU02:FS01:Core PID:14264
07:58:46:WU02:FS01:FahCore 0x22 started
07:58:47:WU02:FS01:0x22:*********************** Log Started 2020-09-16T07:58:46Z ***********************
07:58:47:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
07:58:47:WU02:FS01:0x22:       Core: Core22
07:58:47:WU02:FS01:0x22:       Type: 0x22
07:58:47:WU02:FS01:0x22:    Version: 0.0.11
07:58:47:WU02:FS01:0x22:     Author: Joseph Coffland <[email protected]>
07:58:47:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
07:58:47:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
07:58:47:WU02:FS01:0x22:       Date: Jun 26 2020
07:58:47:WU02:FS01:0x22:       Time: 19:49:16
07:58:47:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
07:58:47:WU02:FS01:0x22:     Branch: core22-0.0.11
07:58:47:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:58:47:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:47:WU02:FS01:0x22:   Platform: win32 10
07:58:47:WU02:FS01:0x22:       Bits: 64
07:58:47:WU02:FS01:0x22:       Mode: Release
07:58:47:WU02:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
07:58:47:WU02:FS01:0x22:             <[email protected]>
07:58:47:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 5932 -checkpoint 15
07:58:47:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
07:58:47:WU02:FS01:0x22:             0 -gpu 0
07:58:47:WU02:FS01:0x22:************************************ libFAH ************************************
07:58:47:WU02:FS01:0x22:       Date: Jun 26 2020
07:58:47:WU02:FS01:0x22:       Time: 19:47:12
07:58:47:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
07:58:47:WU02:FS01:0x22:     Branch: HEAD
07:58:47:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:58:47:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:47:WU02:FS01:0x22:   Platform: win32 10
07:58:47:WU02:FS01:0x22:       Bits: 64
07:58:47:WU02:FS01:0x22:       Mode: Release
07:58:47:WU02:FS01:0x22:************************************ CBang *************************************
07:58:47:WU02:FS01:0x22:       Date: Jun 26 2020
07:58:47:WU02:FS01:0x22:       Time: 19:46:11
07:58:47:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
07:58:47:WU02:FS01:0x22:     Branch: master
07:58:47:WU02:FS01:0x22:   Compiler: Visual C++ 2015
07:58:47:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
07:58:47:WU02:FS01:0x22:   Platform: win32 10
07:58:47:WU02:FS01:0x22:       Bits: 64
07:58:47:WU02:FS01:0x22:       Mode: Release
07:58:47:WU02:FS01:0x22:************************************ System ************************************
07:58:47:WU02:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
07:58:47:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
07:58:47:WU02:FS01:0x22:       CPUs: 32
07:58:47:WU02:FS01:0x22:     Memory: 31.95GiB
07:58:47:WU02:FS01:0x22:Free Memory: 28.78GiB
07:58:47:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
07:58:47:WU02:FS01:0x22: OS Version: 6.2
07:58:47:WU02:FS01:0x22:Has Battery: false
07:58:47:WU02:FS01:0x22: On Battery: false
07:58:47:WU02:FS01:0x22: UTC Offset: -7
07:58:47:WU02:FS01:0x22:        PID: 14264
07:58:47:WU02:FS01:0x22:        CWD: C:\Users\Rwolf01\AppData\Roaming\FAHClient\work
07:58:47:WU02:FS01:0x22:********************************************************************************
07:58:47:WU02:FS01:0x22:Project: 13435 (Run 130, Clone 4, Gen 4)
07:58:47:WU02:FS01:0x22:Unit: 0x0000000412bc7d9a5f5e32432bfe49e5
07:58:47:WU02:FS01:0x22:Reading tar file core.xml
07:58:47:WU02:FS01:0x22:Reading tar file integrator.xml.bz2
07:58:47:WU02:FS01:0x22:Reading tar file state.xml.bz2
07:58:47:WU02:FS01:0x22:Reading tar file system.xml.bz2
07:58:47:WU02:FS01:0x22:Digital signatures verified
07:58:47:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
07:58:47:WU02:FS01:0x22:Version 0.0.11
07:58:47:WU02:FS01:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
07:58:47:WU02:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
07:58:47:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
07:58:47:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
07:59:06:WU02:FS01:0x22:ERROR:exception: Error compiling kernel: 
07:59:06:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
07:59:06:WU02:FS01:0x22:Saving result file science.log
07:59:06:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
07:59:06:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
07:59:06:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13435 run:130 clone:4 gen:4 core:0x22 unit:0x0000000412bc7d9a5f5e32432bfe49e5
07:59:06:WU02:FS01:Uploading 2.43KiB to 18.188.125.154
07:59:06:WU02:FS01:Connecting to 18.188.125.154:8080
07:59:06:WU02:FS01:Upload complete
07:59:06:WU02:FS01:Server responded WORK_ACK (400)
07:59:06:WU02:FS01:Cleaning up
07:59:07:WU00:FS00:0xa7:Completed 27500 out of 250000 steps (11%)
07:59:30:WU00:FS00:0xa7:Completed 30000 out of 250000 steps (12%)
07:59:52:WU00:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
08:00:15:WU00:FS00:0xa7:Completed 35000 out of 250000 steps (14%)
08:00:37:WU00:FS00:0xa7:Completed 37500 out of 250000 steps (15%)
08:01:00:WU00:FS00:0xa7:Completed 40000 out of 250000 steps (16%)
08:01:22:WU00:FS00:0xa7:Completed 42500 out of 250000 steps (17%)
08:01:44:WU00:FS00:0xa7:Completed 45000 out of 250000 steps (18%)
08:02:07:WU00:FS00:0xa7:Completed 47500 out of 250000 steps (19%)
08:02:29:WU00:FS00:0xa7:Completed 50000 out of 250000 steps (20%)
08:02:52:WU00:FS00:0xa7:Completed 52500 out of 250000 steps (21%)
08:03:14:WU00:FS00:0xa7:Completed 55000 out of 250000 steps (22%)
08:03:37:WU00:FS00:0xa7:Completed 57500 out of 250000 steps (23%)
08:03:59:WU00:FS00:0xa7:Completed 60000 out of 250000 steps (24%)
08:04:22:WU00:FS00:0xa7:Completed 62500 out of 250000 steps (25%)
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

For completeness, I shut the system down and swapped the remaining GPU with an identical model in another working system. (I litterally paused it in the middle of folding 4 jobs and shut it down). WHen I rebooted both system, the other one happily picked up where it left off on the 4 jobs, and the problem child proceeded to fail again as shown above.

So it's clearly not the GPU hardware.

I welcome any other suggestions for things to try or inspect.
Top
Rwolf01
Posts: 20
Joined: Mon Jun 01, 2020 8:06 pm

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by Rwolf01 »

>> Can you run GPU-Z to ensure that you can properly detect the 3 GPUs and that it shows CUDA and OpenCL support?

For simplicity, I've pulled 2 of the GPUs. It still fails, so I'd like to debug that configuration before putting the other GPUs back.

Using the latest version of GPU-Z I compare the details on this system, vs one that is working.

The Problem Child has a newer version of the nVidia driver (I upgrader as an early attempt at fixing the problem) They all have OpenCL and the CUDA checkbox checked under "computing".

I do see a difference under "Technologies": The working system has PhysX and OpenGL 4.6 checked. The failing system has PhysX unchecked and OpenGL 1.1 instead of 4.6.

But before we make too much of this, I have a 3rd system which shows the same PhysX and OpenGL configuration as the problem child, and it is folding away just fine.

>> describe how your GPU is connected to the motherboard (using risers, etc.) and what kind of PSU you have.

Host is a Dell T5610, dual Xeon workstation. 2 GPUs plug into the MB directly, the 3rd is on a 180mm x16 ribbon cable riser. (also in use on 2 other systems w/o problems). It has an 825W Dell supply, which provides 2 6-pin PCIe connectors. For the 3rd GPU, I get the 12V from the SATA power bus using an aftermarket adapter cord. (The machine can support up to 6 SATA devices, and it only has 1 tiny SSD, so I'm sure there is plenty of 12V available for the GPU)

I put a power meter on the AC input line earlier and it was well below 825W, even with all slots folding.

Note: My 3 working systems are Dell T3610s. (Single CPU variant of the same basic workstation) They have smaller 685W supplies and can drive 3 GPUs just fine.

>> ... Restart the client and then simply test with one GPU to ensure that the log is readable.

I physically removed 2 of the GPUs and reinstalled FAH. Problem is still there. I've posted the complete log from that simpler config above in 2 parts.

Can you see anythign in that log which might indicate what the problem is?
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by PantherX »

Okay, so a quick recap is that you have 3 systems with identical hardware except one system which has dual CPUs. The dual CPUs is the one that is having issues. You swapped GPUs around from dual CPU to single CPU systems and they work. That tells me that the GPUs are functional.

Out of curiosity, any software differences? F@H doesn't use PhysX or OpenGL (except when rendering in FAHViewer).

The issue seems to be software related due to this error:
ERROR:exception: Error compiling kernel:

It is happening when attempting to start up the WU and fails. Currently it is attempting to use OpenCL which should work fine given you have the drivers installed.

There's a new FahCore_22 version 0.0.12 in development that attempts to debug these issues. While I don't have an ETA, I am aware that it will be soon. In the meantime, I will ask around and see what I can do for you :)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: 3 GPUS say "Failed" (red background) after a few retries

Post by PantherX »

BTW, I notice that you tend to not configure the clients with Donor details. That's absolutely fine. However, if you were keeping track of points, it is suggested to use a username (https://foldingathome.org/support/faq/s ... usernames/) and passkey (https://foldingathome.org/support/faq/points/passkey/).
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
Post Reply

Return to “V7.6.x Public Release Windows/Linux/MacOS X”