GPU WU's constantly failing

rfharbin · Post by **rfharbin** » Wed Jul 18, 2018 5:05 pm

I have had trouble folding GPU WU’s for several weeks now. They fail and then hide the work queue from the FAHControl. I have uninstalled and reinstalled a couple times and updated from the 7.4.4 version to 7.5.1 but nothing helps. I also have searched my computer for any leftover junk after uninstalling and there is none.

I am hoping for a cure from somebody as this is taking a lot of my "extra" time.

Code: Select all

[color=#0000FF][/color]Failed GPU WU 18July18 1st 
******************************* Date: 2018-07-18 *******************************
10:25:23:FS01:Unpaused
10:25:23:WU02:FS01:Starting
10:25:23:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:25:23:WU02:FS01:Started FahCore on PID 16512
10:25:23:WU02:FS01:Core PID:13948
10:25:23:WU02:FS01:FahCore 0x21 started
10:25:23:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:25:23:WU02:FS01:Starting
10:25:23:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:25:23:WU02:FS01:Started FahCore on PID 28800
10:25:23:WU02:FS01:Core PID:19136
10:25:23:WU02:FS01:FahCore 0x21 started
10:25:24:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:26:23:WU02:FS01:Starting
10:26:23:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:26:24:WU02:FS01:Started FahCore on PID 7952
10:26:24:WU02:FS01:Core PID:24884
10:26:24:WU02:FS01:FahCore 0x21 started
10:26:24:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:26:24:WARNING:WU02:FS01:Too many errors, failing
10:26:24:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11715 run:1 clone:1031 gen:32 core:0x21 unit:0x000000278ca304e75b3b01ee01306168
10:26:24:WU02:FS01:Connecting to 140.163.4.231:8080
10:26:24:WU02:FS01:Server responded WORK_ACK (400)
10:26:24:WU02:FS01:Cleaning up
10:26:25:WU01:FS01:Connecting to 65.254.110.245:8080
10:26:25:WU01:FS01:Assigned to work server 128.252.203.4
10:26:25:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.4
10:26:25:WU01:FS01:Connecting to 128.252.203.4:8080
10:26:25:WU01:FS01:Downloading 11.65MiB
10:26:31:WU01:FS01:Download 37.01%
10:26:37:WU01:FS01:Download 79.37%
10:26:39:WU01:FS01:Download complete
10:26:39:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13816 run:0 clone:1495 gen:47 core:0x21 unit:0x0000003480fccb045b3a656bf17681cb
10:26:40:WU01:FS01:Starting
10:26:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:26:40:WU01:FS01:Started FahCore on PID 29256
10:26:40:WU01:FS01:Core PID:13000
10:26:40:WU01:FS01:FahCore 0x21 started
10:26:40:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:26:40:WU01:FS01:Starting
10:26:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:26:40:WU01:FS01:Started FahCore on PID 19672
10:26:40:WU01:FS01:Core PID:21628
10:26:40:WU01:FS01:FahCore 0x21 started
10:26:41:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:27:40:WU01:FS01:Starting
10:27:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:27:40:WU01:FS01:Started FahCore on PID 22016
10:27:40:WU01:FS01:Core PID:17212
10:27:40:WU01:FS01:FahCore 0x21 started
10:27:41:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:28:40:WU01:FS01:Starting
10:28:40:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:28:40:WU01:FS01:Started FahCore on PID 10128
10:28:40:WU01:FS01:Core PID:13980
10:28:40:WU01:FS01:FahCore 0x21 started
10:28:41:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:29:41:WU01:FS01:Starting
10:29:41:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:29:41:WU01:FS01:Started FahCore on PID 28856
10:29:41:WU01:FS01:Core PID:9692
10:29:41:WU01:FS01:FahCore 0x21 started
10:29:41:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:29:41:WARNING:WU01:FS01:Too many errors, failing
10:29:41:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:13816 run:0 clone:1495 gen:47 core:0x21 unit:0x0000003480fccb045b3a656bf17681cb
10:29:41:WU01:FS01:Connecting to 128.252.203.4:8080
10:29:41:WU01:FS01:Server responded WORK_ACK (400)
10:29:41:WU01:FS01:Cleaning up
10:29:42:WU02:FS01:Connecting to 65.254.110.245:8080
10:29:42:WU02:FS01:Assigned to work server 140.163.4.231
10:29:42:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
10:29:42:WU02:FS01:Connecting to 140.163.4.231:8080
10:29:42:WU02:FS01:Downloading 16.53MiB
10:29:48:WU02:FS01:Download 25.33%
10:29:54:WU02:FS01:Download 55.57%
10:30:00:WU02:FS01:Download 85.81%
10:30:02:WU02:FS01:Download complete
10:30:02:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11713 run:18 clone:291 gen:138 core:0x21 unit:0x000000bd8ca304e75adf7aaed4ecbf5e
10:30:03:WU02:FS01:Starting
10:30:03:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:30:03:WU02:FS01:Started FahCore on PID 24580
10:30:03:WU02:FS01:Core PID:22940
10:30:03:WU02:FS01:FahCore 0x21 started
10:30:03:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:30:03:WU02:FS01:Starting
10:30:03:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:30:03:WU02:FS01:Started FahCore on PID 20584
10:30:03:WU02:FS01:Core PID:25332
10:30:03:WU02:FS01:FahCore 0x21 started
10:30:04:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:31:03:WU02:FS01:Starting
10:31:03:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:31:03:WU02:FS01:Started FahCore on PID 2592
10:31:03:WU02:FS01:Core PID:3572
10:31:03:WU02:FS01:FahCore 0x21 started
10:31:04:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:32:03:WU02:FS01:Starting
10:32:03:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:32:03:WU02:FS01:Started FahCore on PID 11140
10:32:03:WU02:FS01:Core PID:21636
10:32:03:WU02:FS01:FahCore 0x21 started
10:32:04:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:33:04:WU02:FS01:Starting
10:33:04:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:33:04:WU02:FS01:Started FahCore on PID 21372
10:33:04:WU02:FS01:Core PID:18608
10:33:04:WU02:FS01:FahCore 0x21 started
10:33:04:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:33:04:WARNING:WU02:FS01:Too many errors, failing
10:33:04:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11713 run:18 clone:291 gen:138 core:0x21 unit:0x000000bd8ca304e75adf7aaed4ecbf5e
10:33:04:WU02:FS01:Connecting to 140.163.4.231:8080
10:33:04:WU02:FS01:Server responded WORK_ACK (400)
10:33:05:WU02:FS01:Cleaning up
10:33:05:WU01:FS01:Connecting to 65.254.110.245:8080
10:33:05:WU01:FS01:Assigned to work server 128.252.203.4
10:33:05:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.4
10:33:05:WU01:FS01:Connecting to 128.252.203.4:8080
10:33:05:WU01:FS01:Downloading 11.65MiB
10:33:11:WU01:FS01:Download 37.01%
10:33:17:WU01:FS01:Download 79.37%
10:33:20:WU01:FS01:Download complete
10:33:20:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13816 run:0 clone:510 gen:60 core:0x21 unit:0x0000004480fccb045b36d01b34cdcce2
10:33:20:WU01:FS01:Starting
10:33:20:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:33:20:WU01:FS01:Started FahCore on PID 25572
10:33:20:WU01:FS01:Core PID:24916
10:33:20:WU01:FS01:FahCore 0x21 started
10:33:20:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:33:21:WU01:FS01:Starting
10:33:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:33:21:WU01:FS01:Started FahCore on PID 19512
10:33:21:WU01:FS01:Core PID:12424
10:33:21:WU01:FS01:FahCore 0x21 started
10:33:21:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:34:21:WU01:FS01:Starting
10:34:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:34:21:WU01:FS01:Started FahCore on PID 27284
10:34:21:WU01:FS01:Core PID:1256
10:34:21:WU01:FS01:FahCore 0x21 started
10:34:21:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:35:21:WU01:FS01:Starting
10:35:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:35:21:WU01:FS01:Started FahCore on PID 21748
10:35:21:WU01:FS01:Core PID:26388
10:35:21:WU01:FS01:FahCore 0x21 started
10:35:21:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:36:21:WU01:FS01:Starting
10:36:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:36:21:WU01:FS01:Started FahCore on PID 11220
10:36:21:WU01:FS01:Core PID:6392
10:36:21:WU01:FS01:FahCore 0x21 started
10:36:21:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:36:21:WARNING:WU01:FS01:Too many errors, failing
10:36:21:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:13816 run:0 clone:510 gen:60 core:0x21 unit:0x0000004480fccb045b36d01b34cdcce2
10:36:21:WU01:FS01:Connecting to 128.252.203.4:8080
10:36:22:WU01:FS01:Server responded WORK_ACK (400)
10:36:22:WU01:FS01:Cleaning up
10:36:22:WU02:FS01:Connecting to 65.254.110.245:8080
10:36:22:WU02:FS01:Assigned to work server 128.252.203.4
10:36:22:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.4
10:36:22:WU02:FS01:Connecting to 128.252.203.4:8080
10:36:23:WU02:FS01:Downloading 11.66MiB
10:36:29:WU02:FS01:Download 40.75%
10:36:35:WU02:FS01:Download 83.64%
10:36:37:WU02:FS01:Download complete
10:36:37:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13816 run:0 clone:2467 gen:4 core:0x21 unit:0x0000000580fccb045b3f7910a8c65bb6
10:36:37:WU02:FS01:Starting
10:36:37:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:36:37:WU02:FS01:Started FahCore on PID 11404
10:36:37:WU02:FS01:Core PID:14944
10:36:37:WU02:FS01:FahCore 0x21 started
10:36:38:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:36:38:WU02:FS01:Starting
10:36:38:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:36:38:WU02:FS01:Started FahCore on PID 20248
10:36:38:WU02:FS01:Core PID:4672
10:36:38:WU02:FS01:FahCore 0x21 started
10:36:38:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:37:38:WU02:FS01:Starting
10:37:38:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:37:38:WU02:FS01:Started FahCore on PID 1184
10:37:38:WU02:FS01:Core PID:24748
10:37:38:WU02:FS01:FahCore 0x21 started
10:37:38:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:38:38:WU02:FS01:Starting
10:38:38:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:38:38:WU02:FS01:Started FahCore on PID 26884
10:38:38:WU02:FS01:Core PID:560
10:38:38:WU02:FS01:FahCore 0x21 started
10:38:38:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:39:38:WU02:FS01:Starting
10:39:38:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:39:38:WU02:FS01:Started FahCore on PID 29020
10:39:38:WU02:FS01:Core PID:27724
10:39:38:WU02:FS01:FahCore 0x21 started
10:39:39:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:39:39:WARNING:WU02:FS01:Too many errors, failing
10:39:39:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:13816 run:0 clone:2467 gen:4 core:0x21 unit:0x0000000580fccb045b3f7910a8c65bb6
10:39:39:WU02:FS01:Connecting to 128.252.203.4:8080
10:39:39:WU02:FS01:Server responded WORK_ACK (400)
10:39:39:WU02:FS01:Cleaning up
10:39:39:WU01:FS01:Connecting to 65.254.110.245:8080
10:39:40:WU01:FS01:Assigned to work server 140.163.4.231
10:39:40:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
10:39:40:WU01:FS01:Connecting to 140.163.4.231:8080
10:39:40:WU01:FS01:Downloading 16.51MiB
10:39:46:WU01:FS01:Download 28.01%
10:39:52:WU01:FS01:Download 58.30%
10:39:58:WU01:FS01:Download 88.59%
10:40:00:WU01:FS01:Download complete
10:40:00:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:25 clone:133 gen:221 core:0x21 unit:0x000001228ca304e75a5cfe213529e43a
10:41:16:WU01:FS01:Downloading core from http://cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah
10:41:16:WU01:FS01:Connecting to cores.foldingathome.org:80
10:41:16:WU01:FS01:FahCore 21: Downloading 3.47MiB
10:41:20:WU01:FS01:FahCore 21: Download complete
10:41:20:WU01:FS01:Valid core signature
10:41:20:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
10:41:20:WU01:FS01:Starting
10:41:20:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:41:20:WU01:FS01:Started FahCore on PID 25748
10:41:20:WU01:FS01:Core PID:28900
10:41:20:WU01:FS01:FahCore 0x21 started
10:41:21:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:41:21:WU01:FS01:Starting
10:41:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:41:21:WU01:FS01:Started FahCore on PID 19868
10:41:21:WU01:FS01:Core PID:26108
10:41:21:WU01:FS01:FahCore 0x21 started
10:41:22:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:42:21:WU01:FS01:Starting
10:42:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:42:21:WU01:FS01:Started FahCore on PID 9608
10:42:21:WU01:FS01:Core PID:29416
10:42:21:WU01:FS01:FahCore 0x21 started
10:42:22:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:43:21:WU01:FS01:Starting
10:43:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:43:21:WU01:FS01:Started FahCore on PID 10524
10:43:21:WU01:FS01:Core PID:23388
10:43:21:WU01:FS01:FahCore 0x21 started
10:43:22:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:44:21:WU01:FS01:Starting
10:44:21:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:44:21:WU01:FS01:Started FahCore on PID 17328
10:44:21:WU01:FS01:Core PID:20120
10:44:21:WU01:FS01:FahCore 0x21 started
10:44:22:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:44:22:WARNING:WU01:FS01:Too many errors, failing
10:44:22:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:25 clone:133 gen:221 core:0x21 unit:0x000001228ca304e75a5cfe213529e43a
10:44:22:WU01:FS01:Connecting to 140.163.4.231:8080
10:44:22:WU01:FS01:Server responded WORK_ACK (400)
10:44:22:WU01:FS01:Cleaning up
10:44:23:WU02:FS01:Connecting to 65.254.110.245:8080
10:44:23:WU02:FS01:Assigned to work server 140.163.4.231
10:44:23:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
10:44:23:WU02:FS01:Connecting to 140.163.4.231:8080
10:44:23:WU02:FS01:Downloading 16.51MiB
10:44:29:WU02:FS01:Download 26.12%
10:44:35:WU02:FS01:Download 55.27%
10:44:41:WU02:FS01:Download 85.55%
10:44:43:WU02:FS01:Download complete
10:44:43:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11713 run:1 clone:265 gen:130 core:0x21 unit:0x000000b68ca304e75adf720aaf2a639f
10:44:43:WU02:FS01:Starting
10:44:43:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:44:43:WU02:FS01:Started FahCore on PID 27680
10:44:43:WU02:FS01:Core PID:27452
10:44:44:WU02:FS01:FahCore 0x21 started
10:44:44:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:44:44:WU02:FS01:Starting
10:44:44:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:44:44:WU02:FS01:Started FahCore on PID 9820
10:44:44:WU02:FS01:Core PID:26308
10:44:44:WU02:FS01:FahCore 0x21 started
10:44:45:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:45:44:WU02:FS01:Starting
10:45:44:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:45:44:WU02:FS01:Started FahCore on PID 19864
10:45:44:WU02:FS01:Core PID:21036
10:45:44:WU02:FS01:FahCore 0x21 started
10:45:45:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:46:44:WU02:FS01:Starting
10:46:44:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:46:44:WU02:FS01:Started FahCore on PID 26068
10:46:44:WU02:FS01:Core PID:2096
10:46:44:WU02:FS01:FahCore 0x21 started
10:46:45:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:47:44:WU02:FS01:Starting
10:47:44:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:47:44:WU02:FS01:Started FahCore on PID 27704
10:47:44:WU02:FS01:Core PID:20860
10:47:44:WU02:FS01:FahCore 0x21 started
10:47:45:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:47:45:WARNING:WU02:FS01:Too many errors, failing
10:47:45:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11713 run:1 clone:265 gen:130 core:0x21 unit:0x000000b68ca304e75adf720aaf2a639f
10:47:45:WU02:FS01:Connecting to 140.163.4.231:8080
10:47:45:WU02:FS01:Server responded WORK_ACK (400)
10:47:45:WU02:FS01:Cleaning up
10:47:46:WU01:FS01:Connecting to 65.254.110.245:8080
10:47:46:WU01:FS01:Assigned to work server 140.163.4.231
10:47:46:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
10:47:46:WU01:FS01:Connecting to 140.163.4.231:8080
10:47:47:WU01:FS01:Downloading 16.53MiB
10:47:53:WU01:FS01:Download 26.84%
10:47:59:WU01:FS01:Download 46.13%
10:48:05:WU01:FS01:Download 76.00%
10:48:09:WU01:FS01:Download complete
10:48:09:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:10 clone:440 gen:97 core:0x21 unit:0x000000888ca304e75adf76db34a1e9c2
10:48:09:WU01:FS01:Starting
10:48:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:48:10:WU01:FS01:Started FahCore on PID 26700
10:48:10:WU01:FS01:Core PID:29124
10:48:10:WU01:FS01:FahCore 0x21 started
10:48:10:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:48:10:WU01:FS01:Starting
10:48:10:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:48:10:WU01:FS01:Started FahCore on PID 26224
10:48:10:WU01:FS01:Core PID:16332
10:48:10:WU01:FS01:FahCore 0x21 started
10:48:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:49:10:WU01:FS01:Starting
10:49:10:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:49:10:WU01:FS01:Started FahCore on PID 26364
10:49:10:WU01:FS01:Core PID:23456
10:49:10:WU01:FS01:FahCore 0x21 started
10:49:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:50:10:WU01:FS01:Starting
10:50:10:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:50:10:WU01:FS01:Started FahCore on PID 9980
10:50:10:WU01:FS01:Core PID:23900
10:50:10:WU01:FS01:FahCore 0x21 started
10:50:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:51:10:WU01:FS01:Starting
10:51:10:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:51:10:WU01:FS01:Started FahCore on PID 20944
10:51:10:WU01:FS01:Core PID:9488
10:51:10:WU01:FS01:FahCore 0x21 started
10:51:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:51:11:WARNING:WU01:FS01:Too many errors, failing
10:51:11:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:10 clone:440 gen:97 core:0x21 unit:0x000000888ca304e75adf76db34a1e9c2
10:51:11:WU01:FS01:Connecting to 140.163.4.231:8080
10:51:11:WU01:FS01:Server responded WORK_ACK (400)
10:51:11:WU01:FS01:Cleaning up
10:51:12:WU02:FS01:Connecting to 65.254.110.245:8080
10:51:12:WU02:FS01:Assigned to work server 128.252.203.4
10:51:12:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 128.252.203.4
10:51:12:WU02:FS01:Connecting to 128.252.203.4:8080
10:51:12:WU02:FS01:Downloading 11.65MiB
10:51:18:WU02:FS01:Download 37.01%
10:51:24:WU02:FS01:Download 79.39%
10:51:26:WU02:FS01:Download complete
10:51:26:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13816 run:0 clone:1947 gen:25 core:0x21 unit:0x0000001e80fccb045b3a656e112090f4
10:51:26:WU02:FS01:Starting
10:51:26:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:51:27:WU02:FS01:Started FahCore on PID 27808
10:51:27:WU02:FS01:Core PID:20996
10:51:27:WU02:FS01:FahCore 0x21 started
10:51:27:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:51:27:WU02:FS01:Starting
10:51:27:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:51:27:WU02:FS01:Started FahCore on PID 12416
10:51:27:WU02:FS01:Core PID:23308
10:51:27:WU02:FS01:FahCore 0x21 started
10:51:28:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:52:27:WU02:FS01:Starting
10:52:27:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:52:27:WU02:FS01:Started FahCore on PID 18644
10:52:27:WU02:FS01:Core PID:29248
10:52:27:WU02:FS01:FahCore 0x21 started
10:52:28:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:53:27:WU02:FS01:Starting
10:53:27:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:53:27:WU02:FS01:Started FahCore on PID 23444
10:53:27:WU02:FS01:Core PID:22128
10:53:27:WU02:FS01:FahCore 0x21 started
10:53:28:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:54:27:WU02:FS01:Starting
10:54:27:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:54:28:WU02:FS01:Started FahCore on PID 9060
10:54:28:WU02:FS01:Core PID:29336
10:54:28:WU02:FS01:FahCore 0x21 started
10:54:28:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:54:28:WARNING:WU02:FS01:Too many errors, failing
10:54:28:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:13816 run:0 clone:1947 gen:25 core:0x21 unit:0x0000001e80fccb045b3a656e112090f4
10:54:28:WU02:FS01:Connecting to 128.252.203.4:8080
10:54:28:WU02:FS01:Server responded WORK_ACK (400)
10:54:28:WU02:FS01:Cleaning up
10:54:29:WU01:FS01:Connecting to 65.254.110.245:8080
10:54:29:WU01:FS01:Assigned to work server 140.163.4.231
10:54:29:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
10:54:29:WU01:FS01:Connecting to 140.163.4.231:8080
10:54:29:WU01:FS01:Downloading 5.42MiB
10:54:35:WU01:FS01:Download 79.62%
10:54:36:WU01:FS01:Download complete
10:54:36:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11715 run:2 clone:364 gen:46 core:0x21 unit:0x000000388ca304e75b29c36d8cc4822b
10:54:36:WU01:FS01:Starting
10:54:36:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:54:36:WU01:FS01:Started FahCore on PID 28456
10:54:36:WU01:FS01:Core PID:6668
10:54:36:WU01:FS01:FahCore 0x21 started
10:54:37:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:54:37:WU01:FS01:Starting
10:54:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:54:37:WU01:FS01:Started FahCore on PID 8776
10:54:37:WU01:FS01:Core PID:15980
10:54:37:WU01:FS01:FahCore 0x21 started
10:54:37:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:55:37:WU01:FS01:Starting
10:55:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:55:37:WU01:FS01:Started FahCore on PID 10428
10:55:37:WU01:FS01:Core PID:16560
10:55:37:WU01:FS01:FahCore 0x21 started
10:55:37:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:56:37:WU01:FS01:Starting
10:56:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:56:37:WU01:FS01:Started FahCore on PID 11600
10:56:37:WU01:FS01:Core PID:15368
10:56:37:WU01:FS01:FahCore 0x21 started
10:56:37:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:57:37:WU01:FS01:Starting
10:57:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 2132 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:57:37:WU01:FS01:Started FahCore on PID 27560
10:57:37:WU01:FS01:Core PID:16332
10:57:37:WU01:FS01:FahCore 0x21 started
10:57:38:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
10:57:38:WARNING:WU01:FS01:Too many errors, failing
10:57:38:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11715 run:2 clone:364 gen:46 core:0x21 unit:0x000000388ca304e75b29c36d8cc4822b
10:57:38:WU01:FS01:Connecting to 140.163.4.231:8080
10:57:38:WU01:FS01:Server responded WORK_ACK (400)
10:57:38:WU01:FS01:Cleaning up

This is what happened after dumping WU 01and starting to fold again.

Code: Select all

[color=#804040]Failed GPU WU 18july18  2nd after Dumping  WU 01
14:49:51:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:49:52:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:50:52:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:51:52:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:52:52:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:52:52:WARNING:WU01:FS01:Too many errors, failing
14:53:08:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:53:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:54:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:55:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:56:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:56:09:WARNING:WU01:FS01:Too many errors, failing
15:02:47:ERROR:WU02:FS01:Exception: Transfer failed
15:02:47:ERROR:WU02:FS01:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
15:02:47:ERROR:WU02:FS01:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
15:02:47:WARNING:WU02:FS01:Exception: Failed to find any IP addresses for assignment servers
15:02:47:ERROR:WU02:FS01:Exception: Could not get an assignment
15:04:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:04:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:05:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:06:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:07:47:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:07:47:WARNING:WU02:FS01:Too many errors, failing
15:08:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:08:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:09:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:10:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:10:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:10:WARNING:WU01:FS01:Too many errors, failing
15:11:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:12:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:13:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:14:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:14:33:WARNING:WU02:FS01:Too many errors, failing
15:16:14:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
15:16:15:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:16:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:17:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:18:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:16:WARNING:WU01:FS01:Too many errors, failing
15:19:38:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:20:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:21:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:39:WARNING:WU01:FS01:Too many errors, failing
15:22:47:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:23:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:24:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:25:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:25:48:WARNING:WU02:FS01:Too many errors, failing
15:26:10:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:26:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:27:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:28:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:12:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:12:WARNING:WU01:FS01:Too many errors, failing
15:29:20:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:30:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:31:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:32:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:32:21:WARNING:WU02:FS01:Too many errors, failing
15:34:03:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
15:34:03:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:34:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:35:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:36:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:04:WARNING:WU01:FS01:Too many errors, failing
15:37:09:ERROR:WU02:FS01:Exception: Server did not assign work unit
15:37:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:38:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:39:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:40:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:40:33:WARNING:WU02:FS01:Too many errors, failing
[/color]

Post by **bruce** » Wed Jul 18, 2018 5:37 pm

1) There's a good reason for the error WARNING:WU01:FS01:Too many errors, failing. It shuts down a slot when a GPU is doing what you're doing, and re-enabling it so that you can have another serices of WUs failed is an excellant way to do damage to FoldingAtHome. You're intentionally dumping more an more WUs when something is clearly wrong with your system. Stop doing that.

2) I checked a few of the WUs that you dumped and found that they were reassigned and someone else successfully completed them, so there's nothing wrong with the WUs themselves -- it's YOUR SYSTEM.

3) As far a diagnosing what's wrong with your system, that's impossible to do from the information you posted.

> Is the GPU overheating?
> Is the GPU overclocked?
> Has the GPU failed?
> Are the drivers corrupted
> Is some other compoonint of your system causing the GPU to receive incorrect data?
> Is this Windows, Linux, or MAC OS-X?
> Have you folded with more than one UserID on your system?
> ??? Something else?

Please post the log showing what happened to the WU you're calling "18july18 2nd " since I need PRCG to search for it.

What steps have you done to diagnose your hardware?

rfharbin wrote:I am hoping for a cure from somebody as this is taking a lot of my "extra" time.

I'm happy to help when I can, but it takes a lot of my "extra" time to. I'm a volunteer just like you, but when you don't provide enough information, I have to start guessing and that's not always useful.

Where is FAH's system description from the top of the log? Where is the segment showing what was happening when the FIRST errors happened?

See my sig block below.

rfharbin · Post by **rfharbin** » Thu Jul 19, 2018 3:32 pm

Thanks for your answer. There is enough info for me to investigate I will be doing this a step at a time. I have had this problem for over a year now and have “fixed” the problem by uninstalling and reinstalling F@H. On September 13, 2017 I decided I was going to find a way to rid of a bad or failed GPU WU as unit and reinstalling F@H was getting to me. When I have a bad or failed CPU WU it is easy to get folding again. (very rarely, I do not remember the last failure of a CPU WU.) I posted a question and was told to use "FAHClient --dump NN”. I first dumped a GPU WU on September 13, 2017. The day before I posted this thread was the second time I used this command.
Before I posted I uninstalled F@H with Revo Uninstall Pro which does a very good job cleaning a program out of the computer and then spent over an hour looking where I know the Client parts can be and searching to be sure all traces of F@H were gone. I did this because of how often I have had trouble the last several weeks. I then deleted my copy of F@H 7.4.4 from my Downloads folder and got a copy of version 7.5.1 and installed it. Less than a week later I noticed that my CPU was not running up in the ninety percent range as it normally does. I copied the log labeled “Failed GPU WU 18July18” and saved it, thus I cannot understand where dumping the GPU WU had any effect on the failure as I had not dumped a GPU WU and had uninstalled and reinstalled multiple times. Next I dumped the GPU failed WU and restarted F@H so the Log files after this restart, “Failed GPU WU 18july18 2nd after Dumping WU 01” may show an effect of Dumping the GPU WU.

The Question about what is my computer as I updated some rather old info in my profile.

Gigabyte GA-Z170X-Gaming 7 MoBo
Cooler Master HAF series RC-932-KKN5 Case
GTX-1060 Graphics card
Cooler Master RS 750 Power Supply
Intel Core i7- 6700K CPU 8M 4.0 Ghz CPU
1 Tetra Seagate ST1000DX001-1NS162
GSKILL Ripjaws 16GB DDR4 Memory
Lite-on DVD/CD Burner
Cooler Master 120mm CPU Cooler
SAMSUNG 850 PRO 2.5" 256GB SSD
Windows 10 Home

> Is the GPU overheating?
No way that this is likely. I previously folded with a GTX 970 card and the GTX 10XX cards are very much cooler. Also my case has three 230 mm fans, on the top, left side, and frount plus a 140 mm fan on the back.
> Has the GPU failed?
Everything about it looks good. It was new this spring.
> Are the drivers corrupted?
Not very likely either as I use Nvidia GeForce Experence. This program downloads new drivers when they are available and tells you that they are ready to install. Also it keeps a copy of the last drivers installed. I reinstalled the drivers a couple weeks ago just in case they could be corrupted and causing my problem.
> Is some other component of your system causing the GPU to receive incorrect data?
I have no idea how to check this. I do not remember having trouble downloading files or programs ever.
> Is this Windows, Linux, or MAC OS-X?
Win10 Home.
> Have you folded with more than one UserID on your system?
Never, ever, also this is a new install after being sure that there wasn’t any former F@H files, folders, and removing registry entries.

What steps have you done to diagnose your hardware?
I have not checked my memory in quite a while, not since last spring or earlier when I was wondering if it was causing me problems. I will run the test again. Is there anything else I can check?

Where is FAH's system description from the top of the log?
There was none but there is a tab with that info on the FAHControl but there is no way I would likely be able to manually copy all that stuff without errors. I tried to attach an image when I started this thread. I will try to learn how again. If there is a specific item you need please ask and I will do my best to manually copy it.

Where is the segment showing what was happening when the FIRST errors happened?
You have me on that query. This is a new install of F@H and I uninstalled and reinstalled F@H a number of times this last year because of a GPU failure. I do not remember what date it first failed. I do have the impression that the failures are happening more frequently with Client 7.5.1 than under Client 7.4.4 even though I updated to 7.5.1 just the day before I posted this thread.

The latest log.
Please notice that the last posted log is included in the log posted below and that the GPU has failed again.

Code: Select all

14:54:09:WU01:FS01:Started FahCore on PID 25224
14:54:09:WU01:FS01:Core PID:19956
14:54:09:WU01:FS01:FahCore 0x21 started
14:54:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:55:09:WU01:FS01:Starting
14:55:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
14:55:09:WU01:FS01:Started FahCore on PID 16248
14:55:09:WU01:FS01:Core PID:18208
14:55:09:WU01:FS01:FahCore 0x21 started
14:55:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:56:09:WU01:FS01:Starting
14:56:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
14:56:09:WU01:FS01:Started FahCore on PID 18060
14:56:09:WU01:FS01:Core PID:11628
14:56:09:WU01:FS01:FahCore 0x21 started
14:56:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
14:56:09:WARNING:WU01:FS01:Too many errors, failing
14:56:09:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:13816 run:0 clone:1188 gen:16 core:0x21 unit:0x0000001380fccb045b3a656947a13b76
14:56:09:WU01:FS01:Connecting to 128.252.203.4:8080
14:56:10:WU02:FS01:Connecting to 65.254.110.245:8080
14:56:10:WU01:FS01:Server responded WORK_ACK (400)
14:56:10:WU01:FS01:Cleaning up
14:56:10:WU02:FS01:Assigned to work server 140.163.4.231
14:56:10:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
14:56:10:WU02:FS01:Connecting to 140.163.4.231:8080
14:56:11:WU02:FS01:Downloading 16.52MiB
14:57:40:FS01:Paused
14:57:50:FS01:Unpaused
14:58:01:FS01:Paused
14:58:08:FS01:Unpaused
15:02:47:WU02:FS01:Download 6.43%
15:02:47:ERROR:WU02:FS01:Exception: Transfer failed
15:02:47:ERROR:WU02:FS01:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
15:02:47:ERROR:WU02:FS01:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
15:02:47:WARNING:WU02:FS01:Exception: Failed to find any IP addresses for assignment servers
15:02:47:ERROR:WU02:FS01:Exception: Could not get an assignment
15:04:24:WU02:FS01:Connecting to 65.254.110.245:8080
15:04:24:WU02:FS01:Assigned to work server 140.163.4.231
15:04:24:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:04:24:WU02:FS01:Connecting to 140.163.4.231:8080
15:04:25:WU02:FS01:Downloading 16.53MiB
15:04:31:WU02:FS01:Download 28.37%
15:04:37:WU02:FS01:Download 58.62%
15:04:43:WU02:FS01:Download 88.88%
15:04:45:WU02:FS01:Download complete
15:04:45:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11713 run:27 clone:135 gen:91 core:0x21 unit:0x000000828ca304e75ae63520b374d9b3
15:04:45:WU02:FS01:Starting
15:04:45:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:04:45:WU02:FS01:Started FahCore on PID 26568
15:04:45:WU02:FS01:Core PID:10312
15:04:45:WU02:FS01:FahCore 0x21 started
15:04:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:04:46:WU02:FS01:Starting
15:04:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:04:46:WU02:FS01:Started FahCore on PID 27664
15:04:46:WU02:FS01:Core PID:7844
15:04:46:WU02:FS01:FahCore 0x21 started
15:04:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:05:46:WU02:FS01:Starting
15:05:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:05:46:WU02:FS01:Started FahCore on PID 13332
15:05:46:WU02:FS01:Core PID:2180
15:05:46:WU02:FS01:FahCore 0x21 started
15:05:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:06:46:WU02:FS01:Starting
15:06:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:06:46:WU02:FS01:Started FahCore on PID 22112
15:06:46:WU02:FS01:Core PID:22592
15:06:46:WU02:FS01:FahCore 0x21 started
15:06:46:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:07:46:WU02:FS01:Starting
15:07:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:07:46:WU02:FS01:Started FahCore on PID 10300
15:07:46:WU02:FS01:Core PID:24772
15:07:46:WU02:FS01:FahCore 0x21 started
15:07:47:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:07:47:WARNING:WU02:FS01:Too many errors, failing
15:07:47:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11713 run:27 clone:135 gen:91 core:0x21 unit:0x000000828ca304e75ae63520b374d9b3
15:07:47:WU02:FS01:Connecting to 140.163.4.231:8080
15:07:47:WU02:FS01:Server responded WORK_ACK (400)
15:07:47:WU02:FS01:Cleaning up
15:07:47:WU01:FS01:Connecting to 65.254.110.245:8080
15:07:47:WU01:FS01:Assigned to work server 140.163.4.231
15:07:47:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:07:47:WU01:FS01:Connecting to 140.163.4.231:8080
15:07:48:WU01:FS01:Downloading 16.52MiB
15:07:54:WU01:FS01:Download 27.99%
15:08:00:WU01:FS01:Download 57.88%
15:08:06:WU01:FS01:Download 88.15%
15:08:08:WU01:FS01:Download complete
15:08:08:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:16 clone:1 gen:295 core:0x21 unit:0x0000015e8ca304e759cd241f65f90dbb
15:08:08:WU01:FS01:Starting
15:08:08:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:08:08:WU01:FS01:Started FahCore on PID 14328
15:08:08:WU01:FS01:Core PID:27944
15:08:08:WU01:FS01:FahCore 0x21 started
15:08:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:08:09:WU01:FS01:Starting
15:08:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:08:09:WU01:FS01:Started FahCore on PID 27268
15:08:09:WU01:FS01:Core PID:20052
15:08:09:WU01:FS01:FahCore 0x21 started
15:08:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:09:09:WU01:FS01:Starting
15:09:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:09:09:WU01:FS01:Started FahCore on PID 11980
15:09:09:WU01:FS01:Core PID:6980
15:09:09:WU01:FS01:FahCore 0x21 started
15:09:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:10:09:WU01:FS01:Starting
15:10:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:10:09:WU01:FS01:Started FahCore on PID 15220
15:10:09:WU01:FS01:Core PID:15016
15:10:09:WU01:FS01:FahCore 0x21 started
15:10:09:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:09:WU01:FS01:Starting
15:11:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:11:09:WU01:FS01:Started FahCore on PID 17916
15:11:09:WU01:FS01:Core PID:7632
15:11:09:WU01:FS01:FahCore 0x21 started
15:11:10:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:10:WARNING:WU01:FS01:Too many errors, failing
15:11:10:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:16 clone:1 gen:295 core:0x21 unit:0x0000015e8ca304e759cd241f65f90dbb
15:11:10:WU01:FS01:Connecting to 140.163.4.231:8080
15:11:10:WU01:FS01:Server responded WORK_ACK (400)
15:11:10:WU02:FS01:Connecting to 65.254.110.245:8080
15:11:10:WU01:FS01:Cleaning up
15:11:10:WU02:FS01:Assigned to work server 140.163.4.231
15:11:10:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:11:10:WU02:FS01:Connecting to 140.163.4.231:8080
15:11:11:WU02:FS01:Downloading 16.52MiB
15:11:17:WU02:FS01:Download 28.37%
15:11:23:WU02:FS01:Download 58.63%
15:11:29:WU02:FS01:Download 88.51%
15:11:31:WU02:FS01:Download complete
15:11:31:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11713 run:2 clone:169 gen:62 core:0x21 unit:0x000000528ca304e75adf7272825f1cc2
15:11:31:WU02:FS01:Starting
15:11:31:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:11:31:WU02:FS01:Started FahCore on PID 18620
15:11:31:WU02:FS01:Core PID:27812
15:11:31:WU02:FS01:FahCore 0x21 started
15:11:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:11:32:WU02:FS01:Starting
15:11:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:11:32:WU02:FS01:Started FahCore on PID 9776
15:11:32:WU02:FS01:Core PID:10052
15:11:32:WU02:FS01:FahCore 0x21 started
15:11:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:12:32:WU02:FS01:Starting
15:12:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:12:32:WU02:FS01:Started FahCore on PID 27284
15:12:32:WU02:FS01:Core PID:10300
15:12:32:WU02:FS01:FahCore 0x21 started
15:12:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:13:32:WU02:FS01:Starting
15:13:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:13:32:WU02:FS01:Started FahCore on PID 22016
15:13:32:WU02:FS01:Core PID:19268
15:13:32:WU02:FS01:FahCore 0x21 started
15:13:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:14:32:WU02:FS01:Starting
15:14:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:14:32:WU02:FS01:Started FahCore on PID 16344
15:14:32:WU02:FS01:Core PID:17720
15:14:32:WU02:FS01:FahCore 0x21 started
15:14:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:14:33:WARNING:WU02:FS01:Too many errors, failing
15:14:33:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11713 run:2 clone:169 gen:62 core:0x21 unit:0x000000528ca304e75adf7272825f1cc2
15:14:33:WU02:FS01:Connecting to 140.163.4.231:8080
15:14:33:WU02:FS01:Server responded WORK_ACK (400)
15:14:33:WU02:FS01:Cleaning up
15:14:33:WU01:FS01:Connecting to 65.254.110.245:8080
15:14:34:WU01:FS01:Assigned to work server 140.163.4.231
15:14:34:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:14:34:WU01:FS01:Connecting to 140.163.4.231:8080
15:14:34:WU01:FS01:Downloading 5.42MiB
15:14:40:WU01:FS01:Download 83.08%
15:14:41:WU01:FS01:Download complete
15:14:41:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11715 run:1 clone:1355 gen:44 core:0x21 unit:0x000000388ca304e75b3c20fa028dd9d9
15:16:10:WU01:FS01:Downloading core from http://cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah
15:16:10:WU01:FS01:Connecting to cores.foldingathome.org:80
15:16:10:WU01:FS01:FahCore 21: Downloading 3.47MiB
15:16:14:WU01:FS01:FahCore 21: Download complete
15:16:14:WU01:FS01:Valid core signature
15:16:14:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
15:16:14:WU01:FS01:Starting
15:16:14:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:16:14:WU01:FS01:Started FahCore on PID 28584
15:16:14:WU01:FS01:Core PID:19072
15:16:14:WU01:FS01:FahCore 0x21 started
15:16:15:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:16:15:WU01:FS01:Starting
15:16:15:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:16:15:WU01:FS01:Started FahCore on PID 22884
15:16:15:WU01:FS01:Core PID:17424
15:16:15:WU01:FS01:FahCore 0x21 started
15:16:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:17:15:WU01:FS01:Starting
15:17:15:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:17:15:WU01:FS01:Started FahCore on PID 26280
15:17:15:WU01:FS01:Core PID:27148
15:17:15:WU01:FS01:FahCore 0x21 started
15:17:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:18:15:WU01:FS01:Starting
15:18:15:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:18:15:WU01:FS01:Started FahCore on PID 23224
15:18:15:WU01:FS01:Core PID:15752
15:18:15:WU01:FS01:FahCore 0x21 started
15:18:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:15:WU01:FS01:Starting
15:19:15:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:19:15:WU01:FS01:Started FahCore on PID 27572
15:19:15:WU01:FS01:Core PID:25832
15:19:15:WU01:FS01:FahCore 0x21 started
15:19:16:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:16:WARNING:WU01:FS01:Too many errors, failing
15:19:16:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11715 run:1 clone:1355 gen:44 core:0x21 unit:0x000000388ca304e75b3c20fa028dd9d9
15:19:16:WU01:FS01:Connecting to 140.163.4.231:8080
15:19:16:WU01:FS01:Server responded WORK_ACK (400)
15:19:16:WU01:FS01:Cleaning up
15:19:17:WU01:FS01:Connecting to 65.254.110.245:8080
15:19:17:WU01:FS01:Assigned to work server 140.163.4.231
15:19:17:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:19:17:WU01:FS01:Connecting to 140.163.4.231:8080
15:19:17:WU01:FS01:Downloading 16.53MiB
15:19:23:WU01:FS01:Download 26.46%
15:19:29:WU01:FS01:Download 56.33%
15:19:35:WU01:FS01:Download 86.57%
15:19:37:WU01:FS01:Download complete
15:19:37:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:10 clone:26 gen:251 core:0x21 unit:0x0000013f8ca304e759d021c11b0f9876
15:19:37:WU01:FS01:Starting
15:19:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:19:37:WU01:FS01:Started FahCore on PID 20860
15:19:37:WU01:FS01:Core PID:10764
15:19:37:WU01:FS01:FahCore 0x21 started
15:19:38:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:19:38:WU01:FS01:Starting
15:19:38:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:19:38:WU01:FS01:Started FahCore on PID 1804
15:19:38:WU01:FS01:Core PID:29076
15:19:38:WU01:FS01:FahCore 0x21 started
15:19:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:20:38:WU01:FS01:Starting
15:20:38:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:20:38:WU01:FS01:Started FahCore on PID 25148
15:20:38:WU01:FS01:Core PID:17728
15:20:38:WU01:FS01:FahCore 0x21 started
15:20:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:21:38:WU01:FS01:Starting
15:21:38:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:21:38:WU01:FS01:Started FahCore on PID 21624
15:21:38:WU01:FS01:Core PID:17372
15:21:38:WU01:FS01:FahCore 0x21 started
15:21:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:38:WU01:FS01:Starting
15:22:38:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:22:38:WU01:FS01:Started FahCore on PID 29340
15:22:38:WU01:FS01:Core PID:10560
15:22:38:WU01:FS01:FahCore 0x21 started
15:22:39:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:39:WARNING:WU01:FS01:Too many errors, failing
15:22:39:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:10 clone:26 gen:251 core:0x21 unit:0x0000013f8ca304e759d021c11b0f9876
15:22:39:WU01:FS01:Connecting to 140.163.4.231:8080
15:22:39:WU01:FS01:Server responded WORK_ACK (400)
15:22:39:WU01:FS01:Cleaning up
15:22:40:WU02:FS01:Connecting to 65.254.110.245:8080
15:22:40:WU02:FS01:Assigned to work server 140.163.4.231
15:22:40:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:22:40:WU02:FS01:Connecting to 140.163.4.231:8080
15:22:40:WU02:FS01:Downloading 5.42MiB
15:22:46:WU02:FS01:Download 79.62%
15:22:47:WU02:FS01:Download complete
15:22:47:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11715 run:2 clone:1563 gen:14 core:0x21 unit:0x000000128ca304e75b3c212228b2fd7d
15:22:47:WU02:FS01:Starting
15:22:47:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:22:47:WU02:FS01:Started FahCore on PID 25044
15:22:47:WU02:FS01:Core PID:1652
15:22:47:WU02:FS01:FahCore 0x21 started
15:22:47:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:22:48:WU02:FS01:Starting
15:22:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:22:48:WU02:FS01:Started FahCore on PID 12496
15:22:48:WU02:FS01:Core PID:12416
15:22:48:WU02:FS01:FahCore 0x21 started
15:22:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:23:48:WU02:FS01:Starting
15:23:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:23:48:WU02:FS01:Started FahCore on PID 28604
15:23:48:WU02:FS01:Core PID:10604
15:23:48:WU02:FS01:FahCore 0x21 started
15:23:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:24:48:WU02:FS01:Starting
15:24:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:24:48:WU02:FS01:Started FahCore on PID 20308
15:24:48:WU02:FS01:Core PID:18720
15:24:48:WU02:FS01:FahCore 0x21 started
15:24:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:25:48:WU02:FS01:Starting
15:25:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:25:48:WU02:FS01:Started FahCore on PID 24440
15:25:48:WU02:FS01:Core PID:13576
15:25:48:WU02:FS01:FahCore 0x21 started
15:25:48:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:25:48:WARNING:WU02:FS01:Too many errors, failing
15:25:48:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11715 run:2 clone:1563 gen:14 core:0x21 unit:0x000000128ca304e75b3c212228b2fd7d
15:25:49:WU02:FS01:Connecting to 140.163.4.231:8080
15:25:49:WU02:FS01:Server responded WORK_ACK (400)
15:25:49:WU02:FS01:Cleaning up
15:25:49:WU01:FS01:Connecting to 65.254.110.245:8080
15:25:49:WU01:FS01:Assigned to work server 140.163.4.231
15:25:49:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:25:49:WU01:FS01:Connecting to 140.163.4.231:8080
15:25:50:WU01:FS01:Downloading 16.53MiB
15:25:56:WU01:FS01:Download 28.36%
15:26:02:WU01:FS01:Download 58.62%
15:26:08:WU01:FS01:Download 88.87%
15:26:10:WU01:FS01:Download complete
15:26:10:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:11 clone:208 gen:180 core:0x21 unit:0x000000f08ca304e75adf770688fabb06
15:26:10:WU01:FS01:Starting
15:26:10:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:26:10:WU01:FS01:Started FahCore on PID 27796
15:26:10:WU01:FS01:Core PID:25920
15:26:10:WU01:FS01:FahCore 0x21 started
15:26:10:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:26:11:WU01:FS01:Starting
15:26:11:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:26:11:WU01:FS01:Started FahCore on PID 29448
15:26:11:WU01:FS01:Core PID:19072
15:26:11:WU01:FS01:FahCore 0x21 started
15:26:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:27:11:WU01:FS01:Starting
15:27:11:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:27:11:WU01:FS01:Started FahCore on PID 17372
15:27:11:WU01:FS01:Core PID:9392
15:27:11:WU01:FS01:FahCore 0x21 started
15:27:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:28:11:WU01:FS01:Starting
15:28:11:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:28:11:WU01:FS01:Started FahCore on PID 25260
15:28:11:WU01:FS01:Core PID:23824
15:28:11:WU01:FS01:FahCore 0x21 started
15:28:11:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:11:WU01:FS01:Starting
15:29:11:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:29:11:WU01:FS01:Started FahCore on PID 17588
15:29:11:WU01:FS01:Core PID:3396
15:29:11:WU01:FS01:FahCore 0x21 started
15:29:12:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:12:WARNING:WU01:FS01:Too many errors, failing
15:29:12:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:11 clone:208 gen:180 core:0x21 unit:0x000000f08ca304e75adf770688fabb06
15:29:12:WU01:FS01:Connecting to 140.163.4.231:8080
15:29:12:WU01:FS01:Server responded WORK_ACK (400)
15:29:12:WU01:FS01:Cleaning up
15:29:12:WU02:FS01:Connecting to 65.254.110.245:8080
15:29:13:WU02:FS01:Assigned to work server 140.163.4.231
15:29:13:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:29:13:WU02:FS01:Connecting to 140.163.4.231:8080
15:29:13:WU02:FS01:Downloading 5.42MiB
15:29:19:WU02:FS01:Download 86.55%
15:29:19:WU02:FS01:Download complete
15:29:20:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11715 run:0 clone:1722 gen:28 core:0x21 unit:0x000000228ca304e75b3c20e92848236b
15:29:20:WU02:FS01:Starting
15:29:20:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:29:20:WU02:FS01:Started FahCore on PID 20656
15:29:20:WU02:FS01:Core PID:25572
15:29:20:WU02:FS01:FahCore 0x21 started
15:29:20:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:29:20:WU02:FS01:Starting
15:29:20:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:29:20:WU02:FS01:Started FahCore on PID 25480
15:29:20:WU02:FS01:Core PID:1912
15:29:20:WU02:FS01:FahCore 0x21 started
15:29:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:30:20:WU02:FS01:Starting
15:30:20:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:30:20:WU02:FS01:Started FahCore on PID 16324
15:30:20:WU02:FS01:Core PID:4568
15:30:20:WU02:FS01:FahCore 0x21 started
15:30:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:31:20:WU02:FS01:Starting
15:31:20:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:31:20:WU02:FS01:Started FahCore on PID 14168
15:31:20:WU02:FS01:Core PID:15064
15:31:20:WU02:FS01:FahCore 0x21 started
15:31:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:32:21:WU02:FS01:Starting
15:32:21:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:32:21:WU02:FS01:Started FahCore on PID 26180
15:32:21:WU02:FS01:Core PID:29648
15:32:21:WU02:FS01:FahCore 0x21 started
15:32:21:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:32:21:WARNING:WU02:FS01:Too many errors, failing
15:32:21:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11715 run:0 clone:1722 gen:28 core:0x21 unit:0x000000228ca304e75b3c20e92848236b
15:32:21:WU02:FS01:Connecting to 140.163.4.231:8080
15:32:22:WU02:FS01:Server responded WORK_ACK (400)
15:32:22:WU02:FS01:Cleaning up
15:32:22:WU01:FS01:Connecting to 65.254.110.245:8080
15:32:22:WU01:FS01:Assigned to work server 140.163.4.231
15:32:22:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:32:22:WU01:FS01:Connecting to 140.163.4.231:8080
15:32:23:WU01:FS01:Downloading 16.53MiB
15:32:29:WU01:FS01:Download 29.87%
15:32:35:WU01:FS01:Download 58.97%
15:32:41:WU01:FS01:Download 88.84%
15:32:43:WU01:FS01:Download complete
15:32:43:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11713 run:21 clone:493 gen:127 core:0x21 unit:0x000000ac8ca304e75adf7c88c625ad6e
15:33:58:WU01:FS01:Downloading core from http://cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah
15:33:58:WU01:FS01:Connecting to cores.foldingathome.org:80
15:33:59:WU01:FS01:FahCore 21: Downloading 3.47MiB
15:34:03:WU01:FS01:FahCore 21: Download complete
15:34:03:WU01:FS01:Valid core signature
15:34:03:WARNING:WU01:FS01:FahCore has not changed since last download, aborting core update
15:34:03:WU01:FS01:Starting
15:34:03:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:34:03:WU01:FS01:Started FahCore on PID 11944
15:34:03:WU01:FS01:Core PID:15816
15:34:03:WU01:FS01:FahCore 0x21 started
15:34:03:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:34:04:WU01:FS01:Starting
15:34:04:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:34:04:WU01:FS01:Started FahCore on PID 11192
15:34:04:WU01:FS01:Core PID:14952
15:34:04:WU01:FS01:FahCore 0x21 started
15:34:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:35:04:WU01:FS01:Starting
15:35:04:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:35:04:WU01:FS01:Started FahCore on PID 29224
15:35:04:WU01:FS01:Core PID:12100
15:35:04:WU01:FS01:FahCore 0x21 started
15:35:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:36:04:WU01:FS01:Starting
15:36:04:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:36:04:WU01:FS01:Started FahCore on PID 28260
15:36:04:WU01:FS01:Core PID:20876
15:36:04:WU01:FS01:FahCore 0x21 started
15:36:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:04:WU01:FS01:Starting
15:37:04:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:37:04:WU01:FS01:Started FahCore on PID 23084
15:37:04:WU01:FS01:Core PID:2180
15:37:04:WU01:FS01:FahCore 0x21 started
15:37:04:WARNING:WU01:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:04:WARNING:WU01:FS01:Too many errors, failing
15:37:04:WU01:FS01:Sending unit results: id:01 state:SEND error:FAILED project:11713 run:21 clone:493 gen:127 core:0x21 unit:0x000000ac8ca304e75adf7c88c625ad6e
15:37:04:WU01:FS01:Connecting to 140.163.4.231:8080
15:37:05:WU01:FS01:Server responded WORK_ACK (400)
15:37:05:WU01:FS01:Cleaning up
15:37:05:WU02:FS01:Connecting to 65.254.110.245:8080
15:37:05:WU02:FS01:Assigned to work server 140.163.4.231
15:37:05:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:37:05:WU02:FS01:Connecting to 140.163.4.231:8080
15:37:09:ERROR:WU02:FS01:Exception: Server did not assign work unit
15:37:09:WU02:FS01:Connecting to 65.254.110.245:8080
15:37:09:WU02:FS01:Assigned to work server 140.163.4.231
15:37:09:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
15:37:09:WU02:FS01:Connecting to 140.163.4.231:8080
15:37:10:WU02:FS01:Downloading 16.52MiB
15:37:16:WU02:FS01:Download 28.75%
15:37:22:WU02:FS01:Download 55.24%
15:37:28:WU02:FS01:Download 83.61%
15:37:31:WU02:FS01:Download complete
15:37:31:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11713 run:15 clone:43 gen:230 core:0x21 unit:0x0000012b8ca304e75a5a52238432d0ba
15:37:31:WU02:FS01:Starting
15:37:31:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:37:31:WU02:FS01:Started FahCore on PID 1256
15:37:31:WU02:FS01:Core PID:26196
15:37:31:WU02:FS01:FahCore 0x21 started
15:37:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:37:32:WU02:FS01:Starting
15:37:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:37:32:WU02:FS01:Started FahCore on PID 9580
15:37:32:WU02:FS01:Core PID:17704
15:37:32:WU02:FS01:FahCore 0x21 started
15:37:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:38:32:WU02:FS01:Starting
15:38:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:38:32:WU02:FS01:Started FahCore on PID 624
15:38:32:WU02:FS01:Core PID:24572
15:38:32:WU02:FS01:FahCore 0x21 started
15:38:32:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:39:32:WU02:FS01:Starting
15:39:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:39:32:WU02:FS01:Started FahCore on PID 872
15:39:32:WU02:FS01:Core PID:16692
15:39:32:WU02:FS01:FahCore 0x21 started
15:39:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:40:32:WU02:FS01:Starting
15:40:32:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 26696 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:40:32:WU02:FS01:Started FahCore on PID 564
15:40:32:WU02:FS01:Core PID:27616
15:40:32:WU02:FS01:FahCore 0x21 started
15:40:33:WARNING:WU02:FS01:FahCore returned: FAILED_1 (0 = 0x0)
15:40:33:WARNING:WU02:FS01:Too many errors, failing
15:40:33:WU02:FS01:Sending unit results: id:02 state:SEND error:FAILED project:11713 run:15 clone:43 gen:230 core:0x21 unit:0x0000012b8ca304e75a5a52238432d0ba
15:40:33:WU02:FS01:Connecting to 140.163.4.231:8080
15:40:33:WU02:FS01:Server responded WORK_ACK (400)
15:40:33:WU02:FS01:Cleaning up
******************************* Date: 2018-07-18 *******************************
******************************* Date: 2018-07-19 *******************************
******************************* Date: 2018-07-19 *******************************
******************************* Date: 2018-07-19 *******************************

Another member of our team says that he is having the same problem and he has multiple computers many better than mine all acting the same way. He wonders if it is a server problem.

I will be going this thread again from the begining later hoping to be able to give you more info.
Thanks again for your time.

Post by **bruce** » Thu Jul 19, 2018 4:33 pm

Good information ... thanks.

The "Transfer failed" error shown in your "latest log" may be a server problem. Some folks have been working on something that looks a lot like that and I don't have all the details for their progress or lack thereof. I'd like to see a page or two before that particular segment.

The "Could not get IP address for assign1.foldingathome.org: No such host is known." gives a clue, but not necessarily sufficient information. That indicates a DNS problem and it should never happen.

Since you're using FAHControl to display the applicable segments of your log, there's a couple of facts that are not obvious at first. In Windows, when FAHClient is restarted, a new log is initiated and the old one is retained in a separate folder called "logs" If you fold for a while without restarting FAHClient, the log can get quite long and FAHControl will truncate the top portions. When that happens, you can override that truncation by selecting the "refresh" button in the lower right. Then you can scroll back to the very beginning of the file. When I do that, I often uncheck the "Follow" button so FAHClient doesn't refresh the display if a new line happens to appear at the bottom, but eventually it's good for Follow to be re-enabled. The system description is ALWAYS at the very beginning of a new log (in addition to being on the "System info" tab) along with a display of other information about the FAH's initialization.

Post by **Joe_H** » Thu Jul 19, 2018 4:49 pm

rfharbin wrote:> Is the GPU overheating?
No way that this is likely. I previously folded with a GTX 970 card and the GTX 10XX cards are very much cooler. Also my case has three 230 mm fans, on the top, left side, and frount plus a 140 mm fan on the back.

Do you have an app to monitor the GPU temperatures? Though it might be unlikely, overheating is still possible even with a lot of fans if the airflow is not right. It can also happene if the fans on the GPU itself do not ramp up their speed fast enough when the GPU goes from unloaded to near full load.

rfharbin wrote:Where is FAH's system description from the top of the log?
There was none but there is a tab with that info on the FAHControl but there is no way I would likely be able to manually copy all that stuff without errors. I tried to attach an image when I started this thread. I will try to learn how again. If there is a specific item you need please ask and I will do my best to manually copy it.

This section of the Welcome topic - viewtopic.php?p=261082&f=24#p261082 - gives directions on locating and posting your log file. Usually all you need to do is to use the Refresh button, it will reload the log file from the beginning.

rfharbin · Post by **rfharbin** » Thu Jul 19, 2018 8:07 pm

My "C" drive checks out OK as does my memory.

I hope this gives you the log info you want.

Code: Select all

*********************** Log Started 2018-07-19T19:55:05Z ***********************
20:11:23:FS01:Unpaused
20:11:24:WU00:FS01:Connecting to 65.254.110.245:8080
20:11:25:WU00:FS01:Assigned to work server 140.163.4.231
20:11:25:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 140.163.4.231
20:11:25:WU00:FS01:Connecting to 140.163.4.231:8080
20:11:25:WU00:FS01:Downloading 16.53MiB
20:11:31:WU00:FS01:Download 19.66%
20:11:37:WU00:FS01:Download 41.21%
20:11:43:WU00:FS01:Download 62.75%
20:11:49:WU00:FS01:Download 92.62%
20:11:50:WU00:FS01:Download complete
20:11:50:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11713 run:3 clone:270 gen:137 core:0x21 unit:0x000000ae8ca304e75adf730cde6a4012
20:11:50:WU00:FS01:Starting
20:11:50:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 8968 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
20:11:50:WU00:FS01:Started FahCore on PID 6288
20:11:50:WU00:FS01:Core PID:17876
20:11:50:WU00:FS01:FahCore 0x21 started
20:11:51:WARNING:WU00:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:11:51:WU00:FS01:Starting
20:11:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 8968 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
20:11:51:WU00:FS01:Started FahCore on PID 11336
20:11:51:WU00:FS01:Core PID:764
20:11:51:WU00:FS01:FahCore 0x21 started
20:11:52:WARNING:WU00:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:12:51:WU00:FS01:Starting
20:12:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 8968 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
20:12:51:WU00:FS01:Started FahCore on PID 14368
20:12:51:WU00:FS01:Core PID:14964
20:12:51:WU00:FS01:FahCore 0x21 started
20:12:52:WARNING:WU00:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:13:51:WU00:FS01:Starting
20:13:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 8968 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
20:13:51:WU00:FS01:Started FahCore on PID 8352
20:13:51:WU00:FS01:Core PID:13748
20:13:51:WU00:FS01:FahCore 0x21 started
20:13:52:WARNING:WU00:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:14:51:WU00:FS01:Starting
20:14:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\WINDOWS\system32\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 705 -lifeline 8968 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
20:14:51:WU00:FS01:Started FahCore on PID 148
20:14:51:WU00:FS01:Core PID:11156
20:14:51:WU00:FS01:FahCore 0x21 started
20:14:52:WARNING:WU00:FS01:FahCore returned: FAILED_1 (0 = 0x0)
20:14:52:WARNING:WU00:FS01:Too many errors, failing

My GPU is running at 73*. It is rated OK up to 94* but under 83* is considered better. I will keep an eye on it when I get the GPU running again as it may run hotter. For some reason the fan speed controls are in the BIOS and I will run the fans faster if the Temperature gets up to 80*.

bollix47 · Post by **bollix47** » Fri Jul 20, 2018 1:03 am

@rfharbin You may already know about this problem but I recently updated my Win 10 Home from version 1709 to 1803. When I finished the update the GPU drivers were the same version but I no longer had opencl support for my GPU. I reinstalled the drivers that I had previously downloaded from nvidia. That cured the lack of opencl and folding worked fine.

rfharbin · Post by **rfharbin** » Fri Jul 20, 2018 2:10 am

bollix47 wrote:@rfharbin You may already know about this problem but I recently updated my Win 10 Home from version 1709 to 1803. When I finished the update the GPU drivers were the same version but I no longer had opencl support for my GPU. I reinstalled the drivers that I had previously downloaded from nvidia. That cured the lack of opencl and folding worked fine.

Microsoft "Improves" the drivers and eliminates parts of them. You can set Win10 to not let the drivers be updated but when you get the spring and fall major updates you will have to make the change again as MS changes the settings back to normal. Nvidia GeForce Experence will have a copy of your current driver, This makes it easy to get the correct driver and then you change Auto driver downloads setting as below. If your card manufacturer does not keep an copy of the driver on your computer as Nvidia GeForce Experence does I would bookmark a link to where the drivers can be downloaded.

I copied this from a Google search.
How to Disable Automatic Driver Downloads on Windows 10
Right click the Start button and select Control Panel.
2. Make your way to System and Security.
Click System.
Click Advanced system settings from the left sidebar.
Select the Hardware tab.
Press the Device Installation Settings button.
Choose No, and then press the Save Changes button.

rfharbin · Post by **rfharbin** » Fri Jul 20, 2018 3:13 pm

Bruce; I got the GPU unit folding again and it is Running at 75*. This cool for my GPU. I Paused With FAHControl then used an End Process command to shut down the Client and then started it up again. It picked up right where it left off But nothing shows in the log for the GPU but [*********************** Log Started 2018-07-20T13:59:32Z ***********************]

Post by **Joe_H** » Fri Jul 20, 2018 4:20 pm

Pleas use the Refresh button. It will reload the log from the beginning. Or use the directions to navigate to the location of the log files and open the current log file in a text editor.

rfharbin · Post by **rfharbin** » Fri Jul 20, 2018 4:38 pm

Joe_H wrote:Pleas use the Refresh button. It will reload the log from the beginning. Or use the directions to navigate to the location of the log files and open the current log file in a text editor.

The FAHControl was giving erroneous info. What I posted above was a copy of the GPU, log 02, which was the one that the FAHControl showed at the time of my post. After reading your post I looked again and the WU has since finished and is now it is showing Folding slot / Work Queue log 01 and it starts from the beginning as is expected. Why it showed a Work Queue 02 log is a mystery to me.

Post by **Joe_H** » Fri Jul 20, 2018 5:56 pm

Unless you click on one of the filters, FAHControl will show information for all slots and WU's. But right after some changes to configurations and updating them it will just show messages that come in afterwards. Refresh will reload the log from the beginning, but apply any filters you have checked off.

Again, the directions linked to above do give you the option of opening the log file in an editor. That will show all messages from the beginning up to the time you open the file. Windows will tell you the file is open for write by another process, so any changes will not be saved to the original log.txt file. The same location will have a folder of older logs, by default up to 16.

Post by **bruce** » Fri Jul 20, 2018 6:22 pm

You are confusing trhe WorkQueue number with the Slot number.

At the beginning of the log, you'll find something like this:

Code: Select all

00:14:25:******************************* System ********************************
00:14:25:            CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
00:14:25:         CPU ID: GenuineIntel Family 6 Model 26 Stepping 4
00:14:25:           CPUs: 8
00:14:25:         Memory: 5.81GiB
00:14:25:    Free Memory: 4.98GiB
00:14:25:        Threads: POSIX_THREADS
00:14:25:     OS Version: 4.13
00:14:25:    Has Battery: false
00:14:25:     On Battery: false
00:14:25:     UTC Offset: -7
00:14:25:            PID: 1750
00:14:25:            CWD: /var/lib/fahclient
00:14:25:             OS: Linux 4.13.0-45-generic x86_64
00:14:25:        OS Arch: AMD64
00:14:25:           GPUs: 2
00:14:25:          GPU 0: Bus:3 Slot:0 Func:0 NVIDIA:4 GM107 [GeForce GTX 750 Ti] 1306
00:14:25:          GPU 1: Bus:4 Slot:0 Func:0 NVIDIA:5 GM206 [GeForce GTX 960] 2308
00:14:25:  CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:5.2 Driver:9.0
00:14:25:  CUDA Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:5.0 Driver:9.0
00:14:25:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.2 Driver:384.130
00:14:25:OpenCL Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:1.2 Driver:384.130
00:14:25:***********************************************************************
00:14:25:<config>
00:14:25:  <!-- Network -->
00:14:25:  <proxy v=':8080'/>
00:14:25:
00:14:25:  <!-- Slot Control -->
00:14:25:  <power v='full'/>
00:14:25:
00:14:25:  <!-- User Information -->
00:14:25:  <passkey v='********************************'/>
00:14:25:  <team v='131'/>
00:14:25:  <user v='borden.b'/>
00:14:25:
00:14:25:  <!-- Folding Slots -->
00:14:25:  <slot id='0' type='CPU'/>
00:14:25:  <slot id='1' type='GPU'.>
00:14:25:  <slot id='2' type='GPU'/>
00:14:25:</config>
00:14:25:Switching to user fahclient
00:14:25:Trying to access database...
00:14:25:Successfully acquired database lock
00:14:25:Enabled folding slot 00: READY cpu:6
00:14:25:Enabled folding slot 01: READY gpu:0:GM107 [GeForce GTX 750 Ti] 1306
00:14:25:Enabled folding slot 02: READY gpu:1:GM206 [GeForce GTX 960] 2308

I have 8 CPUs and 2 GPUs.
My config establishes three slots with ids 0,1,2. The first is a CPU slot and slots 1 and 2 are GPU slots.

In the last 3 lines that I quoted, a relationship is established beteen those slot number an my hardware. That association does not change within a single run (but it might be different in a later run of FAHClient, though that's unlikely.)

Not shown (yet), I may already have WUs that have downloaded started processing or I may not. The three slots will either resume processing on an active WU or they may download a new WU. Each WU has a Queue Number which is assigned when the WU downloads and stays with that WU until it has been uploaded but has NO OTHER RELATIONSHIP with the Slot number.

Suppose a WU is already present with Queue# =1 and it's been partially processed by the CPU (slot 0. It will be reassigned to slot 0. Slots 1 and 2 will download new WUs which will be assigned queue numbers 0 and 2 which will be assigned to the GPUs in slots 1 and 2. You have to pay attention to whether Slot 1 gets WU 0 or 2 and slot 2 gets the other one or the opposite happens.

You'll notice that the messages in the log give you BOTH numbers. For example:

Code: Select all

00:14:33:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************

You'll notice that these messages are from the WU with queue number 02 which is currently running in FAH Slot 00 using FAHCore 0xa7

rfharbin · Post by **rfharbin** » Fri Jul 20, 2018 8:08 pm

Thanks bruce, I will go through this again tomorrow to be sure I have all I can get.
What I have been doing is looking at the Folding slots getting the ID number and noticing the Progress on the right side of the window and then looking at the Work Queue that shows the same Progress and using these numbers to look at the log. In the log, I click on the check boxes to isolate just the log info I want to see. Now the odd thing is that the GPU log was 02 the first time I looked at it but after the then current WU finished the 02 that had nothing in the log disappeared and the Log Started etc. With a new WU and then a log 01 appeared and it seemed to show everything from the beginning that I expected to see the first time when all that was there was [*********************** Log Started 2018-07-20T13:59:32Z ***********************] when the log was the previous number (02) with nothing below appeared in the log 01. The GPU finished the WU with nothing appearing below the Log Started line and then the data seemed to appear under the Log01. The percentages on the right hand side of the FAHControl seemed to fit what I thought had been done.

Yes I may confused and I will go through what you posted again now and tomorrow.
FYI When a GPU WU fails the Work Queue 0x will be there for a while and then Work Queue GPU ID number and percentage will disappear until one stops the Client and starts it again.

I am sure I understand all you have posted above.
In the C:/users/admin~~~/Appdata/ /Roaming/FAHClient/logs/There are a bunch of log.txt files {log-(date #'s)-(6 digits).txt and also under the /FAHClient...there is a log.txt. I assume that the log.txt under the FAHClient is the current running info. and the log.txt files under the logs folder is previous data. I have stopped mine enough I can't be sure but I want to get familiar with this hoping to learn something. Tomorrow.

Post by **bruce** » Sat Jul 21, 2018 3:33 am

rfharbin wrote:FYI When a GPU WU fails the Work Queue 0x will be there for a while and then Work Queue GPU ID number and percentage will disappear until one stops the Client and starts it again.

When you look at the log through the log tab of FAHClient, you'll see the log associated with work queue WU02 for as long as WW02 exists. It starts when the WU downloads and it disappears when the result is uploaded to the server. If you look at the log on disk, it's all still there but you don't have the convenience of a filter.

If you filter by slot, you'll notice that the startup of a new WU and the ending of the previous one overlap for a short time. You can learn to mentally filter the log and that can help if you look at logs on disk.

In the C:/users/admin~~~/Appdata/ /Roaming/FAHClient/logs/There are a bunch of log.txt files {log-(date #'s)-(6 digits).txt and also under the /FAHClient...there is a log.txt. I assume that the log.txt under the FAHClient is the current running info. and the log.txt files under the logs folder is previous data.

Right. As I attempted to explain above, log.txt is the active log and it matches what you see in FAHClient when no filters are applied. When you restart FAHClient, the current log is moved to the logs folder and renamed by data-time. Then a new log.txt is started. By default, 16 previous logs are kept in the folder and the oldest is discarded.

Why did you install FAH under the name admin...? As I said much earlier, it really should be installed as rfharbin (or whatever name belongs to the person who logs on to the computer tu run FAH). When the client is installed normally in Windows, I expect to find my FAH files in C:\users\bruce\AppData\Roaming\FAHClient. In any case, the information can be found about 30 lines from the top of the log .... following a CWD:

In a Windows7 installation, there were several shotcuts created at install-time. They're still somewhere in Windows10 but I don't remember where. (Microsoft always likes to change things.) One was called FAHData which took you to the right location for your installation.

Folding Forum

GPU WU's constantly failing

GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing

Re: GPU WU's constantly failing