I've got 2 finished WU's worth 150-200k each sitting for more than an hour deteriorating in value. Resetting console does not help. Upload gets to 0.05% and fails on one that's over 200MB. Do I need to really dump 400k in WU's or will they upload? They are listing "FAULTY" in the error line... not sure what's up here. They folded on two separate gpu's... bad WU issued? Gonna leave it up and hit the rack I guess. Hope it sorts out.
06:49:14:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14253 run:16 clone:0 gen:8 core:0x22 unit:0x00000008cedfaa925eab708aa0fe1352
06:49:14:WU00:FS01:Uploading 140.85MiB to 206.223.170.146
06:49:14:WU00:FS01:Connecting to 206.223.170.146:8080
06:49:35:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
06:49:35:WU00:FS01:Connecting to 206.223.170.146:80
06:49:56:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 206.223.170.146:80: Bağlanılan uygun olarak belli bir süre içinde yanıt vermediğinden veya kurulan
06:49:56:WARNING:WU00:FS01:bağlantı bağlanılan ana bilgisayar yanıt vermediğinden bir bağlantı kurulamadı.
06:49:56:WU00:FS01:Trying to send results to collection server
06:49:56:WU00:FS01:Uploading 140.85MiB to 52.224.109.74
06:49:56:WU00:FS01:Connecting to 52.224.109.74:8080
06:49:56:ERROR:WU00:FS01:Exception: Transfer failed
i have the same problem too but the WU isn't faulty.
@schapman1978, can you post the section from the log where the two WUs completed? I'm curious to know at what point they aborted as the WU status has "error:FAULTY" identification when uploading the results.
Project: 14253 (Run 231, Clone 0, Gen 0)
Unit: 0x00000000cedfaa925eab708242e5531a
--
Completed 225000 out of 500000 steps (45%)
Completed 230000 out of 500000 steps (46%)
Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
Following exception occured: Particle coordinate is nan
Completed 225000 out of 500000 steps (45%)
Completed 230000 out of 500000 steps (46%)
Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
Following exception occured: Particle coordinate is nan
ERROR:114: Max Retries Reached
Saving result file ../logfile_01.txt
Saving result file badstate-0.xml
Saving result file badstate-1.xml
Saving result file badstate-2.xml
Saving result file checkpointState.xml
Saving result file checkpt.crc
Saving result file positions.xtc
Saving result file science.log
Folding@home Core Shutdown: BAD_WORK_UNIT
10:53:55:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
10:53:55:WU01:FS00:Trying to send results to collection server
10:53:55:WU01:FS00:Uploading 258.14MiB to 52.224.109.74
10:53:55:WU01:FS00:Connecting to 52.224.109.74:8080
10:53:56:ERROR:WU01:FS00:Exception: Transfer failed
The card is overclocked but has done literally thousands of units with a less than 0.1% failure rate.
I'v got three stuck this morning when I woke up - they were all 200k plus but are in the 70's now still waiting to upload. My other machines folding one now too, and I've got another folding as it is. I have almost never had bad workunits that didn't result from a system event or power issue.
We're down to less than half of yesterdays production since midnight.
I've had to truncate the log because of more characters than allowed.
11:31:40:WARNING:WU06:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:31:40:WU06:FS01:Connecting to 18.218.241.186:80
11:31:40:WARNING:WU06:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:31:40:WU06:FS01:Connecting to 65.254.110.245:80
11:31:41:WARNING:WU06:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:31:41:WU06:FS01:Connecting to 18.218.241.186:80
11:31:41:WU06:FS01:Assigned to work server 206.223.170.146
11:31:41:WU06:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 206.223.170.146
11:31:41:WU06:FS01:Connecting to 206.223.170.146:8080
11:31:49:WARNING:WU04:FS01:Exception: Failed to send results to work server: Failed to connect to 206.223.170.146:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:31:49:WU04:FS01:Trying to send results to collection server
11:31:49:WU04:FS01:Uploading 261.61MiB to 52.224.109.74
11:31:49:WU04:FS01:Connecting to 52.224.109.74:8080
11:31:50:ERROR:WU04:FS01:Exception: Transfer failed
11:31:50:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 206.223.170.146:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:31:50:WU00:FS01:Trying to send results to collection server
11:31:50:WU00:FS01:Uploading 279.22MiB to 52.224.109.74
11:31:50:WU00:FS01:Connecting to 52.224.109.74:8080
11:31:54:ERROR:WU00:FS01:Exception: Transfer failed
11:31:56:WU05:FS00:0xa7:Completed 350000 out of 500000 steps (70%)
11:31:57:WU02:FS01:Upload 0.11%
11:32:02:WARNING:WU06:FS01:WorkServer connection failed on port 8080 trying 80
11:32:02:WU06:FS01:Connecting to 206.223.170.146:80
11:32:03:WU02:FS01:Upload 5.19%
11:32:09:WU02:FS01:Upload 9.79%
11:32:10:WU01:FS01:Upload 0.07%
11:32:11:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
11:32:11:WU01:FS01:Trying to send results to collection server
11:32:11:WU01:FS01:Uploading 261.61MiB to 52.224.109.74
11:32:11:WU01:FS01:Connecting to 52.224.109.74:8080
11:32:11:ERROR:WU01:FS01:Exception: Transfer failed
11:32:15:WU02:FS01:Upload 14.87%
11:32:21:WU02:FS01:Upload 19.88%
11:32:23:ERROR:WU06:FS01:Exception: Failed to connect to 206.223.170.146:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:32:25:WU03:FS02:0x22:Completed 425000 out of 500000 steps (85%)
11:32:27:WU02:FS01:Upload 24.82%
11:32:33:WU02:FS01:Upload 30.01%
11:32:39:WU02:FS01:Upload 35.01%
11:32:41:WU05:FS00:0xa7:Completed 355000 out of 500000 steps (71%)
11:32:45:WU02:FS01:Upload 40.06%
11:32:51:WU02:FS01:Upload 44.84%
11:32:57:WU02:FS01:Upload 49.89%
11:33:03:WU02:FS01:Upload 54.82%
11:33:09:WU02:FS01:Upload 59.72%
11:33:15:WU02:FS01:Upload 64.95%
11:33:17:WU06:FS01:Connecting to 65.254.110.245:80
11:33:17:WU06:FS01:Assigned to work server 128.252.203.2
11:33:17:WU06:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 128.252.203.2
11:33:17:WU06:FS01:Connecting to 128.252.203.2:8080
11:33:18:ERROR:WU06:FS01:Exception: Server did not assign work unit
11:33:21:WU02:FS01:Upload 70.14%
11:33:22:WU03:FS02:0x22:Completed 430000 out of 500000 steps (86%)
11:33:25:WU05:FS00:0xa7:Completed 360000 out of 500000 steps (72%)
11:33:27:WU02:FS01:Upload 75.33%
11:33:33:WU02:FS01:Upload 79.86%
11:33:39:WU02:FS01:Upload 84.46%
11:33:45:WU02:FS01:Upload 89.28%
11:33:51:WU02:FS01:Upload 94.18%
11:33:57:WU02:FS01:Upload 99.26%
11:33:58:WU02:FS01:Upload complete
11:33:58:WU02:FS01:Server responded WORK_ACK (400)
11:33:58:WU02:FS01:Cleaning up
11:34:09:WU05:FS00:0xa7:Completed 365000 out of 500000 steps (73%)
11:34:19:WU03:FS02:0x22:Completed 435000 out of 500000 steps (87%)
11:34:53:WU05:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
11:35:16:WU03:FS02:0x22:Completed 440000 out of 500000 steps (88%)
11:35:39:WU05:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
11:35:54:WU06:FS01:Connecting to 65.254.110.245:80
11:35:55:WU06:FS01:Assigned to work server 13.82.98.119
11:35:55:WU06:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 13.82.98.119
11:35:55:WU06:FS01:Connecting to 13.82.98.119:8080
11:36:01:WU06:FS01:Downloading 77.19MiB
11:36:07:WU06:FS01:Download 47.05%
11:36:13:WU06:FS01:Download 92.31%
11:36:13:WU06:FS01:Download complete
11:36:14:WU06:FS01:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:14430 run:0 clone:275 gen:45 core:0x22 unit:0x000000380d5262775e8b4d58ecd21ee0
11:36:14:WU06:FS01:Starting
11:36:14:WU06:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\schap\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 06 -suffix 01 -version 706 -lifeline 6812 -checkpoint 5 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
11:36:14:WU06:FS01:Started FahCore on PID 13560
11:36:14:WU06:FS01:Core PID:9556
11:36:14:WU06:FS01:FahCore 0x22 started
11:36:14:WU06:FS01:0x22:*********************** Log Started 2020-05-05T11:36:14Z ***********************
11:36:14:WU06:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:36:14:WU06:FS01:0x22: Type: 0x22
11:36:14:WU06:FS01:0x22: Core: Core22
11:36:14:WU06:FS01:0x22: Website: https://foldingathome.org/
11:36:14:WU06:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
11:36:14:WU06:FS01:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
11:36:14:WU06:FS01:0x22: <[email protected]>
11:36:14:WU06:FS01:0x22: Args: -dir 06 -suffix 01 -version 706 -lifeline 13560 -checkpoint 5
11:36:14:WU06:FS01:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
11:36:14:WU06:FS01:0x22: 0 -gpu 0
11:36:14:WU06:FS01:0x22: Config: <none>
11:36:14:WU06:FS01:0x22:************************************ Build *************************************
11:36:14:WU06:FS01:0x22: Version: 0.0.5
11:36:14:WU06:FS01:0x22: Date: Apr 22 2020
11:36:14:WU06:FS01:0x22: Time: 04:42:59
11:36:14:WU06:FS01:0x22: Repository: Git
11:36:14:WU06:FS01:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
11:36:14:WU06:FS01:0x22: Branch: HEAD
11:36:14:WU06:FS01:0x22: Compiler: Visual C++ 2008
11:36:14:WU06:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:36:14:WU06:FS01:0x22: Platform: win32 10
11:36:14:WU06:FS01:0x22: Bits: 64
11:36:14:WU06:FS01:0x22: Mode: Release
11:36:14:WU06:FS01:0x22:************************************ System ************************************
11:36:14:WU06:FS01:0x22: CPU: AMD Ryzen 9 3950X 16-Core Processor
11:36:14:WU06:FS01:0x22: CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:36:14:WU06:FS01:0x22: CPUs: 32
11:36:14:WU06:FS01:0x22: Memory: 31.91GiB
11:36:14:WU06:FS01:0x22:Free Memory: 24.50GiB
11:36:14:WU06:FS01:0x22: Threads: WINDOWS_THREADS
11:36:14:WU06:FS01:0x22: OS Version: 6.2
11:36:14:WU06:FS01:0x22:Has Battery: false
11:36:14:WU06:FS01:0x22: On Battery: false
11:36:14:WU06:FS01:0x22: UTC Offset: -4
11:36:14:WU06:FS01:0x22: PID: 9556
11:36:14:WU06:FS01:0x22: CWD: C:\Users\schap\AppData\Roaming\FAHClient\work
11:36:14:WU06:FS01:0x22: OS: Windows 10 Pro
11:36:14:WU06:FS01:0x22: OS Arch: AMD64
11:36:14:WU06:FS01:0x22:********************************************************************************
11:36:14:WU06:FS01:0x22:Project: 14430 (Run 0, Clone 275, Gen 45)
11:36:14:WU06:FS01:0x22:Unit: 0x000000380d5262775e8b4d58ecd21ee0
11:36:14:WU06:FS01:0x22:Reading tar file core.xml
11:36:14:WU06:FS01:0x22:Reading tar file integrator.xml
11:36:14:WU06:FS01:0x22:Reading tar file state.xml
11:36:15:WU06:FS01:0x22:Reading tar file system.xml
11:36:16:WU06:FS01:0x22:Digital signatures verified
11:36:16:WU06:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:36:16:WU06:FS01:0x22:Version 0.0.5
11:36:23:WU05:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
11:36:28:WU03:FS02:0x22:Completed 445000 out of 500000 steps (89%)
11:36:35:WU06:FS01:0x22:Completed 0 out of 2000000 steps (0%)
11:36:35:WU06:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:37:08:WU05:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
11:37:25:WU03:FS02:0x22:Completed 450000 out of 500000 steps (90%)
11:37:42:WU06:FS01:0x22:Completed 20000 out of 2000000 steps (1%)
11:37:53:WU05:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
11:37:58:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:14253 run:1831 clone:0 gen:0 core:0x22 unit:0x00000001cedfaa925eaba95ca9c9bccc
11:37:58:WU04:FS01:Uploading 261.61MiB to 206.223.170.146
11:37:58:WU04:FS01:Connecting to 206.223.170.146:8080
11:37:59:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14253 run:5 clone:0 gen:13 core:0x22 unit:0x0000000ecedfaa925eab7041daa04117
11:37:59:WU00:FS01:Uploading 279.22MiB to 206.223.170.146
11:37:59:WU00:FS01:Connecting to 206.223.170.146:8080
11:38:06:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14253 run:39 clone:0 gen:7 core:0x22 unit:0x0000000acedfaa925eab703e05ef414f
11:38:06:WU01:FS01:Uploading 261.61MiB to 206.223.170.146
11:38:06:WU01:FS01:Connecting to 206.223.170.146:8080
11:38:20:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
11:38:20:WU00:FS01:Connecting to 206.223.170.146:80
11:38:22:WU03:FS02:0x22:Completed 455000 out of 500000 steps (91%)
11:38:24:WU04:FS01:Upload 0.05%
11:38:27:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
11:38:27:WU01:FS01:Connecting to 206.223.170.146:80
11:38:28:WU01:FS01:Upload 0.02%
11:38:37:WU06:FS01:0x22:ERROR:exception: clWaitForEvents
11:38:37:WU06:FS01:0x22:Saving result file ..\logfile_01.txt
11:38:37:WU06:FS01:0x22:Saving result file checkpt.crc
11:38:37:WU06:FS01:0x22:Saving result file science.log
11:38:37:WU06:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
11:38:37:WU05:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
11:38:37:WARNING:WU06:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:38:37:WU06:FS01:Sending unit results: id:06 state:SEND error:FAULTY project:14430 run:0 clone:275 gen:45 core:0x22 unit:0x000000380d5262775e8b4d58ecd21ee0
11:38:37:WU06:FS01:Uploading 10.00KiB to 13.82.98.119
11:38:37:WU06:FS01:Connecting to 13.82.98.119:8080
11:38:38:WU02:FS01:Connecting to 65.254.110.245:80
11:38:38:WU02:FS01:Assigned to work server 206.223.170.146
11:38:38:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 206.223.170.146
11:38:38:WU02:FS01:Connecting to 206.223.170.146:8080
11:38:41:WU01:FS01:Upload 0.05%
11:38:42:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 206.223.170.146:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:38:42:WU00:FS01:Trying to send results to collection server
11:38:42:WU00:FS01:Uploading 279.22MiB to 52.224.109.74
11:38:42:WU00:FS01:Connecting to 52.224.109.74:8080
11:38:59:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
11:38:59:WU02:FS01:Connecting to 206.223.170.146:80
11:39:00:WU06:FS01:Upload complete
11:39:00:WU06:FS01:Server responded WORK_ACK (400)
11:39:00:WU06:FS01:Cleaning up
11:39:01:WU00:FS01:Upload 0.04%
11:39:01:ERROR:WU00:FS01:Exception: Transfer failed
11:39:18:WU04:FS01:Upload 0.07%
11:39:18:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
11:39:18:WU04:FS01:Trying to send results to collection server
11:39:18:WU04:FS01:Uploading 261.61MiB to 52.224.109.74
11:39:18:WU04:FS01:Connecting to 52.224.109.74:8080
11:39:18:ERROR:WU04:FS01:Exception: Transfer failed
11:39:19:WU03:FS02:0x22:Completed 460000 out of 500000 steps (92%)
11:39:20:ERROR:WU02:FS01:Exception: Failed to connect to 206.223.170.146:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:39:20:WU02:FS01:Connecting to 65.254.110.245:80
11:39:21:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:39:21:WU02:FS01:Connecting to 18.218.241.186:80
11:39:21:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:39:21:WU02:FS01:Connecting to 65.254.110.245:80
11:39:21:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:39:21:WU02:FS01:Connecting to 18.218.241.186:80
11:39:21:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:39:21:ERROR:WU02:FS01:Exception: Could not get an assignment
11:39:22:WU05:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
11:39:25:WU01:FS01:Upload 0.07%
11:39:25:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
11:39:25:WU01:FS01:Trying to send results to collection server
11:39:25:WU01:FS01:Uploading 261.61MiB to 52.224.109.74
11:39:25:WU01:FS01:Connecting to 52.224.109.74:8080
11:39:25:ERROR:WU01:FS01:Exception: Transfer failed
11:40:07:WU05:FS00:0xa7:Completed 405000 out of 500000 steps (81%)
11:40:21:WU02:FS01:Connecting to 65.254.110.245:80
11:40:21:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:40:21:WU02:FS01:Connecting to 18.218.241.186:80
11:40:21:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:40:21:WU02:FS01:Connecting to 65.254.110.245:80
11:40:21:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:40:21:WU02:FS01:Connecting to 18.218.241.186:80
11:40:22:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:40:22:ERROR:WU02:FS01:Exception: Could not get an assignment
11:40:31:WU03:FS02:0x22:Completed 465000 out of 500000 steps (93%)
11:40:51:WU05:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
11:41:27:WU03:FS02:0x22:Completed 470000 out of 500000 steps (94%)
11:41:35:WU05:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
11:41:58:WU02:FS01:Connecting to 65.254.110.245:80
11:41:58:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:41:58:WU02:FS01:Connecting to 18.218.241.186:80
11:41:58:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:41:58:WU02:FS01:Connecting to 65.254.110.245:80
11:41:58:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:41:58:WU02:FS01:Connecting to 18.218.241.186:80
11:41:59:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:41:59:ERROR:WU02:FS01:Exception: Could not get an assignment
11:42:19:WU05:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
11:42:24:WU03:FS02:0x22:Completed 475000 out of 500000 steps (95%)
11:43:04:WU05:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
11:43:21:WU03:FS02:0x22:Completed 480000 out of 500000 steps (96%)
11:43:49:WU05:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
11:44:33:WU03:FS02:0x22:Completed 485000 out of 500000 steps (97%)
11:44:33:WU05:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
11:44:35:WU02:FS01:Connecting to 65.254.110.245:80
11:44:35:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:44:35:WU02:FS01:Connecting to 18.218.241.186:80
11:44:35:WARNING:WU02:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:44:35:WU02:FS01:Connecting to 65.254.110.245:80
11:44:36:WU02:FS01:Assigned to work server 3.133.76.19
11:44:36:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 3.133.76.19
11:44:36:WU02:FS01:Connecting to 3.133.76.19:8080
11:44:57:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
11:44:57:WU02:FS01:Connecting to 3.133.76.19:80
11:45:18:WU05:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
11:45:18:ERROR:WU02:FS01:Exception: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:45:30:WU03:FS02:0x22:Completed 490000 out of 500000 steps (98%)
11:46:02:WU05:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
11:46:27:WU03:FS02:0x22:Completed 495000 out of 500000 steps (99%)
11:46:27:WU06:FS02:Connecting to 65.254.110.245:80
11:46:27:WARNING:WU06:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:46:27:WU06:FS02:Connecting to 18.218.241.186:80
11:46:27:WU06:FS02:Assigned to work server 206.223.170.146
11:46:27:WU06:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:TU102 [GeForce RTX 2080 Ti] M 13448 from 206.223.170.146
11:46:27:WU06:FS02:Connecting to 206.223.170.146:8080
11:46:46:WU05:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
11:47:23:WU03:FS02:0x22:Completed 500000 out of 500000 steps (100%)
11:47:31:WU05:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
11:47:38:WU03:FS02:0x22:Saving result file ..\logfile_01.txt
11:47:38:WU03:FS02:0x22:Saving result file checkpointState.xml
11:47:39:WU03:FS02:0x22:Saving result file checkpt.crc
11:47:39:WU03:FS02:0x22:Saving result file positions.xtc
11:47:40:WU03:FS02:0x22:Saving result file science.log
11:47:40:WU03:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
11:47:41:WU03:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:47:41:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:14253 run:222 clone:1 gen:2 core:0x22 unit:0x00000002cedfaa925eab7082d41723b3
11:47:41:WU03:FS02:Uploading 140.86MiB to 206.223.170.146
11:47:41:WU03:FS02:Connecting to 206.223.170.146:8080
11:48:11:ERROR:WU06:FS02:Exception: 10002: Received short response, expected 512 bytes, got 0
11:48:11:WU06:FS02:Connecting to 65.254.110.245:80
11:48:11:WARNING:WU06:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:48:11:WU06:FS02:Connecting to 18.218.241.186:80
11:48:12:WU06:FS02:Assigned to work server 128.252.203.10
11:48:12:WU06:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [GeForce RTX 2080 Ti] M 13448 from 128.252.203.10
11:48:12:WU06:FS02:Connecting to 128.252.203.10:8080
11:48:16:WU05:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
11:48:33:WARNING:WU06:FS02:WorkServer connection failed on port 8080 trying 80
11:48:33:WU06:FS02:Connecting to 128.252.203.10:80
11:48:38:WU03:FS02:Upload 0.13%
11:48:44:WU03:FS02:Upload 6.57%
11:48:49:WU02:FS01:Connecting to 65.254.110.245:80
11:48:49:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:48:49:WU02:FS01:Connecting to 18.218.241.186:80
11:48:50:WU03:FS02:Upload 13.93%
11:48:50:WU02:FS01:Assigned to work server 13.82.98.119
11:48:50:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 13.82.98.119
11:48:50:WU02:FS01:Connecting to 13.82.98.119:8080
11:48:54:ERROR:WU06:FS02:Exception: Failed to connect to 128.252.203.10:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:48:56:WU03:FS02:Upload 21.30%
11:49:00:WU05:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
11:49:02:WU03:FS02:Upload 28.71%
11:49:04:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:14253 run:1831 clone:0 gen:0 core:0x22 unit:0x00000001cedfaa925eaba95ca9c9bccc
11:49:04:WU04:FS01:Uploading 261.61MiB to 206.223.170.146
11:49:04:WU04:FS01:Connecting to 206.223.170.146:8080
11:49:04:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14253 run:5 clone:0 gen:13 core:0x22 unit:0x0000000ecedfaa925eab7041daa04117
11:49:04:WU00:FS01:Uploading 279.22MiB to 206.223.170.146
11:49:04:WU00:FS01:Connecting to 206.223.170.146:8080
11:49:08:WU03:FS02:Upload 36.07%
11:49:11:WU04:FS01:Upload 0.02%
11:49:11:WARNING:WU02:FS01:WorkServer connection failed on port 8080 trying 80
11:49:11:WU02:FS01:Connecting to 13.82.98.119:80
11:49:11:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14253 run:39 clone:0 gen:7 core:0x22 unit:0x0000000acedfaa925eab703e05ef414f
11:49:11:WU01:FS01:Uploading 261.61MiB to 206.223.170.146
11:49:11:WU01:FS01:Connecting to 206.223.170.146:8080
11:49:11:WU06:FS02:Connecting to 65.254.110.245:80
11:49:12:WU06:FS02:Assigned to work server 206.223.170.146
11:49:12:WU06:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [GeForce RTX 2080 Ti] M 13448 from 206.223.170.146
11:49:12:WU06:FS02:Connecting to 206.223.170.146:8080
11:49:14:WU03:FS02:Upload 42.55%
11:49:20:WU03:FS02:Upload 49.78%
11:49:26:WU03:FS02:Upload 56.71%
11:49:32:WU03:FS02:Upload 64.12%
11:49:32:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
11:49:32:WU01:FS01:Connecting to 206.223.170.146:80
11:49:38:WU03:FS02:Upload 71.26%
11:49:40:WU01:FS01:Upload 0.02%
11:49:44:WU03:FS02:Upload 78.36%
11:49:44:WU05:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
11:49:50:WU03:FS02:Upload 85.59%
11:49:56:WU03:FS02:Upload 92.91%
11:49:59:WU04:FS01:Upload 0.07%
11:49:59:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
11:49:59:WU04:FS01:Trying to send results to collection server
11:49:59:WU04:FS01:Uploading 261.61MiB to 52.224.109.74
11:49:59:WU04:FS01:Connecting to 52.224.109.74:8080
11:49:59:ERROR:WU04:FS01:Exception: Transfer failed
11:50:02:WU03:FS02:Upload complete
11:50:02:WU03:FS02:Server responded WORK_ACK (400)
11:50:02:WU03:FS02:Final credit estimate, 212827.00 points
11:50:02:WU03:FS02:Cleaning up
11:50:10:WU02:FS01:Downloading 138.26MiB
11:50:16:WU02:FS01:Download 24.64%
11:50:22:WU02:FS01:Download 56.82%
11:50:28:WU02:FS01:Download 88.55%
11:50:29:WU05:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
11:50:30:WU02:FS01:Download complete
11:50:30:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14416 run:0 clone:1237 gen:31 core:0x22 unit:0x000000290d5262775e84c8a69fb5980e
11:50:30:WU02:FS01:Starting
11:50:30:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\schap\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6812 -checkpoint 5 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
11:50:30:WU02:FS01:Started FahCore on PID 4024
11:50:30:WU02:FS01:Core PID:12572
11:50:30:WU02:FS01:FahCore 0x22 started
11:50:30:WU02:FS01:0x22:*********************** Log Started 2020-05-05T11:50:30Z ***********************
11:50:30:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:50:30:WU02:FS01:0x22: Type: 0x22
11:50:30:WU02:FS01:0x22: Core: Core22
11:50:30:WU02:FS01:0x22: Website: https://foldingathome.org/
11:50:30:WU02:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
11:50:30:WU02:FS01:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
11:50:30:WU02:FS01:0x22: <[email protected]>
11:50:30:WU02:FS01:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 4024 -checkpoint 5
11:50:30:WU02:FS01:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
11:50:30:WU02:FS01:0x22: 0 -gpu 0
11:50:30:WU02:FS01:0x22: Config: <none>
11:50:30:WU02:FS01:0x22:************************************ Build *************************************
11:50:30:WU02:FS01:0x22: Version: 0.0.5
11:50:30:WU02:FS01:0x22: Date: Apr 22 2020
11:50:30:WU02:FS01:0x22: Time: 04:42:59
11:50:30:WU02:FS01:0x22: Repository: Git
11:50:30:WU02:FS01:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
11:50:30:WU02:FS01:0x22: Branch: HEAD
11:50:30:WU02:FS01:0x22: Compiler: Visual C++ 2008
11:50:30:WU02:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:50:30:WU02:FS01:0x22: Platform: win32 10
11:50:30:WU02:FS01:0x22: Bits: 64
11:50:30:WU02:FS01:0x22: Mode: Release
11:50:30:WU02:FS01:0x22:************************************ System ************************************
11:50:30:WU02:FS01:0x22: CPU: AMD Ryzen 9 3950X 16-Core Processor
11:50:30:WU02:FS01:0x22: CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:50:30:WU02:FS01:0x22: CPUs: 32
11:50:30:WU02:FS01:0x22: Memory: 31.91GiB
11:50:30:WU02:FS01:0x22:Free Memory: 27.11GiB
11:50:30:WU02:FS01:0x22: Threads: WINDOWS_THREADS
11:50:30:WU02:FS01:0x22: OS Version: 6.2
11:50:30:WU02:FS01:0x22:Has Battery: false
11:50:30:WU02:FS01:0x22: On Battery: false
11:50:30:WU02:FS01:0x22: UTC Offset: -4
11:50:30:WU02:FS01:0x22: PID: 12572
11:50:30:WU02:FS01:0x22: CWD: C:\Users\schap\AppData\Roaming\FAHClient\work
11:50:30:WU02:FS01:0x22: OS: Windows 10 Pro
11:50:30:WU02:FS01:0x22: OS Arch: AMD64
11:50:30:WU02:FS01:0x22:********************************************************************************
11:50:30:WU02:FS01:0x22:Project: 14416 (Run 0, Clone 1237, Gen 31)
11:50:30:WU02:FS01:0x22:Unit: 0x000000290d5262775e84c8a69fb5980e
11:50:30:WU02:FS01:0x22:Reading tar file core.xml
11:50:30:WU02:FS01:0x22:Reading tar file integrator.xml
11:50:30:WU02:FS01:0x22:Reading tar file state.xml
11:50:31:WU01:FS01:Upload 0.07%
11:50:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
11:50:31:WU01:FS01:Trying to send results to collection server
11:50:31:WU01:FS01:Uploading 261.61MiB to 52.224.109.74
11:50:31:WU01:FS01:Connecting to 52.224.109.74:8080
11:50:32:ERROR:WU01:FS01:Exception: Transfer failed
11:50:32:WU02:FS01:0x22:Reading tar file system.xml
11:50:33:WU02:FS01:0x22:Digital signatures verified
11:50:33:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:50:33:WU02:FS01:0x22:Version 0.0.5
11:50:43:WU00:FS01:Upload 0.04%
11:50:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:50:44:WU00:FS01:Trying to send results to collection server
11:50:44:WU00:FS01:Uploading 279.22MiB to 52.224.109.74
11:50:44:WU00:FS01:Connecting to 52.224.109.74:8080
11:50:44:ERROR:WU00:FS01:Exception: Transfer failed
11:50:56:ERROR:WU06:FS02:Exception: 10002: Received short response, expected 512 bytes, got 0
11:50:56:WU06:FS02:Connecting to 65.254.110.245:80
11:50:56:WARNING:WU06:FS02:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
11:50:56:WU06:FS02:Connecting to 18.218.241.186:80
11:50:56:WARNING:WU06:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
11:50:56:WU06:FS02:Connecting to 65.254.110.245:80
11:50:57:WU06:FS02:Assigned to work server 206.223.170.146
11:50:57:WU06:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [GeForce RTX 2080 Ti] M 13448 from 206.223.170.146
11:50:57:WU06:FS02:Connecting to 206.223.170.146:8080
11:51:07:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
11:51:07:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:51:14:WU05:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
11:51:18:WARNING:WU06:FS02:WorkServer connection failed on port 8080 trying 80
11:51:18:WU06:FS02:Connecting to 206.223.170.146:80
11:51:58:WU05:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
11:52:08:WU06:FS02:Downloading 225.63MiB
11:52:14:WU06:FS02:Download 2.83%
11:52:20:WU06:FS02:Download 6.09%
11:52:26:WU06:FS02:Download 10.22%
11:52:32:WU06:FS02:Download 14.24%
11:52:34:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
11:52:38:WU06:FS02:Download 18.75%
11:52:42:WU05:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
11:52:44:WU06:FS02:Download 26.04%
11:52:50:WU06:FS02:Download 33.13%
11:52:56:WU06:FS02:Download 36.84%
11:53:02:WU06:FS02:Download 41.16%
11:53:08:WU06:FS02:Download 47.04%
11:53:14:WU06:FS02:Download 52.19%
11:53:20:WU06:FS02:Download 57.12%
11:53:26:WU06:FS02:Download 64.29%
11:53:26:WU05:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
11:53:27:WU03:FS00:Connecting to 65.254.110.245:80
11:53:27:WU03:FS00:Assigned to work server 128.252.203.4
11:53:27:WU03:FS00:Requesting new work unit for slot 00: RUNNING cpu:28 from 128.252.203.4
11:53:27:WU03:FS00:Connecting to 128.252.203.4:8080
11:53:27:WU03:FS00:Downloading 4.19MiB
11:53:28:WU03:FS00:Download complete
11:53:28:WU03:FS00:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14542 run:0 clone:568 gen:169 core:0xa7 unit:0x000000b280fccb045e7fbfaf1600bbeb
11:53:32:WU06:FS02:Download 72.30%
11:53:38:WU06:FS02:Download 79.14%
11:53:44:WU06:FS02:Download 85.46%
11:53:50:WU06:FS02:Download 89.58%
11:53:56:WU06:FS02:Download 93.05%
11:53:59:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
11:54:01:WU06:FS02:Download complete
11:54:01:WU06:FS02:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:14253 run:281 clone:1 gen:1 core:0x22 unit:0x00000001cedfaa925eab707f073df835
11:54:01:WU06:FS02:Starting
11:54:01:WU06:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\schap\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 06 -suffix 01 -version 706 -lifeline 6812 -checkpoint 5 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
11:54:01:WU06:FS02:Started FahCore on PID 4124
11:54:01:WU06:FS02:Core PID:5516
11:54:01:WU06:FS02:FahCore 0x22 started
11:54:02:WU06:FS02:0x22:*********************** Log Started 2020-05-05T11:54:02Z ***********************
11:54:02:WU06:FS02:0x22:*************************** Core22 Folding@home Core ***************************
11:54:02:WU06:FS02:0x22: Type: 0x22
11:54:02:WU06:FS02:0x22: Core: Core22
11:54:02:WU06:FS02:0x22: Website: https://foldingathome.org/
11:54:02:WU06:FS02:0x22: Copyright: (c) 2009-2018 foldingathome.org
11:54:02:WU06:FS02:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
11:54:02:WU06:FS02:0x22: <[email protected]>
11:54:02:WU06:FS02:0x22: Args: -dir 06 -suffix 01 -version 706 -lifeline 4124 -checkpoint 5
11:54:02:WU06:FS02:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device
11:54:02:WU06:FS02:0x22: 1 -gpu 1
11:54:02:WU06:FS02:0x22: Config: <none>
11:54:02:WU06:FS02:0x22:************************************ Build *************************************
11:54:02:WU06:FS02:0x22: Version: 0.0.5
11:54:02:WU06:FS02:0x22: Date: Apr 22 2020
11:54:02:WU06:FS02:0x22: Time: 04:42:59
11:54:02:WU06:FS02:0x22: Repository: Git
11:54:02:WU06:FS02:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
11:54:02:WU06:FS02:0x22: Branch: HEAD
11:54:02:WU06:FS02:0x22: Compiler: Visual C++ 2008
11:54:02:WU06:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:54:02:WU06:FS02:0x22: Platform: win32 10
11:54:02:WU06:FS02:0x22: Bits: 64
11:54:02:WU06:FS02:0x22: Mode: Release
11:54:02:WU06:FS02:0x22:************************************ System ************************************
11:54:02:WU06:FS02:0x22: CPU: AMD Ryzen 9 3950X 16-Core Processor
11:54:02:WU06:FS02:0x22: CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:54:02:WU06:FS02:0x22: CPUs: 32
11:54:02:WU06:FS02:0x22: Memory: 31.91GiB
11:54:02:WU06:FS02:0x22:Free Memory: 25.49GiB
11:54:02:WU06:FS02:0x22: Threads: WINDOWS_THREADS
11:54:02:WU06:FS02:0x22: OS Version: 6.2
11:54:02:WU06:FS02:0x22:Has Battery: false
11:54:02:WU06:FS02:0x22: On Battery: false
11:54:02:WU06:FS02:0x22: UTC Offset: -4
11:54:02:WU06:FS02:0x22: PID: 5516
11:54:02:WU06:FS02:0x22: CWD: C:\Users\schap\AppData\Roaming\FAHClient\work
11:54:02:WU06:FS02:0x22: OS: Windows 10 Pro
11:54:02:WU06:FS02:0x22: OS Arch: AMD64
11:54:02:WU06:FS02:0x22:********************************************************************************
11:54:02:WU06:FS02:0x22:Project: 14253 (Run 281, Clone 1, Gen 1)
11:54:02:WU06:FS02:0x22:Unit: 0x00000001cedfaa925eab707f073df835
11:54:02:WU06:FS02:0x22:Reading tar file core.xml
11:54:02:WU06:FS02:0x22:Reading tar file integrator.xml
11:54:02:WU06:FS02:0x22:Reading tar file state.xml
11:54:03:WU06:FS02:0x22:Reading tar file system.xml
11:54:05:WU06:FS02:0x22:Digital signatures verified
11:54:05:WU06:FS02:0x22:Folding@home GPU Core22 Folding@home Core
11:54:05:WU06:FS02:0x22:Version 0.0.5
11:54:10:WU05:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
11:54:11:WU05:FS00:0xa7:Saving result file ..\logfile_01.txt
11:54:11:WU05:FS00:0xa7:Saving result file frame206.trr
11:54:11:WU05:FS00:0xa7:Saving result file frame206.xtc
11:54:11:WU05:FS00:0xa7:Saving result file md.log
11:54:11:WU05:FS00:0xa7:Saving result file science.log
11:54:11:WU05:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
11:54:12:WU05:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:54:12:WU05:FS00:Sending unit results: id:05 state:SEND error:NO_ERROR project:14542 run:0 clone:344 gen:206 core:0xa7 unit:0x000000d180fccb045e7fbfca211044fc
11:54:12:WU05:FS00:Uploading 5.33MiB to 128.252.203.4
11:54:12:WU05:FS00:Connecting to 128.252.203.4:8080
11:54:12:WU03:FS00:Starting
11:54:12:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\schap\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 03 -suffix 01 -version 706 -lifeline 6812 -checkpoint 5 -np 28
11:54:12:WU03:FS00:Started FahCore on PID 3424
11:54:12:WU03:FS00:Core PID:1496
11:54:12:WU03:FS00:FahCore 0xa7 started
11:54:12:WU03:FS00:0xa7:*********************** Log Started 2020-05-05T11:54:12Z ***********************
11:54:12:WU03:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
11:54:12:WU03:FS00:0xa7: Type: 0xa7
11:54:12:WU03:FS00:0xa7: Core: Gromacs
11:54:12:WU03:FS00:0xa7: Args: -dir 03 -suffix 01 -version 706 -lifeline 3424 -checkpoint 5 -np 28
11:54:12:WU03:FS00:0xa7:************************************ CBang *************************************
11:54:12:WU03:FS00:0xa7: Date: Oct 26 2019
11:54:12:WU03:FS00:0xa7: Time: 01:38:25
11:54:12:WU03:FS00:0xa7: Revision: c46a1a011a24143739ac7218c5a435f66777f62f
11:54:12:WU03:FS00:0xa7: Branch: master
11:54:12:WU03:FS00:0xa7: Compiler: Visual C++ 2008
11:54:12:WU03:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:54:12:WU03:FS00:0xa7: Platform: win32 10
11:54:12:WU03:FS00:0xa7: Bits: 64
11:54:12:WU03:FS00:0xa7: Mode: Release
11:54:12:WU03:FS00:0xa7:************************************ System ************************************
11:54:12:WU03:FS00:0xa7: CPU: AMD Ryzen 9 3950X 16-Core Processor
11:54:12:WU03:FS00:0xa7: CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:54:12:WU03:FS00:0xa7: CPUs: 32
11:54:12:WU03:FS00:0xa7: Memory: 31.91GiB
11:54:12:WU03:FS00:0xa7:Free Memory: 25.16GiB
11:54:12:WU03:FS00:0xa7: Threads: WINDOWS_THREADS
11:54:12:WU03:FS00:0xa7: OS Version: 6.2
11:54:12:WU03:FS00:0xa7:Has Battery: false
11:54:12:WU03:FS00:0xa7: On Battery: false
11:54:12:WU03:FS00:0xa7: UTC Offset: -4
11:54:12:WU03:FS00:0xa7: PID: 1496
11:54:12:WU03:FS00:0xa7: CWD: C:\Users\schap\AppData\Roaming\FAHClient\work
11:54:12:WU03:FS00:0xa7:******************************** Build - libFAH ********************************
11:54:12:WU03:FS00:0xa7: Version: 0.0.18
11:54:12:WU03:FS00:0xa7: Author: Joseph Coffland <[email protected]>
11:54:12:WU03:FS00:0xa7: Copyright: 2019 foldingathome.org
11:54:12:WU03:FS00:0xa7: Homepage: https://foldingathome.org/
11:54:12:WU03:FS00:0xa7: Date: Oct 26 2019
11:54:12:WU03:FS00:0xa7: Time: 01:52:30
11:54:12:WU03:FS00:0xa7: Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
11:54:12:WU03:FS00:0xa7: Branch: master
11:54:12:WU03:FS00:0xa7: Compiler: Visual C++ 2008
11:54:12:WU03:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:54:12:WU03:FS00:0xa7: Platform: win32 10
11:54:12:WU03:FS00:0xa7: Bits: 64
11:54:12:WU03:FS00:0xa7: Mode: Release
11:54:12:WU03:FS00:0xa7:************************************ Build *************************************
11:54:12:WU03:FS00:0xa7: SIMD: avx_256
11:54:12:WU03:FS00:0xa7:********************************************************************************
11:54:12:WU03:FS00:0xa7:Project: 14542 (Run 0, Clone 568, Gen 169)
11:54:12:WU03:FS00:0xa7:Unit: 0x000000b280fccb045e7fbfaf1600bbeb
11:54:12:WU03:FS00:0xa7:Reading tar file core.xml
11:54:12:WU03:FS00:0xa7:Reading tar file frame169.tpr
11:54:12:WU03:FS00:0xa7:Digital signatures verified
11:54:12:WU03:FS00:0xa7:Calling: mdrun -s frame169.tpr -o frame169.trr -x frame169.xtc -cpt 5 -nt 28
11:54:12:WU03:FS00:0xa7:Steps: first=84500000 total=500000
11:54:13:WU03:FS00:0xa7:Completed 1 out of 500000 steps (0%)
11:54:14:WU05:FS00:Upload complete
11:54:14:WU05:FS00:Server responded WORK_ACK (400)
11:54:14:WU05:FS00:Final credit estimate, 20933.00 points
11:54:14:WU05:FS00:Cleaning up
11:54:55:WU06:FS02:0x22:Completed 0 out of 500000 steps (0%)
11:54:55:WU06:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:54:59:WU03:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
11:55:24:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
11:55:42:WU03:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
11:55:54:WU06:FS02:0x22:Completed 5000 out of 500000 steps (1%)
11:56:26:WU03:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
11:56:49:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
11:56:51:WU06:FS02:0x22:Completed 10000 out of 500000 steps (2%)
11:57:09:WU03:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
11:57:48:WU06:FS02:0x22:Completed 15000 out of 500000 steps (3%)
11:57:53:WU03:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
11:58:14:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
11:58:37:WU03:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
11:58:45:WU06:FS02:0x22:Completed 20000 out of 500000 steps (4%)
11:59:22:WU03:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
11:59:51:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
11:59:57:WU06:FS02:0x22:Completed 25000 out of 500000 steps (5%)
12:00:05:WU03:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
12:00:49:WU03:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
12:00:54:WU06:FS02:0x22:Completed 30000 out of 500000 steps (6%)
12:01:16:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
12:01:33:WU03:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
12:01:51:WU06:FS02:0x22:Completed 35000 out of 500000 steps (7%)
12:02:16:WU03:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
12:02:41:WU02:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
12:02:48:WU06:FS02:0x22:Completed 40000 out of 500000 steps (8%)
12:03:00:WU03:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
12:03:44:WU03:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
12:04:00:WU06:FS02:0x22:Completed 45000 out of 500000 steps (9%)
12:04:06:WU02:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
12:04:28:WU03:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
12:04:43:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
12:04:43:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
12:04:57:WU06:FS02:0x22:Completed 50000 out of 500000 steps (10%)
12:05:11:WU03:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
12:05:54:WU06:FS02:0x22:Completed 55000 out of 500000 steps (11%)
12:05:55:WU03:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
12:06:09:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
12:06:39:WU03:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
12:06:51:WU06:FS02:0x22:Completed 60000 out of 500000 steps (12%)
12:07:00:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:14253 run:1831 clone:0 gen:0 core:0x22 unit:0x00000001cedfaa925eaba95ca9c9bccc
12:07:00:WU04:FS01:Uploading 261.61MiB to 206.223.170.146
12:07:00:WU04:FS01:Connecting to 206.223.170.146:8080
12:07:01:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14253 run:5 clone:0 gen:13 core:0x22 unit:0x0000000ecedfaa925eab7041daa04117
12:07:01:WU00:FS01:Uploading 279.22MiB to 206.223.170.146
12:07:01:WU00:FS01:Connecting to 206.223.170.146:8080
12:07:08:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14253 run:39 clone:0 gen:7 core:0x22 unit:0x0000000acedfaa925eab703e05ef414f
12:07:08:WU01:FS01:Uploading 261.61MiB to 206.223.170.146
12:07:08:WU01:FS01:Connecting to 206.223.170.146:8080
12:07:16:WU00:FS01:Upload 0.02%
12:07:21:WARNING:WU04:FS01:WorkServer connection failed on port 8080 trying 80
12:07:21:WU04:FS01:Connecting to 206.223.170.146:80
12:07:23:WU03:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
12:07:24:WU01:FS01:Upload 0.05%
12:07:36:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
12:07:36:WU04:FS01:Upload 0.02%
12:08:03:WU06:FS02:0x22:Completed 65000 out of 500000 steps (13%)
12:08:07:WU03:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
12:08:13:WU00:FS01:Upload 0.07%
12:08:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:08:13:WU00:FS01:Trying to send results to collection server
12:08:13:WU00:FS01:Uploading 279.22MiB to 52.224.109.74
12:08:13:WU00:FS01:Connecting to 52.224.109.74:8080
12:08:14:ERROR:WU00:FS01:Exception: Transfer failed
12:08:28:WU01:FS01:Upload 0.07%
12:08:28:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
12:08:28:WU01:FS01:Trying to send results to collection server
12:08:28:WU01:FS01:Uploading 261.61MiB to 52.224.109.74
12:08:28:WU01:FS01:Connecting to 52.224.109.74:8080
12:08:28:ERROR:WU01:FS01:Exception: Transfer failed
12:08:33:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
12:08:33:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
12:08:44:WU04:FS01:Upload 0.07%
12:08:44:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
12:08:44:WU04:FS01:Trying to send results to collection server
12:08:44:WU04:FS01:Uploading 261.61MiB to 52.224.109.74
12:08:44:WU04:FS01:Connecting to 52.224.109.74:8080
12:08:44:ERROR:WU04:FS01:Exception: Transfer failed
12:08:50:WU03:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
12:09:00:WU06:FS02:0x22:Completed 70000 out of 500000 steps (14%)
12:09:34:WU03:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
12:09:57:WU06:FS02:0x22:Completed 75000 out of 500000 steps (15%)
12:09:58:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
12:10:19:WU03:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
12:10:54:WU06:FS02:0x22:Completed 80000 out of 500000 steps (16%)
12:11:04:WU03:FS00:0xa7:Completed 115000 out of 500000 steps (23%)
12:11:27:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
12:11:48:WU03:FS00:0xa7:Completed 120000 out of 500000 steps (24%)
12:12:07:WU06:FS02:0x22:Completed 85000 out of 500000 steps (17%)
anandhanju wrote:@schapman1978, can you post the section from the log where the two WUs completed? I'm curious to know at what point they aborted as the WU status has "error:FAULTY" identification when uploading the results.
I had restarted to try to get things rolling again before I realized there was a bigger problem. I don't have the point in the log where that could have occurred. I'm running another one now and at 7% it gave the message about "is your system overclocked?" and bad state. I think this WU is problematic - but my equipment isn't overclocked, not even PBO on the CPU and I've put my cards from a mild overclock on water to stock just to test it all out. They produce much better overclocked but even overclocked they never had issues with workunits erroring out.
If its a hardware issue it still doesn't explain why none of the units will upload though I don't think. I've got three sitting there right now waiting, and when this next one is finished I bet it'll be a fourth. Hmm
Ok... something clearly happened to these three units and they're not going to upload or be accepted. How do I dump all these workunits without dumping stuff that is working.
Can someone link the tag line to dump specific WU's for me so I don't screw this up please?
RIP 600k of WU's
UPDATE - I couldn't use the FAHClient --dump XX - it would show executed in console but never removed from GUI. I found them in appdata/roaming and deleted the three "FAULTY" folders and cleared out the mess. No assistance required.
I was able to complete a 14253 workunit and upload it (the 4th unit) and for what ever reason it didn't error out so if it has a tagline of "FAULTY" it might as well get manually deleted I guess?
Normally, faulty WU completions are sent back to the server so that it can be sent to another user to be retried. Most (>99%) of these failures are due to some instability in the computer and WUs that are faulty in one will typically complete in another one. However, there are rare occurrences where the WU itself is bad and the server will blacklist the WU from being sent out if x number of folders report faulty completions.
The thing with faulty completions is that the results file is larger than usual -- in your case, 260MB, while normal results are around 130MB. The server that accepts the results may not be willing to accept such large faulty ones. The new COVID-19 projects are huge and the researchers are looking into the best way to address this.
Dumping your faulty results is fine in this case and what you've done is correct. Removing the specific numbered directories from the FAH work folder (e.g., 00, 01, 02 depending on the ones that contain the faulty results) and restarting the client will cause the client to dump that WU. If you hadn't done this, the client would've kept trying and dumped it after the WU passed the deadline in a few days.
Lastly, faulty returns are not worth much. You do not get bonus points which form a bulk of the points for GPU folding and only receive partial points prorated until the point of failure, if the server is able to determine the % complete from the uploaded results.
Thanks for the reply - I’m glad I’m back up and running. I’ve only started folding again in the last several weeks after ten years away from it - I’m sure there’s lots for me to learn still!
