faulty WU project:13414 run:3839 clone:50 gen:0

[aetch]

It came into my system crashed twice without reaching 1% and got dumped faulty.

It seems I wasn't the first or the last to dump it faulty.

Code: Select all

22:27:18:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13414 run:3839 clone:50 gen:0 core:0x22 unit:0x0000000112bc7d9a5ef50d5cd4c8dba1
22:27:18:WU01:FS01:Starting
22:27:18:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Helen\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 16684 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
22:27:18:WU01:FS01:Started FahCore on PID 13940
22:27:18:WU01:FS01:Core PID:17616
22:27:18:WU01:FS01:FahCore 0x22 started
22:27:19:WU01:FS01:0x22:*********************** Log Started 2020-06-27T22:27:18Z ***********************
22:27:19:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:27:19:WU01:FS01:0x22:       Core: Core22
22:27:19:WU01:FS01:0x22:       Type: 0x22
22:27:19:WU01:FS01:0x22:    Version: 0.0.10
22:27:19:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
22:27:19:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
22:27:19:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
22:27:19:WU01:FS01:0x22:       Date: Jun 16 2020
22:27:19:WU01:FS01:0x22:       Time: 14:33:22
22:27:19:WU01:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
22:27:19:WU01:FS01:0x22:     Branch: core22-0.0.10
22:27:19:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:19:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:19:WU01:FS01:0x22:   Platform: win32 10
22:27:19:WU01:FS01:0x22:       Bits: 64
22:27:19:WU01:FS01:0x22:       Mode: Release
22:27:19:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
22:27:19:WU01:FS01:0x22:             <[email protected]>
22:27:19:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 13940 -checkpoint 15
22:27:19:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
22:27:19:WU01:FS01:0x22:             0 -gpu 0
22:27:19:WU01:FS01:0x22:************************************ libFAH ************************************
22:27:19:WU01:FS01:0x22:       Date: Jun 15 2020
22:27:19:WU01:FS01:0x22:       Time: 18:05:04
22:27:19:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
22:27:19:WU01:FS01:0x22:     Branch: HEAD
22:27:19:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:19:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:19:WU01:FS01:0x22:   Platform: win32 10
22:27:19:WU01:FS01:0x22:       Bits: 64
22:27:19:WU01:FS01:0x22:       Mode: Release
22:27:19:WU01:FS01:0x22:************************************ CBang *************************************
22:27:19:WU01:FS01:0x22:       Date: Jun 16 2020
22:27:19:WU01:FS01:0x22:       Time: 14:31:33
22:27:19:WU01:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
22:27:19:WU01:FS01:0x22:     Branch: HEAD
22:27:19:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:19:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:19:WU01:FS01:0x22:   Platform: win32 10
22:27:19:WU01:FS01:0x22:       Bits: 64
22:27:19:WU01:FS01:0x22:       Mode: Release
22:27:19:WU01:FS01:0x22:************************************ System ************************************
22:27:19:WU01:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:27:19:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:27:19:WU01:FS01:0x22:       CPUs: 24
22:27:19:WU01:FS01:0x22:     Memory: 15.95GiB
22:27:19:WU01:FS01:0x22:Free Memory: 12.41GiB
22:27:19:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
22:27:19:WU01:FS01:0x22: OS Version: 6.2
22:27:19:WU01:FS01:0x22:Has Battery: false
22:27:19:WU01:FS01:0x22: On Battery: false
22:27:19:WU01:FS01:0x22: UTC Offset: 1
22:27:19:WU01:FS01:0x22:        PID: 17616
22:27:19:WU01:FS01:0x22:        CWD: C:\Users\Helen\AppData\Roaming\FAHClient\work
22:27:19:WU01:FS01:0x22:********************************************************************************
22:27:19:WU01:FS01:0x22:Project: 13414 (Run 3839, Clone 50, Gen 0)
22:27:19:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5ef50d5cd4c8dba1
22:27:19:WU01:FS01:0x22:Reading tar file core.xml
22:27:19:WU01:FS01:0x22:Reading tar file integrator.xml
22:27:19:WU01:FS01:0x22:Reading tar file state.xml
22:27:19:WU01:FS01:0x22:Reading tar file system.xml
22:27:19:WU01:FS01:0x22:Digital signatures verified
22:27:19:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:27:19:WU01:FS01:0x22:Version 0.0.10
22:27:19:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
22:27:19:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:27:19:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
22:27:19:WU01:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
22:27:34:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
22:27:45:WU01:FS01:0x22:An exception occurred at step 250: Particle coordinate is nan
22:27:45:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
22:27:45:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
22:27:46:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
22:27:46:WU01:FS01:Starting
22:27:46:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Helen\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 16684 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
22:27:46:WU01:FS01:Started FahCore on PID 14092
22:27:46:WU01:FS01:Core PID:14216
22:27:46:WU01:FS01:FahCore 0x22 started
22:27:47:WU01:FS01:0x22:*********************** Log Started 2020-06-27T22:27:47Z ***********************
22:27:47:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:27:47:WU01:FS01:0x22:       Core: Core22
22:27:47:WU01:FS01:0x22:       Type: 0x22
22:27:47:WU01:FS01:0x22:    Version: 0.0.10
22:27:47:WU01:FS01:0x22:     Author: Joseph Coffland <[email protected]>
22:27:47:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
22:27:47:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
22:27:47:WU01:FS01:0x22:       Date: Jun 16 2020
22:27:47:WU01:FS01:0x22:       Time: 14:33:22
22:27:47:WU01:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
22:27:47:WU01:FS01:0x22:     Branch: core22-0.0.10
22:27:47:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:47:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:47:WU01:FS01:0x22:   Platform: win32 10
22:27:47:WU01:FS01:0x22:       Bits: 64
22:27:47:WU01:FS01:0x22:       Mode: Release
22:27:47:WU01:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
22:27:47:WU01:FS01:0x22:             <[email protected]>
22:27:47:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14092 -checkpoint 15
22:27:47:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
22:27:47:WU01:FS01:0x22:             0 -gpu 0
22:27:47:WU01:FS01:0x22:************************************ libFAH ************************************
22:27:47:WU01:FS01:0x22:       Date: Jun 15 2020
22:27:47:WU01:FS01:0x22:       Time: 18:05:04
22:27:47:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
22:27:47:WU01:FS01:0x22:     Branch: HEAD
22:27:47:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:47:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:47:WU01:FS01:0x22:   Platform: win32 10
22:27:47:WU01:FS01:0x22:       Bits: 64
22:27:47:WU01:FS01:0x22:       Mode: Release
22:27:47:WU01:FS01:0x22:************************************ CBang *************************************
22:27:47:WU01:FS01:0x22:       Date: Jun 16 2020
22:27:47:WU01:FS01:0x22:       Time: 14:31:33
22:27:47:WU01:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
22:27:47:WU01:FS01:0x22:     Branch: HEAD
22:27:47:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:27:47:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:27:47:WU01:FS01:0x22:   Platform: win32 10
22:27:47:WU01:FS01:0x22:       Bits: 64
22:27:47:WU01:FS01:0x22:       Mode: Release
22:27:47:WU01:FS01:0x22:************************************ System ************************************
22:27:47:WU01:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:27:47:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:27:47:WU01:FS01:0x22:       CPUs: 24
22:27:47:WU01:FS01:0x22:     Memory: 15.95GiB
22:27:47:WU01:FS01:0x22:Free Memory: 12.41GiB
22:27:47:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
22:27:47:WU01:FS01:0x22: OS Version: 6.2
22:27:47:WU01:FS01:0x22:Has Battery: false
22:27:47:WU01:FS01:0x22: On Battery: false
22:27:47:WU01:FS01:0x22: UTC Offset: 1
22:27:47:WU01:FS01:0x22:        PID: 14216
22:27:47:WU01:FS01:0x22:        CWD: C:\Users\Helen\AppData\Roaming\FAHClient\work
22:27:47:WU01:FS01:0x22:********************************************************************************
22:27:47:WU01:FS01:0x22:Project: 13414 (Run 3839, Clone 50, Gen 0)
22:27:47:WU01:FS01:0x22:Unit: 0x0000000112bc7d9a5ef50d5cd4c8dba1
22:27:47:WU01:FS01:0x22:Digital signatures verified
22:27:47:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:27:47:WU01:FS01:0x22:Version 0.0.10
22:27:47:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
22:27:47:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:27:47:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
22:27:47:WU01:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
22:27:54:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
22:27:59:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
22:28:10:WU01:FS01:0x22:An exception occurred at step 250: Particle coordinate is nan
22:28:10:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
22:28:10:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
22:28:10:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
22:28:10:WU01:FS01:0x22:Saving result file globals.csv
22:28:10:WU01:FS01:0x22:Saving result file science.log
22:28:10:WU01:FS01:0x22:Saving result file state.xml
22:28:11:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
22:28:11:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
22:28:11:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13414 run:3839 clone:50 gen:0 core:0x22 unit:0x0000000112bc7d9a5ef50d5cd4c8dba1
22:28:11:WU01:FS01:Uploading 7.09MiB to 18.188.125.154
22:28:11:WU01:FS01:Connecting to 18.188.125.154:8080
22:28:12:WU02:FS01:Connecting to assign1.foldingathome.org:80
22:28:12:WU02:FS01:Assigned to work server 128.252.203.1
22:28:12:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:TU104 [GeForce RTX 2070 SUPER] 8218 from 128.252.203.1
22:28:12:WU02:FS01:Connecting to 128.252.203.1:8080
22:28:17:WU01:FS01:Upload 34.36%
22:28:21:WU02:FS01:Downloading 24.10MiB
22:28:23:WU01:FS01:Upload 74.00%
22:28:27:WU02:FS01:Download 3.89%
22:28:27:WU01:FS01:Upload complete
22:28:27:WU01:FS01:Server responded WORK_ACK (400)
22:28:27:WU01:FS01:Cleaning up

[bruce]

All projects have an osccasional bad WU and it looks like you got one. Project 13414 does have several. The error report that was uploaded:
SEND error:FAULTY project:13414 run:3839 clone:50 gen:0 will be examined for an explanation and likely will lead to an improvement either in the construction of future WUs or to an improvement in FAHCore_22.

[JohnChodera]

We're working on reducing the failure rates for the experimental 134xx series (which uses a new kind of nonequilibrium simulation workload) supporting the COVID Moonshot (http://postera.ai/covid), which helps us prioritize compounds for synthesis to develop a new therapy for COVID-19. Each of these pairs of projects only runs for a few days, and we make systematic improvements each iteration.

We *do* test these thoroughly on our local GPU cluster before putting them on FAH, but there are still some failures when they make their way out onto the wide variety of GPU hardware on Folding@home. We're working on core22 improvements that will bring back more information about failures so we can increase robustness. We do analyze any failures that are uploaded back to the work servers, so no need to report the failures unless they are NOT uploaded for some reason. We really need to hear about the failures that do NOT make it back to the work server.

Thanks again for bearing with us!

~ John Chodera // MSKCC

[aetch]

No worries.
I didn't know what was collected from faulty/failed work units and I know that you'd rather have more info than less, having down some software development myself (that's not an offer, I'm not that good).

[bruce]

When the latest code is successfully returning quite a bit of completed work and we have a collection of really challenging WUs that have been crashing, it' good to let some crash as long as the error reports are reaching the development team. Don't assume that failures are, in fact, failures.

[zookeeny]

If it helps, here's the log file for a 13414 unit that sat for 13 hours doing nothing until I dumped it. I have an RX 5700 XT running under Pop_OS Linux:

Code: Select all

00:25:00:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13414 run:3534 clone:50 gen:1 core:0x22 unit:0x0000000512bc7d9a5ef50d675ea5a9cf
00:25:04:WU00:FS01:FahCore 0x22 started
00:25:04:WU00:FS01:0x22:*********************** Log Started 2020-06-29T00:25:04Z ***********************
00:25:04:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
00:25:04:WU00:FS01:0x22:       Core: Core22
00:25:04:WU00:FS01:0x22:       Type: 0x22
00:25:04:WU00:FS01:0x22:    Version: 0.0.10
00:25:04:WU00:FS01:0x22:     Author: Joseph Coffland <[email protected]>
00:25:04:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
00:25:04:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
00:25:04:WU00:FS01:0x22:       Date: Jun 16 2020
00:25:04:WU00:FS01:0x22:       Time: 15:55:31
00:25:04:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
00:25:04:WU00:FS01:0x22:     Branch: core22-0.0.10
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
00:25:04:WU00:FS01:0x22:             <[email protected]>
00:25:04:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 25434 -checkpoint 15
00:25:04:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
00:25:04:WU00:FS01:0x22:************************************ libFAH ************************************
00:25:04:WU00:FS01:0x22:       Date: Jun 2 2020
00:25:04:WU00:FS01:0x22:       Time: 00:07:31
00:25:04:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
00:25:04:WU00:FS01:0x22:     Branch: HEAD
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:************************************ CBang *************************************
00:25:04:WU00:FS01:0x22:       Date: May 31 2020
00:25:04:WU00:FS01:0x22:       Time: 20:16:34
00:25:04:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
00:25:04:WU00:FS01:0x22:     Branch: HEAD
00:25:04:WU00:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
00:25:04:WU00:FS01:0x22:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
00:25:04:WU00:FS01:0x22:             -fPIC
00:25:04:WU00:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
00:25:04:WU00:FS01:0x22:       Bits: 64
00:25:04:WU00:FS01:0x22:       Mode: Release
00:25:04:WU00:FS01:0x22:************************************ System ************************************
00:25:04:WU00:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
00:25:04:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
00:25:04:WU00:FS01:0x22:       CPUs: 24
00:25:04:WU00:FS01:0x22:     Memory: 31.37GiB
00:25:04:WU00:FS01:0x22:Free Memory: 24.47GiB
00:25:04:WU00:FS01:0x22:    Threads: POSIX_THREADS
00:25:04:WU00:FS01:0x22: OS Version: 5.4
00:25:04:WU00:FS01:0x22:Has Battery: false
00:25:04:WU00:FS01:0x22: On Battery: false
00:25:04:WU00:FS01:0x22: UTC Offset: -4
00:25:04:WU00:FS01:0x22:        PID: 25438
00:25:04:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
00:25:04:WU00:FS01:0x22:********************************************************************************
00:25:04:WU00:FS01:0x22:Project: 13414 (Run 3534, Clone 50, Gen 1)
00:25:04:WU00:FS01:0x22:Unit: 0x0000000512bc7d9a5ef50d675ea5a9cf
00:25:04:WU00:FS01:0x22:Reading tar file core.xml
00:25:04:WU00:FS01:0x22:Reading tar file integrator.xml
00:25:04:WU00:FS01:0x22:Reading tar file state.xml
00:25:04:WU00:FS01:0x22:Reading tar file system.xml
00:25:04:WU00:FS01:0x22:Digital signatures verified
00:25:04:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
00:25:04:WU00:FS01:0x22:Version 0.0.10
00:25:05:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
00:25:05:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
00:25:05:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
00:25:05:WU00:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
13:12:12:WU00:FS01:0x22:Caught signal SIGINT(2) on PID 25438
13:12:12:WU00:FS01:0x22:Exiting, please wait. . .
13:13:14:WU00:Sending unit results: id:00 state:SEND error:DUMPED project:13414 run:3534 clone:50 gen:1 core:0x22 unit:0x0000000512bc7d9a5ef50d675ea5a9cf