two WUs in « Cleanup »
Posted: Sat Mar 28, 2020 1:39 pm
I've been folding for 10+ years and never seen this before.
I have two WUs in « Cleanup » Status shown in FAHControl (v7.5.1. OSX 10.13.6).
To me it looks like :
- It finished and eventually sent in p14538-r0-c239-g6 (WU00).
- It downloaded p14539-r0-c480-g2 and it failed as soon as it started (WU01).
- It downloaded p14538-r0-c318-g11 and now folds successfully (WU03).
- It's trying to cleanup the previous two and i's not making it (WU00 & WU01).
BTW, I have had several p14539 fail on both my (old) GPUs.
I have two WUs in « Cleanup » Status shown in FAHControl (v7.5.1. OSX 10.13.6).
Code: Select all
12:04:23:WU00:FS02:0x22:Completed 3960000 out of 4000000 steps (99%)
12:04:24:WU01:FS02:Connecting to assign-GPU.stanford.edu:80
12:04:24:WU01:FS02:News:
12:04:24:WU01:FS02:Assigned to work server 13.82.98.119
12:04:24:WU01:FS02:Requesting new work unit for slot 02: RUNNING gpu:2:Baffin [Polaris11] from 13.82.98.119
12:04:24:WU01:FS02:Connecting to 13.82.98.119:8080
12:04:25:WU01:FS02:Downloading 119.45MiB
12:04:31:WU01:FS02:Download 18.84%
12:04:37:WU01:FS02:Download 37.52%
12:04:43:WU01:FS02:Download 57.71%
12:04:49:WU01:FS02:Download 77.02%
12:04:55:WU01:FS02:Download 95.81%
12:04:56:WU01:FS02:Download complete
12:04:56:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14539 run:0 clone:480 gen:2 core:0x22 unit:0x000000020d5262775e7bf1fa8ea17d8e
12:07:28:WU00:FS02:0x22:Completed 4000000 out of 4000000 steps (100%)
12:07:29:WU00:FS02:0x22:Saving result file ..\logfile_01.txt
12:07:29:WU00:FS02:0x22:Saving result file checkpointState.xml
12:07:29:WU00:FS02:0x22:Saving result file checkpt.crc
12:07:29:WU00:FS02:0x22:Saving result file positions.xtc
12:07:29:WU00:FS02:0x22:Saving result file science.log
12:07:29:WU00:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
12:07:29:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:07:29:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:14538 run:0 clone:359 gen:6 core:0x22 unit:0x000000070d5262775e7b9cdf6ab52b24
12:07:29:WU00:FS02:Uploading 14.70MiB to 13.82.98.119
12:07:29:WU00:FS02:Connecting to 13.82.98.119:8080
12:07:29:WU01:FS02:Starting
12:07:29:WU01:FS02:Running FahCore: C:\Users\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Folding/FAHClient/cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 703 -lifeline 5008 -checkpoint 5 -gpu 1 -gpu-vendor ati
12:07:29:WU01:FS02:Started FahCore on PID 2332
12:07:29:WU01:FS02:Core PID:6276
12:07:29:WU01:FS02:FahCore 0x22 started
12:07:30:WU01:FS02:0x22:*********************** Log Started 2020-03-28T12:07:30Z ***********************
12:07:30:WU01:FS02:0x22:*************************** Core22 Folding@home Core ***************************
12:07:30:WU01:FS02:0x22: Type: 0x22
12:07:30:WU01:FS02:0x22: Core: Core22
12:07:30:WU01:FS02:0x22: Website: https://foldingathome.org/
12:07:30:WU01:FS02:0x22: Copyright: (c) 2009-2018 foldingathome.org
12:07:30:WU01:FS02:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
12:07:30:WU01:FS02:0x22: <[email protected]>
12:07:30:WU01:FS02:0x22: Args: -dir 01 -suffix 01 -version 703 -lifeline 2332 -checkpoint 5 -gpu 1
12:07:30:WU01:FS02:0x22: -gpu-vendor ati
12:07:30:WU01:FS02:0x22: Config: <none>
12:07:30:WU01:FS02:0x22:************************************ Build *************************************
12:07:30:WU01:FS02:0x22: Version: 0.0.2
12:07:30:WU01:FS02:0x22: Date: Dec 6 2019
12:07:30:WU01:FS02:0x22: Time: 21:30:31
12:07:30:WU01:FS02:0x22: Repository: Git
12:07:30:WU01:FS02:0x22: Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
12:07:30:WU01:FS02:0x22: Branch: HEAD
12:07:30:WU01:FS02:0x22: Compiler: Visual C++ 2008
12:07:30:WU01:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:07:30:WU01:FS02:0x22: Platform: win32 10
12:07:30:WU01:FS02:0x22: Bits: 64
12:07:30:WU01:FS02:0x22: Mode: Release
12:07:30:WU01:FS02:0x22:************************************ System ************************************
12:07:30:WU01:FS02:0x22: CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
12:07:30:WU01:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
12:07:30:WU01:FS02:0x22: CPUs: 12
12:07:30:WU01:FS02:0x22: Memory: 12.00GiB
12:07:30:WU01:FS02:0x22:Free Memory: 8.14GiB
12:07:30:WU01:FS02:0x22: Threads: WINDOWS_THREADS
12:07:30:WU01:FS02:0x22: OS Version: 6.1
12:07:30:WU01:FS02:0x22:Has Battery: false
12:07:30:WU01:FS02:0x22: On Battery: false
12:07:30:WU01:FS02:0x22: UTC Offset: -4
12:07:30:WU01:FS02:0x22: PID: 6276
12:07:30:WU01:FS02:0x22: CWD: C:\Users\Folding\FAHClient\work
12:07:30:WU01:FS02:0x22: OS: Windows 7 Ultimate Service Pack 1
12:07:30:WU01:FS02:0x22: OS Arch: AMD64
12:07:30:WU01:FS02:0x22:********************************************************************************
12:07:30:WU01:FS02:0x22:Project: 14539 (Run 0, Clone 480, Gen 2)
12:07:30:WU01:FS02:0x22:Unit: 0x000000020d5262775e7bf1fa8ea17d8e
12:07:30:WU01:FS02:0x22:Reading tar file core.xml
12:07:30:WU01:FS02:0x22:Reading tar file integrator.xml
12:07:30:WU01:FS02:0x22:Reading tar file state.xml
12:07:31:WU01:FS02:0x22:Reading tar file system.xml
12:07:33:WU01:FS02:0x22:Digital signatures verified
12:07:33:WU01:FS02:0x22:Folding@home GPU Core22 Folding@home Core
12:07:33:WU01:FS02:0x22:Version 0.0.2
12:07:35:WU00:FS02:Upload 15.74%
12:07:41:WU00:FS02:Upload 33.60%
12:07:47:WU00:FS02:Upload 45.08%
12:07:52:WU01:FS02:0x22:ERROR:exception: Error invoking kernel sortShortList: clEnqueueNDRangeKernel (-5)
12:07:52:WU01:FS02:0x22:Saving result file ..\logfile_01.txt
12:07:52:WU01:FS02:0x22:Saving result file science.log
12:07:52:WU01:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
12:07:53:WU00:FS02:Upload 55.71%
12:07:53:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
12:07:53:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:14539 run:0 clone:480 gen:2 core:0x22 unit:0x000000020d5262775e7bf1fa8ea17d8e
12:07:53:WU01:FS02:Uploading 8.00KiB to 13.82.98.119
12:07:53:WU01:FS02:Connecting to 13.82.98.119:8080
12:07:53:WU01:FS02:Upload complete
12:07:53:WU01:FS02:Server responded WORK_ACK (400)
12:07:53:WU01:FS02:Cleaning up
12:07:53:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:07:53:WU03:FS02:Connecting to assign-GPU.stanford.edu:80
12:07:53:WU01:FS02:Cleaning up
12:07:53:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:07:54:WU03:FS02:News:
12:07:54:WU03:FS02:Assigned to work server 13.82.98.119
12:07:54:WU03:FS02:Requesting new work unit for slot 02: READY gpu:2:Baffin [Polaris11] from 13.82.98.119
12:07:54:WU03:FS02:Connecting to 13.82.98.119:8080
12:07:54:WU03:FS02:Downloading 13.26MiB
12:07:59:WU00:FS02:Upload 64.22%
12:07:59:WU03:FS02:Download complete
12:07:59:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14538 run:0 clone:318 gen:11 core:0x22 unit:0x0000000b0d5262775e7b9cdf546b235e
12:07:59:WU03:FS02:Starting
12:07:59:WU03:FS02:Running FahCore: C:\Users\Folding\FAHClient/FAHCoreWrapper.exe C:/Users/Folding/FAHClient/cores/cores.foldingathome.org/v7/win/64bit/beta/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 703 -lifeline 5008 -checkpoint 5 -gpu 1 -gpu-vendor ati
12:07:59:WU03:FS02:Started FahCore on PID 8644
12:07:59:WU03:FS02:Core PID:6460
12:07:59:WU03:FS02:FahCore 0x22 started
12:07:59:WU03:FS02:0x22:*********************** Log Started 2020-03-28T12:07:59Z ***********************
12:07:59:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
12:07:59:WU03:FS02:0x22: Type: 0x22
12:07:59:WU03:FS02:0x22: Core: Core22
12:07:59:WU03:FS02:0x22: Website: https://foldingathome.org/
12:07:59:WU03:FS02:0x22: Copyright: (c) 2009-2018 foldingathome.org
12:07:59:WU03:FS02:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
12:07:59:WU03:FS02:0x22: <[email protected]>
12:07:59:WU03:FS02:0x22: Args: -dir 03 -suffix 01 -version 703 -lifeline 8644 -checkpoint 5 -gpu 1
12:07:59:WU03:FS02:0x22: -gpu-vendor ati
12:07:59:WU03:FS02:0x22: Config: <none>
12:07:59:WU03:FS02:0x22:************************************ Build *************************************
12:07:59:WU03:FS02:0x22: Version: 0.0.2
12:07:59:WU03:FS02:0x22: Date: Dec 6 2019
12:07:59:WU03:FS02:0x22: Time: 21:30:31
12:07:59:WU03:FS02:0x22: Repository: Git
12:07:59:WU03:FS02:0x22: Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
12:07:59:WU03:FS02:0x22: Branch: HEAD
12:07:59:WU03:FS02:0x22: Compiler: Visual C++ 2008
12:07:59:WU03:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:07:59:WU03:FS02:0x22: Platform: win32 10
12:07:59:WU03:FS02:0x22: Bits: 64
12:07:59:WU03:FS02:0x22: Mode: Release
12:07:59:WU03:FS02:0x22:************************************ System ************************************
12:07:59:WU03:FS02:0x22: CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
12:07:59:WU03:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
12:07:59:WU03:FS02:0x22: CPUs: 12
12:07:59:WU03:FS02:0x22: Memory: 12.00GiB
12:07:59:WU03:FS02:0x22:Free Memory: 7.99GiB
12:07:59:WU03:FS02:0x22: Threads: WINDOWS_THREADS
12:07:59:WU03:FS02:0x22: OS Version: 6.1
12:07:59:WU03:FS02:0x22:Has Battery: false
12:07:59:WU03:FS02:0x22: On Battery: false
12:07:59:WU03:FS02:0x22: UTC Offset: -4
12:07:59:WU03:FS02:0x22: PID: 6460
12:07:59:WU03:FS02:0x22: CWD: C:\Users\Folding\FAHClient\work
12:07:59:WU03:FS02:0x22: OS: Windows 7 Ultimate Service Pack 1
12:07:59:WU03:FS02:0x22: OS Arch: AMD64
12:07:59:WU03:FS02:0x22:********************************************************************************
12:07:59:WU03:FS02:0x22:Project: 14538 (Run 0, Clone 318, Gen 11)
12:07:59:WU03:FS02:0x22:Unit: 0x0000000b0d5262775e7b9cdf546b235e
12:07:59:WU03:FS02:0x22:Reading tar file core.xml
12:07:59:WU03:FS02:0x22:Reading tar file integrator.xml
12:07:59:WU03:FS02:0x22:Reading tar file state.xml
12:07:59:WU03:FS02:0x22:Reading tar file system.xml
12:07:59:WU03:FS02:0x22:Digital signatures verified
12:07:59:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
12:07:59:WU03:FS02:0x22:Version 0.0.2
12:08:05:WU00:FS02:Upload 75.27%
12:08:06:WU03:FS02:0x22:Completed 0 out of 4000000 steps (0%)
12:08:06:WU03:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:08:11:WU00:FS02:Upload 86.33%
12:08:17:WU00:FS02:Upload 97.39%
12:08:18:WU00:FS02:Upload complete
12:08:18:WU00:FS02:Server responded WORK_ACK (400)
12:08:18:WU00:FS02:Final credit estimate, 27958.00 points
12:08:18:WU00:FS02:Cleaning up
12:08:18:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:08:19:WU00:FS02:Cleaning up
12:08:19:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:08:54:WU01:FS02:Cleaning up
12:08:54:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:09:19:WU00:FS02:Cleaning up
12:09:19:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:10:31:WU01:FS02:Cleaning up
12:10:31:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:10:56:WU00:FS02:Cleaning up
12:10:56:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:11:12:WU03:FS02:0x22:Completed 40000 out of 4000000 steps (1%)
12:13:08:WU01:FS02:Cleaning up
12:13:08:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:13:33:WU00:FS02:Cleaning up
12:13:33:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:14:18:WU03:FS02:0x22:Completed 80000 out of 4000000 steps (2%)
12:17:22:WU01:FS02:Cleaning up
12:17:22:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:17:23:WU03:FS02:0x22:Completed 120000 out of 4000000 steps (3%)
12:17:47:WU00:FS02:Cleaning up
12:17:47:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:20:28:WU03:FS02:0x22:Completed 160000 out of 4000000 steps (4%)
12:23:33:WU03:FS02:0x22:Completed 200000 out of 4000000 steps (5%)
12:24:14:WU01:FS02:Cleaning up
12:24:14:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:24:39:WU00:FS02:Cleaning up
12:24:39:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:26:39:WU03:FS02:0x22:Completed 240000 out of 4000000 steps (6%)
12:29:45:WU03:FS02:0x22:Completed 280000 out of 4000000 steps (7%)
12:32:50:WU03:FS02:0x22:Completed 320000 out of 4000000 steps (8%)
12:35:19:WU01:FS02:Cleaning up
12:35:19:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:35:44:WU00:FS02:Cleaning up
12:35:44:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:35:56:WU03:FS02:0x22:Completed 360000 out of 4000000 steps (9%)
12:39:01:WU03:FS02:0x22:Completed 400000 out of 4000000 steps (10%)
12:42:06:WU03:FS02:0x22:Completed 440000 out of 4000000 steps (11%)
12:45:12:WU03:FS02:0x22:Completed 480000 out of 4000000 steps (12%)
12:48:17:WU03:FS02:0x22:Completed 520000 out of 4000000 steps (13%)
12:51:23:WU03:FS02:0x22:Completed 560000 out of 4000000 steps (14%)
12:53:16:WU01:FS02:Cleaning up
12:53:16:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
12:53:41:WU00:FS02:Cleaning up
12:53:41:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
12:54:28:WU03:FS02:0x22:Completed 600000 out of 4000000 steps (15%)
12:57:33:WU03:FS02:0x22:Completed 640000 out of 4000000 steps (16%)
13:00:39:WU03:FS02:0x22:Completed 680000 out of 4000000 steps (17%)
13:03:45:WU03:FS02:0x22:Completed 720000 out of 4000000 steps (18%)
13:06:50:WU03:FS02:0x22:Completed 760000 out of 4000000 steps (19%)
13:09:55:WU03:FS02:0x22:Completed 800000 out of 4000000 steps (20%)
13:13:01:WU03:FS02:0x22:Completed 840000 out of 4000000 steps (21%)
13:16:07:WU03:FS02:0x22:Completed 880000 out of 4000000 steps (22%)
13:19:13:WU03:FS02:0x22:Completed 920000 out of 4000000 steps (23%)
13:22:18:WU03:FS02:0x22:Completed 960000 out of 4000000 steps (24%)
13:22:18:WU01:FS02:Cleaning up
13:22:18:ERROR:WU01:FS02:Exception: Failed to remove directory './work/01': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\01\logfile_01.txt"
13:22:43:WU00:FS02:Cleaning up
13:22:43:ERROR:WU00:FS02:Exception: Failed to remove directory './work/00': boost::filesystem::remove: Le processus ne peut pas accder au fichier car ce fichier est utilis par un autre processus: ".\work\00\logfile_01.txt"
- It finished and eventually sent in p14538-r0-c239-g6 (WU00).
- It downloaded p14539-r0-c480-g2 and it failed as soon as it started (WU01).
- It downloaded p14538-r0-c318-g11 and now folds successfully (WU03).
- It's trying to cleanup the previous two and i's not making it (WU00 & WU01).
BTW, I have had several p14539 fail on both my (old) GPUs.