13405 (328, 43, 2) went from 45% to 25%

[Jandska]

Hello again,

this GPU project was paused at friday 8th May and then I shut down the computer:

Code: Select all

20:18:12:WU01:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
20:18:53:WU03:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
20:22:48:WU03:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
20:24:59:FS00:Paused
20:24:59:FS01:Paused
20:24:59:FS00:Shutting core down
20:24:59:FS01:Shutting core down
20:24:59:WU01:FS01:0x22:WARNING:Console control signal 1 on PID 8912
20:25:00:WU03:FS00:0xa7:WARNING:Console control signal 1 on PID 5148
20:25:00:WU03:FS00:0xa7:Exiting, please wait. . .
20:25:00:WU01:FS01:0x22:Exiting, please wait. . .
20:25:00:WU01:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
20:25:01:WU03:FS00:0xa7:Folding@home Core Shutdown: INTERRUPTED
20:25:01:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
20:25:01:WU03:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
20:25:42:Removing old file 'configs/config-20200502-183949.xml'
20:25:42:Saving configuration to config.xml
20:25:42:<config>
20:25:42:  <!-- Network -->
20:25:42:  <proxy v=':8080'/>
20:25:42:
20:25:42:  <!-- Slot Control -->
20:25:42:  <power v='MEDIUM'/>
20:25:42:
20:25:42:  <!-- User Information -->
20:25:42:  <passkey v='*****'/>
20:25:42:  <team v='246927'/>
20:25:42:  <user v='Jandska'/>
20:25:42:
20:25:42:  <!-- Folding Slots -->
20:25:42:  <slot id='0' type='CPU'>
20:25:42:    <paused v='true'/>
20:25:42:  </slot>
20:25:42:  <slot id='1' type='GPU'>
20:25:42:    <paused v='true'/>
20:25:42:  </slot>
20:25:42:</config>

Today, I wanted to fold again...so turned my computer on and unpaused and then I noticed the project is at 25%:

Code: Select all

*********************** Log Started 2020-05-10T09:21:53Z ***********************
09:21:53:****************************** FAHClient ******************************
09:21:53:        Version: 7.6.9
09:21:53:         Author: Joseph Coffland <[email protected]>
09:21:53:      Copyright: 2020 foldingathome.org
09:21:53:       Homepage: https://foldingathome.org/
09:21:53:           Date: Apr 17 2020
09:21:53:           Time: 11:13:06
09:21:53:       Revision: 398c2b17fa535e0cc6c9d10856b2154c32771646
09:21:53:         Branch: master
09:21:53:       Compiler: Visual C++ 2008
09:21:53:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:21:53:       Platform: win32 10
09:21:53:           Bits: 32
09:21:53:           Mode: Release
09:21:53:         Config: C:\Users\Janda\AppData\Roaming\FAHClient\config.xml
09:21:53:******************************** CBang ********************************
09:21:53:           Date: Apr 17 2020
09:21:53:           Time: 11:10:09
09:21:53:       Revision: 2fb0be7809c5e45287a122ca5fbc15b5ae859a3b
09:21:53:         Branch: master
09:21:53:       Compiler: Visual C++ 2008
09:21:53:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:21:53:       Platform: win32 10
09:21:53:           Bits: 32
09:21:53:           Mode: Release
09:21:53:******************************* System ********************************
09:21:53:            CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz
09:21:53:         CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
09:21:53:           CPUs: 4
09:21:53:         Memory: 7.97GiB
09:21:53:    Free Memory: 5.20GiB
09:21:53:        Threads: WINDOWS_THREADS
09:21:53:     OS Version: 6.2
09:21:53:    Has Battery: false
09:21:53:     On Battery: false
09:21:53:     UTC Offset: 2
09:21:53:            PID: 11332
09:21:53:            CWD: C:\Users\Janda\AppData\Roaming\FAHClient
09:21:53:             OS: Windows 10 Enterprise
09:21:53:        OS Arch: AMD64
09:21:53:           GPUs: 1
09:21:53:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP107 [GeForce GTX 1050 LP] 1862
09:21:53:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:11.0
09:21:53:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:445.75
09:21:53:  Win32 Service: false
09:21:53:******************************* libFAH ********************************
09:21:53:           Date: Apr 15 2020
09:21:53:           Time: 14:53:14
09:21:53:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
09:21:53:         Branch: master
09:21:53:       Compiler: Visual C++ 2008
09:21:53:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:21:53:       Platform: win32 10
09:21:53:           Bits: 32
09:21:53:           Mode: Release
09:21:53:***********************************************************************
09:21:53:<config>
09:21:53:  <!-- Network -->
09:21:53:  <proxy v=':8080'/>
09:21:53:
09:21:53:  <!-- Slot Control -->
09:21:53:  <power v='MEDIUM'/>
09:21:53:
09:21:53:  <!-- User Information -->
09:21:53:  <passkey v='*****'/>
09:21:53:  <team v='246927'/>
09:21:53:  <user v='Jandska'/>
09:21:53:
09:21:53:  <!-- Folding Slots -->
09:21:53:  <slot id='0' type='CPU'>
09:21:53:    <paused v='true'/>
09:21:53:  </slot>
09:21:53:  <slot id='1' type='GPU'>
09:21:53:    <paused v='true'/>
09:21:53:  </slot>
09:21:53:</config>
09:21:53:Trying to access database...
09:21:53:Successfully acquired database lock
09:21:53:Enabled folding slot 00: PAUSED cpu:2 (by user)
09:21:53:Enabled folding slot 01: PAUSED gpu:0:GP107 [GeForce GTX 1050 LP] 1862 (by user)
09:22:16:FS00:Unpaused
09:22:16:FS01:Unpaused
09:22:16:WU01:FS01:Starting
09:22:16:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Janda\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 11332 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
09:22:32:WU01:FS01:Started FahCore on PID 9408
09:22:32:WU01:FS01:Core PID:672
09:22:32:WU01:FS01:FahCore 0x22 started
09:22:33:WU03:FS00:Starting
09:22:33:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Janda\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_a7.fah/FahCore_a7.exe -dir 03 -suffix 01 -version 706 -lifeline 11332 -checkpoint 15 -np 2
09:22:40:WU01:FS01:0x22:*********************** Log Started 2020-05-10T09:22:39Z ***********************
09:22:40:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
09:22:40:WU01:FS01:0x22:       Type: 0x22
09:22:40:WU01:FS01:0x22:       Core: Core22
09:22:40:WU01:FS01:0x22:    Website: https://foldingathome.org/
09:22:40:WU01:FS01:0x22:  Copyright: (c) 2009-2018 foldingathome.org
09:22:40:WU01:FS01:0x22:     Author: John Chodera <[email protected]> and Rafal Wiewiora
09:22:40:WU01:FS01:0x22:             <[email protected]>
09:22:40:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 9408 -checkpoint 15
09:22:40:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
09:22:40:WU01:FS01:0x22:             0 -gpu 0
09:22:40:WU01:FS01:0x22:     Config: <none>
09:22:40:WU01:FS01:0x22:************************************ Build *************************************
09:22:40:WU01:FS01:0x22:    Version: 0.0.5
09:22:40:WU01:FS01:0x22:       Date: Apr 22 2020
09:22:40:WU01:FS01:0x22:       Time: 04:42:59
09:22:40:WU01:FS01:0x22: Repository: Git
09:22:40:WU01:FS01:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
09:22:40:WU01:FS01:0x22:     Branch: HEAD
09:22:40:WU01:FS01:0x22:   Compiler: Visual C++ 2008
09:22:40:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:22:40:WU01:FS01:0x22:   Platform: win32 10
09:22:40:WU01:FS01:0x22:       Bits: 64
09:22:40:WU01:FS01:0x22:       Mode: Release
09:22:40:WU01:FS01:0x22:************************************ System ************************************
09:22:40:WU01:FS01:0x22:        CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz
09:22:40:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
09:22:40:WU01:FS01:0x22:       CPUs: 4
09:22:40:WU01:FS01:0x22:     Memory: 7.97GiB
09:22:40:WU01:FS01:0x22:Free Memory: 4.99GiB
09:22:40:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
09:22:40:WU01:FS01:0x22: OS Version: 6.2
09:22:40:WU01:FS01:0x22:Has Battery: false
09:22:40:WU01:FS01:0x22: On Battery: false
09:22:40:WU01:FS01:0x22: UTC Offset: 2
09:22:40:WU01:FS01:0x22:        PID: 672
09:22:40:WU01:FS01:0x22:        CWD: C:\Users\Janda\AppData\Roaming\FAHClient\work
09:22:40:WU01:FS01:0x22:         OS: Windows 10 Pro
09:22:40:WU01:FS01:0x22:    OS Arch: AMD64
09:22:40:WU01:FS01:0x22:********************************************************************************
09:22:40:WU01:FS01:0x22:Project: 13405 (Run 328, Clone 43, Gen 2)
09:22:40:WU01:FS01:0x22:Unit: 0x0000000312bc7d9a5eb3a38c310f1863
09:22:40:WU01:FS01:0x22:Digital signatures verified
09:22:40:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
09:22:40:WU01:FS01:0x22:Version 0.0.5
09:22:43:WU01:FS01:0x22:  Found a checkpoint file
09:22:43:WU03:FS00:Started FahCore on PID 6612
09:22:44:WU03:FS00:Core PID:15312
09:22:44:WU03:FS00:FahCore 0xa7 started
09:22:45:WU03:FS00:0xa7:*********************** Log Started 2020-05-10T09:22:45Z ***********************
09:22:45:WU03:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
09:22:45:WU03:FS00:0xa7:       Type: 0xa7
09:22:45:WU03:FS00:0xa7:       Core: Gromacs
09:22:45:WU03:FS00:0xa7:       Args: -dir 03 -suffix 01 -version 706 -lifeline 6612 -checkpoint 15 -np 2
09:22:45:WU03:FS00:0xa7:************************************ CBang *************************************
09:22:45:WU03:FS00:0xa7:       Date: Oct 26 2019
09:22:45:WU03:FS00:0xa7:       Time: 01:38:35
09:22:45:WU03:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
09:22:45:WU03:FS00:0xa7:     Branch: master
09:22:45:WU03:FS00:0xa7:   Compiler: Visual C++ 2008
09:22:45:WU03:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:22:45:WU03:FS00:0xa7:   Platform: win32 10
09:22:45:WU03:FS00:0xa7:       Bits: 64
09:22:45:WU03:FS00:0xa7:       Mode: Release
09:22:45:WU03:FS00:0xa7:************************************ System ************************************
09:22:45:WU03:FS00:0xa7:        CPU: Intel(R) Pentium(R) CPU G4560 @ 3.50GHz
09:22:45:WU03:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
09:22:45:WU03:FS00:0xa7:       CPUs: 4
09:22:45:WU03:FS00:0xa7:     Memory: 7.97GiB
09:22:45:WU03:FS00:0xa7:Free Memory: 4.97GiB
09:22:45:WU03:FS00:0xa7:    Threads: WINDOWS_THREADS
09:22:45:WU03:FS00:0xa7: OS Version: 6.2
09:22:45:WU03:FS00:0xa7:Has Battery: false
09:22:45:WU03:FS00:0xa7: On Battery: false
09:22:45:WU03:FS00:0xa7: UTC Offset: 2
09:22:45:WU03:FS00:0xa7:        PID: 15312
09:22:45:WU03:FS00:0xa7:        CWD: C:\Users\Janda\AppData\Roaming\FAHClient\work
09:22:45:WU03:FS00:0xa7:******************************** Build - libFAH ********************************
09:22:45:WU03:FS00:0xa7:    Version: 0.0.18
09:22:45:WU03:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
09:22:45:WU03:FS00:0xa7:  Copyright: 2019 foldingathome.org
09:22:45:WU03:FS00:0xa7:   Homepage: https://foldingathome.org/
09:22:45:WU03:FS00:0xa7:       Date: Oct 26 2019
09:22:45:WU03:FS00:0xa7:       Time: 01:52:44
09:22:45:WU03:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
09:22:45:WU03:FS00:0xa7:     Branch: master
09:22:45:WU03:FS00:0xa7:   Compiler: Visual C++ 2008
09:22:45:WU03:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:22:45:WU03:FS00:0xa7:   Platform: win32 10
09:22:45:WU03:FS00:0xa7:       Bits: 64
09:22:45:WU03:FS00:0xa7:       Mode: Release
09:22:45:WU03:FS00:0xa7:************************************ Build *************************************
09:22:45:WU03:FS00:0xa7:       SIMD: sse2
09:22:45:WU03:FS00:0xa7:********************************************************************************
09:22:45:WU03:FS00:0xa7:Project: 16406 (Run 705, Clone 4, Gen 167)
09:22:45:WU03:FS00:0xa7:Unit: 0x000000b4a8f5c67d5e801f1c363bb9a0
09:22:45:WU03:FS00:0xa7:Digital signatures verified
09:22:45:WU03:FS00:0xa7:Calling: mdrun -s frame167.tpr -o frame167.trr -x frame167.xtc -cpi state.cpt -cpt 15 -nt 2
09:22:47:WU03:FS00:0xa7:Steps: first=83500000 total=500000
09:22:50:WU03:FS00:0xa7:Completed 417772 out of 500000 steps (83%)
09:22:54:Removing old file 'configs/config-20200502-224858.xml'
09:22:54:Saving configuration to config.xml
09:22:54:<config>
09:22:54:  <!-- Network -->
09:22:54:  <proxy v=':8080'/>
09:22:54:
09:22:54:  <!-- Slot Control -->
09:22:54:  <power v='MEDIUM'/>
09:22:54:
09:22:54:  <!-- User Information -->
09:22:54:  <passkey v='*****'/>
09:22:54:  <team v='246927'/>
09:22:54:  <user v='Jandska'/>
09:22:54:
09:22:54:  <!-- Folding Slots -->
09:22:54:  <slot id='0' type='CPU'/>
09:22:54:  <slot id='1' type='GPU'/>
09:22:54:</config>
09:23:08:WU01:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
09:23:08:WU01:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:24:39:WU03:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
09:29:04:WU03:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
09:30:47:WU01:FS01:0x22:Completed 260000 out of 1000000 steps (26%)

I couldn't have been folding during yesterday because I was at work and not at home at all. This project is quite difficult for my GF 1050 GPU and it won't probably be completed in time...

[rwh202]

Edit - sorry please ignore, I misread

[Neil-B]

As I read it the GPU slot had clocked 45% when paused then restarted and showed 25% after restart the next morning … Maybe the checkpoints are every 25% on this project? and the GPU reset to the last checkpoint.

[rwh202]

Apologies, I misread initially
Yes the GPU dropped from 45 to 25.
I've only processed 1 13405, 2 13404 and found another log for one on this forum.
Plotting TPF, I see a spikes at 25% and 75% (but curiously not at 50% on all) that would indicate check pointing at those intervals.

[Jandska]

Oh, I see....do you think it could happen even when I pause it only, or only when shutting down the computer?

[rwh202]

If the check pointing is at 25%, then yes, pausing or however stopping processing will result in a loss of work up to that amount.
I've asked on the project thread if this is intentional, because 2.5% or 5% is more typical.

[Jandska]

Ok, thank you for your answers Neil and rwh

[Rel25917]

The checkpoint for these projects is indeed 25%. Something about new scientific mumbo jumbo going on between 25 and 50% that they dont want checkpoints happening in yet. They are still testing things out. It was posted in another thread here somewhere.

[bruce]

I had almost the same thing happen. I paused at 41% assuming it would go back to 40%. If I had let it run a bit longer and gotten to 51% it probably would not have gone back to 25% ... since 50% should have been check-pointed. This, indeed is an unusual project somehow associated with the COVID MoonShot. Recently it has beem just two projects. It's only being assigned to high-end GPUs.

[Jandska]

Yeah I thought I would have finished the WU "in time" but by going back to 25% it wasn't possible. Also I don't think that GeForce 1050 is/or ever was a high-end GPU....my TPF was 7,75 minutes or so (ok I'm also running only on "medium" so maybe if my folding preference was "full" it would be better)

[Neil-B]

by "it time" do you mean within the Timeout or within the Expiration deadline ... If the first, don't worry too much - if the later then it is a shame, but it happens.

[Jandska]

I mean Expiration deadline that was 2 days but unfortunately I wasn't at home yesterday due to planned work.

[Neil-B]

Hey … it happens … short deadline on that WU due to the moonshot nature - and an unexpected delay … shame to lose work, but actually being a very short timeout/deadline someone else picked it up swiftly and completed it https://apps.foldingathome.org/wu#proje ... e=43&gen=2 so barring a slight delay (and these happen don't fret) the science continues.

[bruce]

These two projects are unusual in another regard. Rather than being optimized for long trajectories (many Gen for each PRC) they're aimed at many runs and clones, each of which can run in parallel if there are enough Donors helping with the project. I think that's equivalent to no more that 3 Gens (0, 1, 2) per trajectory.

They were going for speed. I think the project may have been completed. I'm not getting any more of them.

[Jandska]

I know it's not related to this topic but I will ask anyway. Is it normal if I change slides from "medium" to "full" or the other way around and a WU crashes? Because once I did that and it crashed (I think it was only a CPU WU) so I don't do it anymore and leave it at medium.