184XX series work units assigned to slow CPUs

[southofnonorth]

I have just received WU 18419 - I have 3.49 days to complete this - ETA is 6.31 days! Never had anything this long before (I'm just using a bog-standard Asus laptop with Pentium Gold)

[psaam0001]

[deleted by original poster]

[Joe_H]

I have just received WU 18419 - I have 3.49 days to complete this - ETA is 6.31 days! Never had anything this long before (I'm just using a bog-standard Asus laptop with Pentium Gold)

I have split this from the topic it was originally posted in, the projects in the 18400 range currently are all CPU projects, not GPU. Please post the first 100 or so lines of your log file and a section showing processing of this Project 18419 WU.

They have been placing limits on what systems get these WUs, information on your system is important in providing guidance to the researchers on what systems should be excluded. Also the initial estimates the client will display may be off until several percent of the WU has been processed.

[aetch]

Also the initial estimates the client will display may be off until several percent of the WU has been processed.

The timeout for the 184xx projects is 4 days.
If it has 3.49 days left to run it means it's been running for ~12.1/2 hours, with the ETA of 6.31 days it suggests a total run time of 6.82 days.
At 12.1/2 hours into a 6.82 days runtime its current progress should be roughly 7-8%.
That's assuming it's a 24/7 system.

I have noticed that when a client receives a work unit from a new project it matches the ETA to the timeout until until the first fold has been logged.

[southofnonorth]

This is the log:-

Code: Select all

*********************** Log Started 2021-09-25T07:01:03Z ***********************
07:01:03:Trying to access database...
07:01:03:Successfully acquired database lock
07:01:03:Read GPUs.txt
07:01:03:Enabled folding slot 00: READY cpu:4
07:01:04:****************************** FAHClient ******************************
07:01:04:        Version: 7.6.13
07:01:04:         Author: Joseph Coffland <[email protected]>
07:01:04:      Copyright: 2020 foldingathome.org
07:01:04:       Homepage: https://foldingathome.org/
07:01:04:           Date: Apr 27 2020
07:01:04:           Time: 21:21:01
07:01:04:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
07:01:04:         Branch: master
07:01:04:       Compiler: Visual C++ 2008
07:01:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:01:04:       Platform: win32 10
07:01:04:           Bits: 32
07:01:04:           Mode: Release
07:01:04:         Config: C:\Users\steve\AppData\Roaming\FAHClient\config.xml
07:01:04:******************************** CBang ********************************
07:01:04:           Date: Apr 24 2020
07:01:04:           Time: 17:07:55
07:01:04:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
07:01:04:         Branch: master
07:01:04:       Compiler: Visual C++ 2008
07:01:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:01:04:       Platform: win32 10
07:01:04:           Bits: 32
07:01:04:           Mode: Release
07:01:04:******************************* System ********************************
07:01:04:            CPU: Intel(R) Pentium(R) CPU 4417U @ 2.30GHz
07:01:04:         CPU ID: GenuineIntel Family 6 Model 142 Stepping 10
07:01:04:           CPUs: 4
07:01:04:         Memory: 3.88GiB
07:01:04:    Free Memory: 1.66GiB
07:01:04:        Threads: WINDOWS_THREADS
07:01:04:     OS Version: 6.2
07:01:04:    Has Battery: true
07:01:04:     On Battery: false
07:01:04:     UTC Offset: 1
07:01:04:            PID: 12904
07:01:04:            CWD: C:\Users\steve\AppData\Roaming\FAHClient
07:01:04:  Win32 Service: false
07:01:04:             OS: Windows 10 Enterprise
07:01:04:        OS Arch: AMD64
07:01:04:           GPUs: 1
07:01:04:          GPU 0: Bus:0 Slot:2 Func:0 INTEL:1 KBL GT1 [HD Graphics 610]
07:01:04:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': The
07:01:04:                 specified module could not be found.
07:01:04:
07:01:04:OpenCL Device 0: Platform:0 Device:0 Bus:NA Slot:NA Compute:2.1 Driver:26.20
07:01:04:******************************* libFAH ********************************
07:01:04:           Date: Apr 15 2020
07:01:04:           Time: 14:53:14
07:01:04:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
07:01:04:         Branch: master
07:01:04:       Compiler: Visual C++ 2008
07:01:04:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:01:04:       Platform: win32 10
07:01:04:           Bits: 32
07:01:04:           Mode: Release
07:01:04:***********************************************************************
07:01:04:<config>
07:01:04:  <!-- Folding Slot Configuration -->
07:01:04:  <cause v='HIGH_PRIORITY'/>
07:01:04:
07:01:04:  <!-- Network -->
07:01:04:  <proxy v=':8080'/>
07:01:04:
07:01:04:  <!-- Slot Control -->
07:01:04:  <pause-on-battery v='false'/>
07:01:04:  <power v='FULL'/>
07:01:04:
07:01:04:  <!-- User Information -->
07:01:04:  <passkey v='*****'/>
07:01:04:  <user v='southofnonorth'/>
07:01:04:
07:01:04:  <!-- Folding Slots -->
07:01:04:  <slot id='0' type='CPU'/>
07:01:04:</config>
07:01:04:WU00:FS00:Starting
07:01:04:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\steve\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-sse2/a8-0.0.12/Core_a8.fah/FahCore_a8.exe -dir 00 -suffix 01 -version 706 -lifeline 12904 -checkpoint 15 -np 4
07:01:04:WU00:FS00:Started FahCore on PID 13892
07:01:04:WU00:FS00:Core PID:5488
07:01:04:WU00:FS00:FahCore 0xa8 started
07:01:05:WU00:FS00:0xa8:*********************** Log Started 2021-09-25T07:01:04Z ***********************
07:01:05:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
07:01:05:WU00:FS00:0xa8:       Core: Gromacs
07:01:05:WU00:FS00:0xa8:       Type: 0xa8
07:01:05:WU00:FS00:0xa8:    Version: 0.0.12
07:01:05:WU00:FS00:0xa8:     Author: Joseph Coffland <[email protected]>
07:01:05:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
07:01:05:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
07:01:05:WU00:FS00:0xa8:       Date: Jan 16 2021
07:01:05:WU00:FS00:0xa8:       Time: 11:25:04
07:01:05:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:01:05:WU00:FS00:0xa8:     Branch: master
07:01:05:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
07:01:05:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
07:01:05:WU00:FS00:0xa8:   Platform: win32 10
07:01:05:WU00:FS00:0xa8:       Bits: 64
07:01:05:WU00:FS00:0xa8:       Mode: Release
07:01:05:WU00:FS00:0xa8:       SIMD: sse2
07:01:05:WU00:FS00:0xa8:     OpenMP: ON
07:01:05:WU00:FS00:0xa8:       CUDA: OFF
07:01:05:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 13892 -checkpoint 15 -np
07:01:05:WU00:FS00:0xa8:             4
07:01:05:WU00:FS00:0xa8:************************************ libFAH ************************************
07:01:05:WU00:FS00:0xa8:       Date: Jan 16 2021
07:01:05:WU00:FS00:0xa8:       Time: 11:24:13
07:01:05:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:01:05:WU00:FS00:0xa8:     Branch: master
07:01:05:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
07:01:05:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
07:01:05:WU00:FS00:0xa8:   Platform: win32 10
07:01:05:WU00:FS00:0xa8:       Bits: 64
07:01:05:WU00:FS00:0xa8:       Mode: Release
07:01:05:WU00:FS00:0xa8:************************************ CBang *************************************
07:01:05:WU00:FS00:0xa8:       Date: Jan 16 2021
07:01:05:WU00:FS00:0xa8:       Time: 11:23:53
07:01:05:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:01:05:WU00:FS00:0xa8:     Branch: master
07:01:05:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
07:01:05:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
07:01:05:WU00:FS00:0xa8:   Platform: win32 10
07:01:05:WU00:FS00:0xa8:       Bits: 64
07:01:05:WU00:FS00:0xa8:       Mode: Release
07:01:05:WU00:FS00:0xa8:************************************ System ************************************
07:01:05:WU00:FS00:0xa8:        CPU: Intel(R) Pentium(R) CPU 4417U @ 2.30GHz
07:01:05:WU00:FS00:0xa8:     CPU ID: GenuineIntel Family 6 Model 142 Stepping 10
07:01:05:WU00:FS00:0xa8:       CPUs: 4
07:01:05:WU00:FS00:0xa8:     Memory: 3.88GiB
07:01:05:WU00:FS00:0xa8:Free Memory: 1.60GiB
07:01:05:WU00:FS00:0xa8:    Threads: WINDOWS_THREADS
07:01:05:WU00:FS00:0xa8: OS Version: 6.2
07:01:05:WU00:FS00:0xa8:Has Battery: true
07:01:05:WU00:FS00:0xa8: On Battery: false
07:01:05:WU00:FS00:0xa8: UTC Offset: 1
07:01:05:WU00:FS00:0xa8:        PID: 5488
07:01:05:WU00:FS00:0xa8:        CWD: C:\Users\steve\AppData\Roaming\FAHClient\work
07:01:05:WU00:FS00:0xa8:********************************************************************************
07:01:05:WU00:FS00:0xa8:Project: 18419 (Run 28, Clone 5, Gen 1)
07:01:05:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
07:01:05:WU00:FS00:0xa8:Digital signatures verified
07:01:05:WU00:FS00:0xa8:Calling: mdrun -c frame1.gro -s frame1.tpr -x frame1.xtc -cpi state.cpt -cpt 15 -nt 4 -ntmpi 1
07:01:05:WU00:FS00:0xa8:Steps: first=10000000 total=20000000
07:01:09:WU00:FS00:0xa8:Completed 2100801 out of 10000000 steps (21%)
07:05:05:8:127.0.0.1:New Web session
08:28:32:42:127.0.0.1:New Web session
08:32:28:WU00:FS00:0xa8:Completed 2200000 out of 10000000 steps (22%)

[gunnarre]

The performance of that CPU is similar to my i5-2520M I think. It completes 18210 and 18212 WUs successfully, but never been assigned 18419 WUs.

[southofnonorth]

Well, that was a waste of 5 days! Eventually got to about 60% and got pulled - no points either!

[Joe_H]

I have checked into this bit further. The limits for assigning WUs from this project may have been changed, so it should not get assigned to you again.

In general though, your 4417U is limited by being a 2 core CPU with 4 threads available through HT. Also as a Pentium Intel has not enabled AVX support on the processor which makes a significant difference in processing speed.

As the improvement in processing speed is not great using the HT provided extra CPU threads, I would suggest trying setting the CPU processing to 2 threads and seeing if that gets you close to the same TPF. The processor can also run cooler without the HT section of the processor cores active.

[southofnonorth]

Thank you for that information - I'll give it a try - I have had no problem processing all the other WU's I have had (but less fan noise would be nice!)

[DavidCChapman]

I've also seen this problem, running on a 7-year-old AMD FX-6300 with 6 processors (not using the GPU - it's too cheap to run F@H). Earlier this week I had a work unit expire; the log has been purged so I don't know the project number. This morning a work unit for project 18416 was assigned. Initial TPF was estimated at 57 minutes, jumping to 1 hour, 15 minutes after the first fold. ETA is 5.00 days now (2012-10-02T19:40:05Z) with an expiration of 2021-10-06T18:22:47Z, so it will expire too.

I hate to waste CPU cycles, so some tuning on the F@H side would be appreciated.

The log so far is:

Code: Select all

13:42:48:WU00:FS00:Starting
13:42:48:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\David\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a8-0.0.12/Core_a8.fah/FahCore_a8.exe -dir 00 -suffix 01 -version 706 -lifeline 10384 -checkpoint 15 -np 6
13:42:48:WU00:FS00:Started FahCore on PID 2296
13:42:48:WU00:FS00:Core PID:11204
13:42:48:WU00:FS00:FahCore 0xa8 started
13:42:48:WU00:FS00:0xa8:*********************** Log Started 2021-10-02T13:42:48Z ***********************
13:42:48:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
13:42:48:WU00:FS00:0xa8:       Core: Gromacs
13:42:48:WU00:FS00:0xa8:       Type: 0xa8
13:42:48:WU00:FS00:0xa8:    Version: 0.0.12
13:42:48:WU00:FS00:0xa8:     Author: Joseph Coffland <[email protected]>
13:42:48:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
13:42:48:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
13:42:48:WU00:FS00:0xa8:       Date: Jan 16 2021
13:42:48:WU00:FS00:0xa8:       Time: 11:56:07
13:42:48:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
13:42:48:WU00:FS00:0xa8:     Branch: master
13:42:48:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
13:42:48:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:42:48:WU00:FS00:0xa8:   Platform: win32 10
13:42:48:WU00:FS00:0xa8:       Bits: 64
13:42:48:WU00:FS00:0xa8:       Mode: Release
13:42:48:WU00:FS00:0xa8:       SIMD: avx_256
13:42:48:WU00:FS00:0xa8:     OpenMP: ON
13:42:48:WU00:FS00:0xa8:       CUDA: OFF
13:42:48:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 2296 -checkpoint 15 -np 6
13:42:48:WU00:FS00:0xa8:************************************ libFAH ************************************
13:42:48:WU00:FS00:0xa8:       Date: Jan 16 2021
13:42:48:WU00:FS00:0xa8:       Time: 11:24:13
13:42:48:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
13:42:48:WU00:FS00:0xa8:     Branch: master
13:42:48:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
13:42:48:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:42:48:WU00:FS00:0xa8:   Platform: win32 10
13:42:48:WU00:FS00:0xa8:       Bits: 64
13:42:48:WU00:FS00:0xa8:       Mode: Release
13:42:48:WU00:FS00:0xa8:************************************ CBang *************************************
13:42:48:WU00:FS00:0xa8:       Date: Jan 16 2021
13:42:48:WU00:FS00:0xa8:       Time: 11:23:53
13:42:48:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
13:42:48:WU00:FS00:0xa8:     Branch: master
13:42:48:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
13:42:48:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:42:48:WU00:FS00:0xa8:   Platform: win32 10
13:42:48:WU00:FS00:0xa8:       Bits: 64
13:42:48:WU00:FS00:0xa8:       Mode: Release
13:42:48:WU00:FS00:0xa8:************************************ System ************************************
13:42:48:WU00:FS00:0xa8:        CPU: AMD FX(tm)-6300 Six-Core Processor
13:42:48:WU00:FS00:0xa8:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
13:42:48:WU00:FS00:0xa8:       CPUs: 6
13:42:48:WU00:FS00:0xa8:     Memory: 15.90GiB
13:42:48:WU00:FS00:0xa8:Free Memory: 13.09GiB
13:42:48:WU00:FS00:0xa8:    Threads: WINDOWS_THREADS
13:42:48:WU00:FS00:0xa8: OS Version: 6.2
13:42:48:WU00:FS00:0xa8:Has Battery: false
13:42:48:WU00:FS00:0xa8: On Battery: false
13:42:48:WU00:FS00:0xa8: UTC Offset: -7
13:42:48:WU00:FS00:0xa8:        PID: 11204
13:42:48:WU00:FS00:0xa8:        CWD: C:\Users\David\AppData\Roaming\FAHClient\work
13:42:48:WU00:FS00:0xa8:********************************************************************************
13:42:48:WU00:FS00:0xa8:Project: 18416 (Run 45, Clone 1, Gen 4)
13:42:48:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
13:42:48:WU00:FS00:0xa8:Reading tar file core.xml
13:42:48:WU00:FS00:0xa8:Reading tar file frame4.tpr
13:42:48:WU00:FS00:0xa8:Digital signatures verified
13:42:48:WU00:FS00:0xa8:Calling: mdrun -c frame4.gro -s frame4.tpr -x frame4.xtc -cpt 15 -nt 6 -ntmpi 1
13:42:48:WU00:FS00:0xa8:Steps: first=40000000 total=50000000
13:42:52:WU00:FS00:0xa8:Completed 1 out of 10000000 steps (0%)
14:57:47:WU00:FS00:0xa8:Completed 100000 out of 10000000 steps (1%)
16:12:50:WU00:FS00:0xa8:Completed 200000 out of 10000000 steps (2%)
******************************* Date: 2021-10-02 *******************************
17:28:21:WU00:FS00:0xa8:Completed 300000 out of 10000000 steps (3%)
18:47:53:WU00:FS00:0xa8:Completed 400000 out of 10000000 steps (4%)

[Joe_H]

I hate to waste CPU cycles, so some tuning on the F@H side would be appreciated.

Not much more they can do with the current server and client software. Set the number of CPU threads down to 4 and you will not get these. That is the current setting, CPU threads greater than 4. Not the best as higher performance newer CPUs can handle these well. For example my system with an i7-7700K was doing these in just over 24 hours for about 135k points and is now not getting them.

[debs3759]

I've also seen this problem, running on a 7-year-old AMD FX-6300 with 6 processors

the Bulldozer CPUs only had one FPU core for each pair of integer cores, so I would set it to only use 3 CPUs. Might make a difference to what work you get assigned.

[DavidCChapman]

These are the first expiring work units in 18 months of folding with this machine. Normally work units finish well before the expiration time. Reducing the core/thread count would reduce the amount of folding I could do all the time.

[aetch]

the log has been purged

check your logs folder - C:\Users\David\AppData\Roaming\FAHClient\logs
by default the client will store up to your previous 16 logs in here.

[DavidCChapman]

Thanks for the pointer to the log files. The first expired work unit was for project 18418. Each fold required about 1 hour 15 minutes, though I see some variation. I can post the log fragment if anyone thinks it's useful, but it will look much the same as the first.