Project: 12420 (Run 45, Clone 4, Gen 27) seems slow

[HendricksSA]

Having problems with this work unit. Normal frames for 12420 are about 9 minutes ... now that I switched to v8.3.18 I am seeing just over 2 hour frames. I did get a couple of errors in the log - the usual failure to prevent sleep that I think I can ignore (right?) and one for
18:19:48:E :OUT1:Failed response: CONNECT. Thoughts? Should probably start a new topic but is a 5 minute checkpoint normal for the v8 client?

Code: Select all

18:19:48:I1:*********************** Folding@home Client ***********************
18:19:48:I1: Version: 8.3.18
18:19:48:I1: Author: Joseph Coffland <[email protected]>
18:19:48:I1: Org: foldingathome.org
18:19:48:I1: Copyright: 2023-2024, foldingathome.org
18:19:48:I1: Homepage: https://foldingathome.org/
18:19:48:I1: License: GPL-3.0-or-later
18:19:48:I1: URL: https://v8-3.foldingathome.org/
18:19:48:I1: Date: Jul 12 2024
18:19:48:I1: Time: 13:26:31
18:19:48:I1: Revision: 99ae953ee7b1c0b3070161cfcf9150184f76bd96
18:19:48:I1: Branch: master
18:19:48:I1: Compiler: GNU 8.3.0
18:19:48:I1: Options: -Wsuggest-override -faligned-new -std=c++17 -fsigned-char
18:19:48:I1: -ffunction-sections -fdata-sections -O3 -funroll-loops -fno-pie
18:19:48:I1: Platform: linux 4.19.0-26-cloud-amd64
18:19:48:I1: Bits: 64
18:19:48:I1: Mode: Release
18:19:48:I1: Args: --config=/etc/fah-client/config.xml
18:19:48:I1: --log=/var/log/fah-client/log.txt
18:19:48:I1: --log-rotate-dir=/var/log/fah-client/
18:19:48:I1: Config: /etc/fah-client/config.xml
18:19:48:I1:****************************** CBang ******************************
18:19:48:I1: Version: 1.7.2
18:19:48:I1: Author: Joseph Coffland <[email protected]>
18:19:48:I1: Org: Cauldron Development
18:19:48:I1: Copyright: Cauldron Development, 2003-2024
18:19:48:I1: Homepage: https://cauldrondevelopment.com/
18:19:48:I1: License: LGPL-2.1-or-later
18:19:48:I1: Date: Jun 24 2024
18:19:48:I1: Time: 13:29:44
18:19:48:I1: Revision: 1b05ea96f0ed3043c32b78a66dbf50a9b2002289
18:19:48:I1: Branch: master
18:19:48:I1: Compiler: GNU 8.3.0
18:19:48:I1: Options: -Wsuggest-override -faligned-new -std=c++17 -fsigned-char
18:19:48:I1: -ffunction-sections -fdata-sections -O3 -funroll-loops -fno-pie
18:19:48:I1: -fPIC
18:19:48:I1: Platform: linux 4.19.0-26-cloud-amd64
18:19:48:I1: Bits: 64
18:19:48:I1: Mode: Release
18:19:48:I1:***************************** System ******************************
18:19:48:I1: CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
18:19:48:I1: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:19:48:I1: CPUs: 48
18:19:48:I1: Memory: 62.77GiB
18:19:48:I1:Free Memory: 61.54GiB
18:19:48:I1: OS Version: 6.8
18:19:48:I1:Has Battery: false
18:19:48:I1: On Battery: false
18:19:48:I1: Hostname: coffeeserver
18:19:48:I1: UTC Offset: -6
18:19:48:I1: PID: 1831
18:19:48:I1: CWD: /var/lib/fah-client
18:19:48:I1: Exec: /usr/bin/fah-client
18:19:48:I1:*******************************************************************
18:19:48:I2:<config>
18:19:48:I2: <!-- User Information -->
18:19:48:I2: <passkey v='*****'/>
18:19:48:I2: <user v='HendricksSA'/>
18:19:48:I2:</config>
18:19:48:I1:Opening Database
18:19:48:I1:F@H ID = 7S_v9HIxdehCsmHCw3fWaWnsn7maGO1kzRPgjZjlKS4
18:19:48:I3:Loading default group
18:19:48:I3:Loading default resource group
18:19:48:I1:Listening for HTTP on 127.0.0.1:7396
18:19:48:I3:WU6:Loading work unit 6 with ID znc93hAXrPFVsUmTvr1xip4qQGi17pA63zIke9jJmhY
18:19:48:I3:Loaded 1 wus.
18:19:48:W :OpenCL not supported: Failed to open dynamic library 'libOpenCL.so': libOpenCL.so: cannot open shared object file: No such file or directory
18:19:48:W :CUDA not supported: Failed to open dynamic library 'libcuda.so': libcuda.so: cannot open shared object file: No such file or directory
18:19:48:W :CON1:DNS lookup failed for api.foldingathome.org
18:19:48:E :OUT1:Failed response: CONNECT
18:20:08:I1:OUT2:> GET https://api.foldingathome.org/machine/7S_v9HIxdehCsmHCw3fWaWnsn7maGO1kzRPgjZjlKS4 HTTP/1.1
18:20:09:I1:OUT2:< HTTP/1.1 200 HTTP_OK
18:20:09:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
18:20:10:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
18:20:10:I1:Logging into node account
18:23:53:I1:Loaded cores/fahcore-a8-lin-64bit-avx2_256-0.0.12/FahCore_a8
18:23:53:I3:WU6:Running FahCore: /var/lib/fah-client/cores/fahcore-a8-lin-64bit-avx2_256-0.0.12/FahCore_a8 -dir znc93hAXrPFVsUmTvr1xip4qQGi17pA63zIke9jJmhY -suffix 01 -version 8.3.18 -lifeline 1831 -np 48
18:23:53:I3:WU6:Started FahCore on PID 7809
18:23:54:E :Exception: Failed to prevent sleep: Permission denied
18:23:54:I1:WU6:*********************** Log Started 2024-11-30T18:23:54Z ***********************
18:23:54:I1:WU6:************************** Gromacs Folding@home Core ***************************
18:23:54:I1:WU6: Core: Gromacs
18:23:54:I1:WU6: Type: 0xa8
18:23:54:I1:WU6: Version: 0.0.12
18:23:54:I1:WU6: Author: Joseph Coffland <[email protected]>
18:23:54:I1:WU6: Copyright: 2020 foldingathome.org
18:23:54:I1:WU6: Homepage: https://foldingathome.org/
18:23:54:I1:WU6: Date: Jan 16 2021
18:23:54:I1:WU6: Time: 19:24:44
18:23:54:I1:WU6: Compiler: GNU 8.3.0
18:23:54:I1:WU6: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
18:23:54:I1:WU6: -fdata-sections -O3 -funroll-loops -fno-pie
18:23:54:I1:WU6: Platform: linux2 4.15.0-128-generic
18:23:54:I1:WU6: Bits: 64
18:23:54:I1:WU6: Mode: Release
18:23:54:I1:WU6: SIMD: avx2_256
18:23:54:I1:WU6: OpenMP: ON
18:23:54:I1:WU6: CUDA: OFF
18:23:54:I1:WU6: Args: -dir znc93hAXrPFVsUmTvr1xip4qQGi17pA63zIke9jJmhY -suffix 01
18:23:54:I1:WU6: -version 8.3.18 -lifeline 1831 -np 48
18:23:54:I1:WU6:************************************ libFAH ************************************
18:23:54:I1:WU6: Date: Jan 16 2021
18:23:54:I1:WU6: Time: 19:21:38
18:23:54:I1:WU6: Compiler: GNU 8.3.0
18:23:54:I1:WU6: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
18:23:54:I1:WU6: -fdata-sections -O3 -funroll-loops -fno-pie
18:23:54:I1:WU6: Platform: linux2 4.15.0-128-generic
18:23:54:I1:WU6: Bits: 64
18:23:54:I1:WU6: Mode: Release
18:23:54:I1:WU6:************************************ CBang *************************************
18:23:54:I1:WU6: Date: Jan 16 2021
18:23:54:I1:WU6: Time: 19:21:24
18:23:54:I1:WU6: Compiler: GNU 8.3.0
18:23:54:I1:WU6: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
18:23:54:I1:WU6: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
18:23:54:I1:WU6: Platform: linux2 4.15.0-128-generic
18:23:54:I1:WU6: Bits: 64
18:23:54:I1:WU6: Mode: Release
18:23:54:I1:WU6:************************************ System ************************************
18:23:54:I1:WU6: CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
18:23:54:I1:WU6: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:23:54:I1:WU6: CPUs: 48
18:23:54:I1:WU6: Memory: 62.77GiB
18:23:54:I1:WU6:Free Memory: 58.99GiB
18:23:54:I1:WU6: Threads: POSIX_THREADS
18:23:54:I1:WU6: OS Version: 6.8
18:23:54:I1:WU6:Has Battery: false
18:23:54:I1:WU6: On Battery: false
18:23:54:I1:WU6: UTC Offset: -6
18:23:54:I1:WU6: PID: 7809
18:23:54:I1:WU6: CWD: /var/lib/fah-client/work
18:23:54:I1:WU6:********************************************************************************
18:23:54:I1:WU6:Project: 12420 (Run 45, Clone 4, Gen 27)
18:23:54:I1:WU6:Unit: 0x00000000000000000000000000000000
18:23:54:I1:WU6:Digital signatures verified
18:23:54:I1:WU6:Calling: mdrun -c frame27.gro -s frame27.tpr -x frame27.xtc -cpi state.cpt -cpt 5 -nt 48 -ntmpi 1
18:23:54:I1:WU6:Steps: first=135000000 total=140000000
18:23:59:I1:WU6:Completed 2307444 out of 5000000 steps (46%)
18:49:24:I1:WU6:Completed 2350000 out of 5000000 steps (47%)
20:50:41:I1:WU6:Completed 2400000 out of 5000000 steps (48%)
22:52:09:I1:WU6:Completed 2450000 out of 5000000 steps (49%)
00:53:21:I1:WU6:Completed 2500000 out of 5000000 steps (50%)

[Joe_H]

Have you checked off the "Keep Awake" setting in the machine configuration on the web control? I know I needed to do that on my laptop to keep processing at full speed.

Otherwise actual WU processing is done by the same folding cores used by the v7 client. So WUs should run at the same speeds as before when using v.8.3.18.

[HendricksSA]

Joe_H, <Keep Awake> is checked. I do not believe my machine is throttling back or sleeping. It is running at normal temps and top shows about 4400%. Frames are still over 2 hours and the client shows over 20 hours to go. I hope to make timeout ... but it will be close. PPD is down from 550,000 to 43,000. I do not know if other work units will be affected in the same way since this is the only project I have been working on after changing to v8.3.18.

[HendricksSA]

This work unit continues to get worse. Client v8.3.18 now estimates about 3 hours to ETA. I will be lucky to beat the deadline at this point. Plus another error has shown up. Should I worry?

16:41:01:I1:WU6:Completed 4700000 out of 5000000 steps (94%)
18:42:27:I1:WU6:Completed 4750000 out of 5000000 steps (95%)
20:43:59:I1:WU6:Completed 4800000 out of 5000000 steps (96%)
21:43:18:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
21:43:18:I1:OUT5:> GET https://api.foldingathome.org/machine/7 ... RPgjZjlKS4 HTTP/1.1
21:43:22:I1:OUT5:< HTTP/1.1 200 HTTP_OK
21:43:22:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:45:34:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:45:35:I1:OUT6:> GET https://api.foldingathome.org/machine/7 ... RPgjZjlKS4 HTTP/1.1
21:45:35:I1:OUT6:< HTTP/1.1 200 HTTP_OK
21:45:35:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:45:36:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:45:36:I1:Logging into node account
22:45:25:I1:WU6:Completed 4850000 out of 5000000 steps (97%)