Folding Forum

Community driven support forum for Folding@home
https://foldingforum.org/

16950 slowing down at the end

https://foldingforum.org/viewtopic.php?t=36680

Page 1 of 1

16950 slowing down at the end

Posted: Sat Jan 09, 2021 8:59 pm
by JimF
I don't think that I have ever seen this before. But P16950 on my Ryzen 3950X (Ubuntu 18.04) started at TPF of 3 minutes, and is slowing down at the end to 30 minutes.
I am wondering whether it will finish or not.

Code: Select all

14:59:54:WU00:FS01:Connecting to 129.32.209.203:8080
15:00:12:WU02:FS01:Download 100.00%
15:00:12:WU02:FS01:Download complete
15:00:12:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:16950 run:48 clone:4 gen:12 core:0xa8 unit:0x000000040000000c0000423600000030
15:00:12:WU02:FS01:Starting
15:00:12:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx2-256/a8-0.0.9/Core_a8.fah/FahCore_a8 -dir 02 -suffix 01 -version 706 -lifeline 1569 -checkpoint 15 -np 31
15:00:12:WU02:FS01:Started FahCore on PID 11871
15:00:12:WU02:FS01:Core PID:11875
15:00:12:WU02:FS01:FahCore 0xa8 started
15:00:12:WU02:FS01:0xa8:*********************** Log Started 2021-01-09T15:00:12Z ***********************
15:00:12:WU02:FS01:0xa8:************************** Gromacs Folding@home Core ***************************
15:00:12:WU02:FS01:0xa8:       Core: Gromacs
15:00:12:WU02:FS01:0xa8:       Type: 0xa8
15:00:12:WU02:FS01:0xa8:    Version: 0.0.9
15:00:12:WU02:FS01:0xa8:     Author: Joseph Coffland <[email protected]>
15:00:12:WU02:FS01:0xa8:  Copyright: 2020 foldingathome.org
15:00:12:WU02:FS01:0xa8:   Homepage: https://foldingathome.org/
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:15:07
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:       SIMD: avx2_256
15:00:12:WU02:FS01:0xa8:     OpenMP: ON
15:00:12:WU02:FS01:0xa8:       CUDA: OFF
15:00:12:WU02:FS01:0xa8:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11871 -checkpoint 15 -np
15:00:12:WU02:FS01:0xa8:             31
15:00:12:WU02:FS01:0xa8:************************************ libFAH ************************************
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:12:00
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:************************************ CBang *************************************
15:00:12:WU02:FS01:0xa8:       Date: Oct 28 2020
15:00:12:WU02:FS01:0xa8:       Time: 22:11:46
15:00:12:WU02:FS01:0xa8:   Compiler: GNU 8.3.0
15:00:12:WU02:FS01:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
15:00:12:WU02:FS01:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
15:00:12:WU02:FS01:0xa8:   Platform: linux2 4.15.0-108-generic
15:00:12:WU02:FS01:0xa8:       Bits: 64
15:00:12:WU02:FS01:0xa8:       Mode: Release
15:00:12:WU02:FS01:0xa8:************************************ System ************************************
15:00:12:WU02:FS01:0xa8:        CPU: AMD Ryzen 9 3950X 16-Core Processor
15:00:12:WU02:FS01:0xa8:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:00:12:WU02:FS01:0xa8:       CPUs: 32
15:00:12:WU02:FS01:0xa8:     Memory: 46.99GiB
15:00:12:WU02:FS01:0xa8:Free Memory: 40.08GiB
15:00:12:WU02:FS01:0xa8:    Threads: POSIX_THREADS
15:00:12:WU02:FS01:0xa8: OS Version: 5.4
15:00:12:WU02:FS01:0xa8:Has Battery: false
15:00:12:WU02:FS01:0xa8: On Battery: false
15:00:12:WU02:FS01:0xa8: UTC Offset: -5
15:00:12:WU02:FS01:0xa8:        PID: 11875
15:00:12:WU02:FS01:0xa8:        CWD: /var/lib/fahclient/work
15:00:12:WU02:FS01:0xa8:********************************************************************************
15:00:12:WU02:FS01:0xa8:Project: 16950 (Run 48, Clone 4, Gen 12)
15:00:12:WU02:FS01:0xa8:Unit: 0x00000000000000000000000000000000
15:00:12:WU02:FS01:0xa8:Reading tar file core.xml
15:00:12:WU02:FS01:0xa8:Reading tar file frame12.tpr
15:00:12:WU02:FS01:0xa8:Digital signatures verified
15:00:12:WU02:FS01:0xa8:Calling: mdrun -c frame12.gro -s frame12.tpr -x frame12.xtc -cpt 15 -nt 31 -ntmpi 1
15:00:12:WU02:FS01:0xa8:Steps: first=60000000 total=65000000
15:00:13:WU02:FS01:0xa8:Completed 1 out of 5000000 steps (0%)
15:00:28:WU00:FS01:Upload complete
15:00:28:WU00:FS01:Server responded WORK_ACK (400)
15:00:28:WU00:FS01:Final credit estimate, 6057.00 points
15:00:28:WU00:FS01:Cleaning up
15:03:10:WU02:FS01:0xa8:Completed 50000 out of 5000000 steps (1%)
15:06:06:WU02:FS01:0xa8:Completed 100000 out of 5000000 steps (2%)
15:09:03:WU02:FS01:0xa8:Completed 150000 out of 5000000 steps (3%)
15:11:59:WU02:FS01:0xa8:Completed 200000 out of 5000000 steps (4%)
15:14:54:WU02:FS01:0xa8:Completed 250000 out of 5000000 steps (5%)
15:17:49:WU02:FS01:0xa8:Completed 300000 out of 5000000 steps (6%)
15:20:44:WU02:FS01:0xa8:Completed 350000 out of 5000000 steps (7%)
15:23:40:WU02:FS01:0xa8:Completed 400000 out of 5000000 steps (8%)
15:26:36:WU02:FS01:0xa8:Completed 450000 out of 5000000 steps (9%)
15:29:31:WU02:FS01:0xa8:Completed 500000 out of 5000000 steps (10%)
15:32:26:WU02:FS01:0xa8:Completed 550000 out of 5000000 steps (11%)
15:35:21:WU02:FS01:0xa8:Completed 600000 out of 5000000 steps (12%)
15:38:16:WU02:FS01:0xa8:Completed 650000 out of 5000000 steps (13%)
15:41:13:WU02:FS01:0xa8:Completed 700000 out of 5000000 steps (14%)
15:44:07:WU02:FS01:0xa8:Completed 750000 out of 5000000 steps (15%)
******************************* Date: 2021-01-09 *******************************
15:47:03:WU02:FS01:0xa8:Completed 800000 out of 5000000 steps (16%)
15:49:58:WU02:FS01:0xa8:Completed 850000 out of 5000000 steps (17%)
15:52:54:WU02:FS01:0xa8:Completed 900000 out of 5000000 steps (18%)
15:55:51:WU02:FS01:0xa8:Completed 950000 out of 5000000 steps (19%)
15:58:44:WU02:FS01:0xa8:Completed 1000000 out of 5000000 steps (20%)
16:01:39:WU02:FS01:0xa8:Completed 1050000 out of 5000000 steps (21%)
16:04:35:WU02:FS01:0xa8:Completed 1100000 out of 5000000 steps (22%)
16:07:31:WU02:FS01:0xa8:Completed 1150000 out of 5000000 steps (23%)
16:10:28:WU02:FS01:0xa8:Completed 1200000 out of 5000000 steps (24%)
16:13:26:WU02:FS01:0xa8:Completed 1250000 out of 5000000 steps (25%)
16:16:22:WU02:FS01:0xa8:Completed 1300000 out of 5000000 steps (26%)
16:19:17:WU02:FS01:0xa8:Completed 1350000 out of 5000000 steps (27%)
16:22:12:WU02:FS01:0xa8:Completed 1400000 out of 5000000 steps (28%)
16:25:08:WU02:FS01:0xa8:Completed 1450000 out of 5000000 steps (29%)
16:28:04:WU02:FS01:0xa8:Completed 1500000 out of 5000000 steps (30%)
16:30:59:WU02:FS01:0xa8:Completed 1550000 out of 5000000 steps (31%)
16:33:54:WU02:FS01:0xa8:Completed 1600000 out of 5000000 steps (32%)
16:36:49:WU02:FS01:0xa8:Completed 1650000 out of 5000000 steps (33%)
16:39:45:WU02:FS01:0xa8:Completed 1700000 out of 5000000 steps (34%)
16:42:41:WU02:FS01:0xa8:Completed 1750000 out of 5000000 steps (35%)
16:45:36:WU02:FS01:0xa8:Completed 1800000 out of 5000000 steps (36%)
16:48:33:WU02:FS01:0xa8:Completed 1850000 out of 5000000 steps (37%)
16:51:28:WU02:FS01:0xa8:Completed 1900000 out of 5000000 steps (38%)
16:54:24:WU02:FS01:0xa8:Completed 1950000 out of 5000000 steps (39%)
16:57:21:WU02:FS01:0xa8:Completed 2000000 out of 5000000 steps (40%)
17:00:16:WU02:FS01:0xa8:Completed 2050000 out of 5000000 steps (41%)
17:03:11:WU02:FS01:0xa8:Completed 2100000 out of 5000000 steps (42%)
17:06:07:WU02:FS01:0xa8:Completed 2150000 out of 5000000 steps (43%)
17:09:03:WU02:FS01:0xa8:Completed 2200000 out of 5000000 steps (44%)
17:11:59:WU02:FS01:0xa8:Completed 2250000 out of 5000000 steps (45%)
17:14:56:WU02:FS01:0xa8:Completed 2300000 out of 5000000 steps (46%)
17:17:52:WU02:FS01:0xa8:Completed 2350000 out of 5000000 steps (47%)
17:20:48:WU02:FS01:0xa8:Completed 2400000 out of 5000000 steps (48%)
17:23:43:WU02:FS01:0xa8:Completed 2450000 out of 5000000 steps (49%)
17:26:38:WU02:FS01:0xa8:Completed 2500000 out of 5000000 steps (50%)
17:29:35:WU02:FS01:0xa8:Completed 2550000 out of 5000000 steps (51%)
17:32:31:WU02:FS01:0xa8:Completed 2600000 out of 5000000 steps (52%)
17:35:27:WU02:FS01:0xa8:Completed 2650000 out of 5000000 steps (53%)
17:38:23:WU02:FS01:0xa8:Completed 2700000 out of 5000000 steps (54%)
17:41:18:WU02:FS01:0xa8:Completed 2750000 out of 5000000 steps (55%)
17:44:14:WU02:FS01:0xa8:Completed 2800000 out of 5000000 steps (56%)
17:47:09:WU02:FS01:0xa8:Completed 2850000 out of 5000000 steps (57%)
17:50:04:WU02:FS01:0xa8:Completed 2900000 out of 5000000 steps (58%)
17:52:58:WU02:FS01:0xa8:Completed 2950000 out of 5000000 steps (59%)
17:55:54:WU02:FS01:0xa8:Completed 3000000 out of 5000000 steps (60%)
17:58:50:WU02:FS01:0xa8:Completed 3050000 out of 5000000 steps (61%)
18:01:45:WU02:FS01:0xa8:Completed 3100000 out of 5000000 steps (62%)
18:04:42:WU02:FS01:0xa8:Completed 3150000 out of 5000000 steps (63%)
18:07:36:WU02:FS01:0xa8:Completed 3200000 out of 5000000 steps (64%)
18:10:33:WU02:FS01:0xa8:Completed 3250000 out of 5000000 steps (65%)
18:13:28:WU02:FS01:0xa8:Completed 3300000 out of 5000000 steps (66%)
18:16:24:WU02:FS01:0xa8:Completed 3350000 out of 5000000 steps (67%)
18:19:21:WU02:FS01:0xa8:Completed 3400000 out of 5000000 steps (68%)
18:22:17:WU02:FS01:0xa8:Completed 3450000 out of 5000000 steps (69%)
18:25:11:WU02:FS01:0xa8:Completed 3500000 out of 5000000 steps (70%)
18:28:07:WU02:FS01:0xa8:Completed 3550000 out of 5000000 steps (71%)
18:31:02:WU02:FS01:0xa8:Completed 3600000 out of 5000000 steps (72%)
18:33:57:WU02:FS01:0xa8:Completed 3650000 out of 5000000 steps (73%)
18:36:53:WU02:FS01:0xa8:Completed 3700000 out of 5000000 steps (74%)
18:39:48:WU02:FS01:0xa8:Completed 3750000 out of 5000000 steps (75%)
18:42:43:WU02:FS01:0xa8:Completed 3800000 out of 5000000 steps (76%)
18:45:38:WU02:FS01:0xa8:Completed 3850000 out of 5000000 steps (77%)
18:48:30:WU02:FS01:0xa8:Completed 3900000 out of 5000000 steps (78%)
18:51:26:WU02:FS01:0xa8:Completed 3950000 out of 5000000 steps (79%)
18:54:21:WU02:FS01:0xa8:Completed 4000000 out of 5000000 steps (80%)
18:57:16:WU02:FS01:0xa8:Completed 4050000 out of 5000000 steps (81%)
19:00:13:WU02:FS01:0xa8:Completed 4100000 out of 5000000 steps (82%)
19:03:09:WU02:FS01:0xa8:Completed 4150000 out of 5000000 steps (83%)
19:06:06:WU02:FS01:0xa8:Completed 4200000 out of 5000000 steps (84%)
19:09:02:WU02:FS01:0xa8:Completed 4250000 out of 5000000 steps (85%)
19:11:57:WU02:FS01:0xa8:Completed 4300000 out of 5000000 steps (86%)
19:14:53:WU02:FS01:0xa8:Completed 4350000 out of 5000000 steps (87%)
19:17:49:WU02:FS01:0xa8:Completed 4400000 out of 5000000 steps (88%)
19:20:45:WU02:FS01:0xa8:Completed 4450000 out of 5000000 steps (89%)
19:23:42:WU02:FS01:0xa8:Completed 4500000 out of 5000000 steps (90%)
19:26:37:WU02:FS01:0xa8:Completed 4550000 out of 5000000 steps (91%)
19:29:34:WU02:FS01:0xa8:Completed 4600000 out of 5000000 steps (92%)
19:32:30:WU02:FS01:0xa8:Completed 4650000 out of 5000000 steps (93%)
19:35:26:WU02:FS01:0xa8:Completed 4700000 out of 5000000 steps (94%)
19:38:23:WU02:FS01:0xa8:Completed 4750000 out of 5000000 steps (95%)
19:40:41:FS01:Finishing
19:41:19:WU02:FS01:0xa8:Completed 4800000 out of 5000000 steps (96%)
20:09:23:WU02:FS01:0xa8:Completed 4850000 out of 5000000 steps (97%)
20:39:35:WU02:FS01:0xa8:Completed 4900000 out of 5000000 steps (98%)

Re: 16950 slowing down at the end

Posted: Sat Jan 09, 2021 9:54 pm
by JimF
OK, it finally finished up and uploaded apparently OK. It was just a bit unusual.

Code: Select all

20:48:45:FS01:Finishing
21:11:32:WU02:FS01:0xa8:Completed 4950000 out of 5000000 steps (99%)
******************************* Date: 2021-01-09 *******************************
21:49:56:WU02:FS01:0xa8:Completed 5000000 out of 5000000 steps (100%)
21:49:56:WU02:FS01:0xa8:Saving result file ../logfile_01.txt
21:49:56:WU02:FS01:0xa8:Saving result file dhdl.xvg
21:49:56:WU02:FS01:0xa8:Saving result file frame12.gro
21:49:56:WU02:FS01:0xa8:Saving result file frame12.xtc
21:49:56:WU02:FS01:0xa8:Saving result file md.log
21:49:56:WU02:FS01:0xa8:Saving result file science.log
21:49:56:WU02:FS01:0xa8:Saving result file state.cpt
21:49:56:WU02:FS01:0xa8:Folding@home Core Shutdown: FINISHED_UNIT
21:49:56:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
21:49:56:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:16950 run:48 clone:4 gen:12 core:0xa8 unit:0x000000040000000c0000423600000030
21:49:56:WU02:FS01:Uploading 21.69MiB to 129.32.209.203
21:49:56:WU02:FS01:Connecting to 129.32.209.203:8080
21:50:02:WU02:FS01:Upload 54.74%
21:50:43:WU02:FS01:Upload complete
21:50:43:WU02:FS01:Server responded WORK_ACK (400)
21:50:43:WU02:FS01:Final credit estimate, 66427.00 points
21:50:43:WU02:FS01:Cleaning up

Re: 16950 slowing down at the end

Posted: Sat Jan 09, 2021 10:12 pm
by psaam0001
I've seen some CPU WU's on each of my systems appear to start slow, but then they appear to reach a decent completion speed after the 2% mark is reached.

But I'm not worried. I'll just keep calm and fold on. After all, COVID's demise is on my schedule--until further notice.

Paul

Re: 16950 slowing down at the end

Posted: Sun Jan 10, 2021 12:59 am
by bruce
FAHCore_a8 is relatively new to me, but with some older cores and with older projects, this was explained by noting that the shape of the protein is changing. The motions of each atom is influenced mostly by the nearby atoms. If the atoms are sparse, the calculations are faster than if the atoms are closely packed because more calculations must be performed during each time-step.

In this particular case, that may or may not be the reason.

It does make it difficult to assign an appropriate number of points to all WUs in the project.

Re: 16950 slowing down at the end

Posted: Sun Jan 10, 2021 2:07 am
by JimF
bruce wrote:It does make it difficult to assign an appropriate number of points to all WUs in the project.
True, that was another anomaly, but not one that I am worried about. I get concerned about "stuck" work units, which are prevalent on some projects.
I have not seen them here though. If it is just a result of the workload, I can live with that.

Re: 16950 slowing down at the end

Posted: Sat Feb 06, 2021 4:52 am
by Ultrafire3
I've had this issue with GPU WUs if I suddenly do something GPU intensive and then stop. Even after I free up the GPU, the TPF stays tripled in length.

Re: 16950 slowing down at the end

Posted: Thu Feb 11, 2021 8:51 am
by PantherX
Welcome to the F@H Forum Ultrafire3,

Please note that the triple TPF will gradually come back to normal once at least 4% of the WU has been completed without any interruptions. Generally, the TFP being displayed uses the last 3% to calculate it thus, the "lag" in updating it.
All times are UTC
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited