Multiple WU's Fail downld/upld to 155.247.166.*

[bruce]

I had a similar problem with 155.247.166.219 (also at temple.edu) last night, but I rebooted which cleared out the active connections and since restarting, have had no difficulties.

[bruce]

I had a problem last night with vav3 & vav4 which matches bollix47's description ... and rebooting cleared out the broken connections. As far as I can tell, the problems were all with projects distributed from temple.edu and campus IS must have done something to fix it.

[FaaR]

Code: Select all

23:07:22:WU01:FS00:0xa7:Completed 2500000 out of 2500000 steps (100%)
23:07:25:WU01:FS00:0xa7:Saving result file ..\logfile_01.txt
23:07:25:WU01:FS00:0xa7:Saving result file ener.edr
23:07:25:WU01:FS00:0xa7:Saving result file frame13.trr
23:07:25:WU01:FS00:0xa7:Saving result file md.log
23:07:25:WU01:FS00:0xa7:Saving result file pullf.xvg
23:07:25:WU01:FS00:0xa7:Saving result file pullx.xvg
23:07:25:WU01:FS00:0xa7:Saving result file science.log
23:07:25:WU01:FS00:0xa7:Saving result file traj_comp.xtc
23:07:25:WU01:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
23:07:26:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
23:07:26:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:13794 run:2 clone:151 gen:13 core:0xa7 unit:0x0000000e0002894c5d5f339cd1d61790
23:07:26:WU01:FS00:Uploading 26.72MiB to 155.247.166.220
23:07:26:WU01:FS00:Connecting to 155.247.166.220:8080
23:07:26:WU03:FS00:Download 48.73%
23:07:32:WU01:FS00:Upload 5.85%
23:07:35:WU03:FS00:Download 49.75%
23:07:38:WU01:FS00:Upload 13.10%
23:07:43:WU03:FS00:Download 51.78%
23:07:44:WU01:FS00:Upload 22.69%
23:07:50:WU01:FS00:Upload 32.52%
23:07:51:WU03:FS00:Download 53.81%
23:07:56:WU01:FS00:Upload 42.11%
23:08:02:WU01:FS00:Upload 49.36%
23:08:08:WU01:FS00:Upload 58.48%
23:08:14:WU01:FS00:Upload 67.61%
23:08:20:WU01:FS00:Upload 77.90%
23:08:26:WU01:FS00:Upload 90.06%
23:08:32:WU01:FS00:Upload complete
23:08:32:WU01:FS00:Server responded WORK_ACK (400)
23:08:32:WU01:FS00:Final credit estimate, 16226.00 points
23:08:32:WU01:FS00:Cleaning up
23:08:39:WU03:FS00:Download 54.83%
23:08:47:WU03:FS00:Download 56.86%
23:08:54:WU03:FS00:Download 58.89%
23:09:03:WU03:FS00:Download 60.92%
23:09:11:WU03:FS00:Download 62.95%
23:09:21:WU03:FS00:Download 63.96%
23:09:31:WU03:FS00:Download 64.98%
23:09:52:WU03:FS00:Download 67.01%
23:09:59:WU03:FS00:Download 69.04%
23:10:05:WU03:FS00:Download 70.06%
23:10:12:WU03:FS00:Download 72.09%
23:10:21:WU03:FS00:Download 73.10%
23:10:28:WU03:FS00:Download 75.13%
23:10:35:WU03:FS00:Download 77.16%
23:10:42:WU03:FS00:Download 79.19%
23:10:49:WU03:FS00:Download 82.24%
23:10:56:WU03:FS00:Download 84.27%
23:11:03:WU03:FS00:Download 86.30%
23:12:39:WU03:FS00:Download 87.77%
23:12:39:ERROR:WU03:FS00:Exception: Transfer failed
23:12:39:WU03:FS00:Connecting to 65.254.110.245:8080
23:12:40:WU03:FS00:Assigned to work server 155.247.166.219
23:12:40:WU03:FS00:Requesting new work unit for slot 00: READY cpu:17 from 155.247.166.219
23:12:40:WU03:FS00:Connecting to 155.247.166.219:8080
23:12:41:WU03:FS00:Downloading 6.16MiB
23:12:49:WU03:FS00:Download 4.06%
23:12:56:WU03:FS00:Download 6.09%
23:13:02:WU03:FS00:Download 8.12%
23:13:11:WU03:FS00:Download 11.17%
23:13:18:WU03:FS00:Download 13.20%
23:13:26:WU03:FS00:Download 16.24%
23:13:35:WU03:FS00:Download 18.28%

"Download speed" is roughly 1-2% of the total WU size - however large that is, not that large surely - roughly every TEN SECONDS. (500Mbit/s cable, so there's ample capacity!) And then on top of it, download failed for whatever reason, I'm not sure why exactly. Simply too slow? So it restarted, and then it did actually finish the download, even though that one went slow as well, so now it's back to folding again.

Downloads have been slow for me all day long, both CPU and GPU WUs. No idea what's going on...

[MeeLee]

I've found a temporary solution.
The servers are connected to a wifi network; the room the server is in, is inaccessible for wired LAN networks (I'm not allowed to drill in the walls).
And by rerouting the wireless router closer to the server, now downloads are a lot faster, but uploads are still slow as hell.My ISP only granted me a 1mbit upload, where of FAH seems to have trouble uploading less than half a percent per second, of a WU.

A lot of WUs got lost due to network collision, and WU queuing this way, are fighting for the little bandwidth there is; especially lately.
The mention has been made that WUs are much larger in size lately, and I'm thinking they're related.
Bruce, could you post a bug on github on this please?
The larger WUs are colliding (network issues) and holding/freezing up the slots, it seems of a lot of people on the forum.

This temporary solution on my end, of moving the router, helps in clearing at least the download queue faster.
The upload queues are still between 1 to 4 WUs uploading at a time, but they get sped up slightly in part because barely any downloads are eating bandwidth.
This may be (one of) the cause(s) of slot slowdowns, freezes.

[MeeLee]

I think it's been reported that server 155.247.166.220 uploads really slow, and may be the cause of all these issues.
A 15MB file with me takes almost 5 minutes to download, while other 70MB WUs get downloaded in a few seconds.

[prcowley]

another me too.

The downloads have been getting slower and slower to download, sometimes taking multiple minutes.

An hour ago I removed the slot and added it back again. A unit downloaded (a small one) when it completed to 93% (I have lowered the percentage since the downloads were taking a long time) it started to download the next unit. It did just a wee bit then stopped and has been like that with no further action for 5 mins.

Below is the log file from when I re-added the GPU slot

Cheers
Pete

06:40:46:Adding folding slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380
06:40:46:Removing old file 'configs/config-20190924-042417.xml'
06:40:46:Saving configuration to /etc/fahclient/config.xml
06:40:46:<config>
06:40:46: <!-- Network -->
06:40:46: <proxy v=':8080'/>
06:40:46:
06:40:46: <!-- User Information -->
06:40:46: <passkey v='********************************'/>
06:40:46: <team v='163'/>
06:40:46: <user v='pcowley'/>
06:40:46:
06:40:46: <!-- Work Unit Control -->
06:40:46: <next-unit-percentage v='93'/>
06:40:46:
06:40:46: <!-- Folding Slots -->
06:40:46: <slot id='0' type='GPU'>
06:40:46: <opencl-index v='0'/>
06:40:46: </slot>
06:40:46:</config>
06:40:47:WU01:FS00:Connecting to 65.254.110.245:8080
06:40:48:WU01:FS00:Assigned to work server 128.252.203.10
06:40:48:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 128.252.203.10
06:40:48:WU01:FS00:Connecting to 128.252.203.10:8080
06:40:49:WU01:FS00:Downloading 69.92MiB
06:40:52:Removing old file 'configs/config-20190924-042432.xml'
06:40:52:Saving configuration to /etc/fahclient/config.xml
06:40:52:<config>
06:40:52: <!-- Network -->
06:40:52: <proxy v=':8080'/>
06:40:52:
06:40:52: <!-- User Information -->
06:40:52: <passkey v='********************************'/>
06:40:52: <team v='163'/>
06:40:52: <user v='pcowley'/>
06:40:52:
06:40:52: <!-- Work Unit Control -->
06:40:52: <next-unit-percentage v='93'/>
06:40:52:
06:40:52: <!-- Folding Slots -->
06:40:52: <slot id='0' type='GPU'>
06:40:52: <opencl-index v='0'/>
06:40:52: </slot>
06:40:52:</config>
06:40:55:WU01:FS00:Download 20.02%
06:41:01:WU01:FS00:Download 52.74%
06:41:07:WU01:FS00:Download 89.21%
06:41:08:WU01:FS00:Download complete
06:41:08:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14230 run:374 clone:0 gen:46 core:0x21 unit:0x0000004080fccb0a5d654ddf126ec99d
06:41:08:WU01:FS00:Starting
06:41:08:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 2110 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
06:41:08:WU01:FS00:Started FahCore on PID 7668
06:41:08:WU01:FS00:Core PID:7672
06:41:08:WU01:FS00:FahCore 0x21 started
06:41:08:WU01:FS00:0x21:*********************** Log Started 2019-09-26T06:41:08Z ***********************
06:41:08:WU01:FS00:0x21:Project: 14230 (Run 374, Clone 0, Gen 46)
06:41:08:WU01:FS00:0x21:Unit: 0x0000004080fccb0a5d654ddf126ec99d
06:41:08:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
06:41:08:WU01:FS00:0x21:Machine: 0
06:41:08:WU01:FS00:0x21:Reading tar file core.xml
06:41:08:WU01:FS00:0x21:Reading tar file integrator.xml
06:41:08:WU01:FS00:0x21:Reading tar file state.xml
06:41:08:WU01:FS00:0x21:Reading tar file system.xml
06:41:08:WU01:FS00:0x21:Digital signatures verified
06:41:08:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
06:41:08:WU01:FS00:0x21:Version 0.0.20
06:41:20:WU01:FS00:0x21:Completed 0 out of 500000 steps (0%)
06:41:20:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:41:54:WU01:FS00:0x21:Completed 5000 out of 500000 steps (1%)
06:42:28:WU01:FS00:0x21:Completed 10000 out of 500000 steps (2%)
06:43:02:WU01:FS00:0x21:Completed 15000 out of 500000 steps (3%)
06:43:36:WU01:FS00:0x21:Completed 20000 out of 500000 steps (4%)
06:44:14:WU01:FS00:0x21:Completed 25000 out of 500000 steps (5%)
06:44:49:WU01:FS00:0x21:Completed 30000 out of 500000 steps (6%)
06:45:23:WU01:FS00:0x21:Completed 35000 out of 500000 steps (7%)
06:45:57:WU01:FS00:0x21:Completed 40000 out of 500000 steps (8%)
06:46:35:WU01:FS00:0x21:Completed 45000 out of 500000 steps (9%)
06:47:10:WU01:FS00:0x21:Completed 50000 out of 500000 steps (10%)
06:47:45:WU01:FS00:0x21:Completed 55000 out of 500000 steps (11%)
06:48:19:WU01:FS00:0x21:Completed 60000 out of 500000 steps (12%)
06:48:57:WU01:FS00:0x21:Completed 65000 out of 500000 steps (13%)
06:49:31:WU01:FS00:0x21:Completed 70000 out of 500000 steps (14%)
06:50:06:WU01:FS00:0x21:Completed 75000 out of 500000 steps (15%)
06:50:40:WU01:FS00:0x21:Completed 80000 out of 500000 steps (16%)
06:51:18:WU01:FS00:0x21:Completed 85000 out of 500000 steps (17%)
06:51:52:WU01:FS00:0x21:Completed 90000 out of 500000 steps (18%)
06:52:26:WU01:FS00:0x21:Completed 95000 out of 500000 steps (19%)
06:53:00:WU01:FS00:0x21:Completed 100000 out of 500000 steps (20%)
******************************* Date: 2019-09-26 *******************************
06:53:38:WU01:FS00:0x21:Completed 105000 out of 500000 steps (21%)
06:54:12:WU01:FS00:0x21:Completed 110000 out of 500000 steps (22%)
06:54:46:WU01:FS00:0x21:Completed 115000 out of 500000 steps (23%)
06:55:21:WU01:FS00:0x21:Completed 120000 out of 500000 steps (24%)
06:55:59:WU01:FS00:0x21:Completed 125000 out of 500000 steps (25%)
06:56:33:WU01:FS00:0x21:Completed 130000 out of 500000 steps (26%)
06:57:07:WU01:FS00:0x21:Completed 135000 out of 500000 steps (27%)
06:57:42:WU01:FS00:0x21:Completed 140000 out of 500000 steps (28%)
06:58:20:WU01:FS00:0x21:Completed 145000 out of 500000 steps (29%)
06:58:54:WU01:FS00:0x21:Completed 150000 out of 500000 steps (30%)
06:59:27:WU01:FS00:0x21:Completed 155000 out of 500000 steps (31%)
07:00:01:WU01:FS00:0x21:Completed 160000 out of 500000 steps (32%)
07:00:39:WU01:FS00:0x21:Completed 165000 out of 500000 steps (33%)
07:01:13:WU01:FS00:0x21:Completed 170000 out of 500000 steps (34%)
07:01:47:WU01:FS00:0x21:Completed 175000 out of 500000 steps (35%)
07:02:21:WU01:FS00:0x21:Completed 180000 out of 500000 steps (36%)
07:02:59:WU01:FS00:0x21:Completed 185000 out of 500000 steps (37%)
07:03:33:WU01:FS00:0x21:Completed 190000 out of 500000 steps (38%)
07:04:06:WU01:FS00:0x21:Completed 195000 out of 500000 steps (39%)
07:04:41:WU01:FS00:0x21:Completed 200000 out of 500000 steps (40%)
07:05:19:WU01:FS00:0x21:Completed 205000 out of 500000 steps (41%)
07:05:54:WU01:FS00:0x21:Completed 210000 out of 500000 steps (42%)
07:06:29:WU01:FS00:0x21:Completed 215000 out of 500000 steps (43%)
07:07:03:WU01:FS00:0x21:Completed 220000 out of 500000 steps (44%)
07:07:41:WU01:FS00:0x21:Completed 225000 out of 500000 steps (45%)
07:08:15:WU01:FS00:0x21:Completed 230000 out of 500000 steps (46%)
07:08:49:WU01:FS00:0x21:Completed 235000 out of 500000 steps (47%)
07:09:23:WU01:FS00:0x21:Completed 240000 out of 500000 steps (48%)
07:10:01:WU01:FS00:0x21:Completed 245000 out of 500000 steps (49%)
07:10:35:WU01:FS00:0x21:Completed 250000 out of 500000 steps (50%)
07:11:09:WU01:FS00:0x21:Completed 255000 out of 500000 steps (51%)
07:11:43:WU01:FS00:0x21:Completed 260000 out of 500000 steps (52%)
07:12:21:WU01:FS00:0x21:Completed 265000 out of 500000 steps (53%)
07:12:55:WU01:FS00:0x21:Completed 270000 out of 500000 steps (54%)
07:13:30:WU01:FS00:0x21:Completed 275000 out of 500000 steps (55%)
07:14:05:WU01:FS00:0x21:Completed 280000 out of 500000 steps (56%)
07:14:43:WU01:FS00:0x21:Completed 285000 out of 500000 steps (57%)
07:15:17:WU01:FS00:0x21:Completed 290000 out of 500000 steps (58%)
07:15:51:WU01:FS00:0x21:Completed 295000 out of 500000 steps (59%)
07:16:25:WU01:FS00:0x21:Completed 300000 out of 500000 steps (60%)
07:17:03:WU01:FS00:0x21:Completed 305000 out of 500000 steps (61%)
07:17:38:WU01:FS00:0x21:Completed 310000 out of 500000 steps (62%)
07:18:12:WU01:FS00:0x21:Completed 315000 out of 500000 steps (63%)
07:18:47:WU01:FS00:0x21:Completed 320000 out of 500000 steps (64%)
07:19:25:WU01:FS00:0x21:Completed 325000 out of 500000 steps (65%)
07:20:00:WU01:FS00:0x21:Completed 330000 out of 500000 steps (66%)
07:20:35:WU01:FS00:0x21:Completed 335000 out of 500000 steps (67%)
07:21:09:WU01:FS00:0x21:Completed 340000 out of 500000 steps (68%)
07:21:47:WU01:FS00:0x21:Completed 345000 out of 500000 steps (69%)
07:22:21:WU01:FS00:0x21:Completed 350000 out of 500000 steps (70%)
07:22:56:WU01:FS00:0x21:Completed 355000 out of 500000 steps (71%)
07:23:31:WU01:FS00:0x21:Completed 360000 out of 500000 steps (72%)
07:24:10:WU01:FS00:0x21:Completed 365000 out of 500000 steps (73%)
07:24:44:WU01:FS00:0x21:Completed 370000 out of 500000 steps (74%)
07:25:19:WU01:FS00:0x21:Completed 375000 out of 500000 steps (75%)
07:25:53:WU01:FS00:0x21:Completed 380000 out of 500000 steps (76%)
07:26:31:WU01:FS00:0x21:Completed 385000 out of 500000 steps (77%)
07:27:06:WU01:FS00:0x21:Completed 390000 out of 500000 steps (78%)
07:27:40:WU01:FS00:0x21:Completed 395000 out of 500000 steps (79%)
07:28:15:WU01:FS00:0x21:Completed 400000 out of 500000 steps (80%)
07:28:53:WU01:FS00:0x21:Completed 405000 out of 500000 steps (81%)
07:29:26:WU01:FS00:0x21:Completed 410000 out of 500000 steps (82%)
07:30:00:WU01:FS00:0x21:Completed 415000 out of 500000 steps (83%)
07:30:34:WU01:FS00:0x21:Completed 420000 out of 500000 steps (84%)
07:31:13:WU01:FS00:0x21:Completed 425000 out of 500000 steps (85%)
07:31:48:WU01:FS00:0x21:Completed 430000 out of 500000 steps (86%)
07:32:22:WU01:FS00:0x21:Completed 435000 out of 500000 steps (87%)
07:32:56:WU01:FS00:0x21:Completed 440000 out of 500000 steps (88%)
07:33:35:WU01:FS00:0x21:Completed 445000 out of 500000 steps (89%)
07:34:09:WU01:FS00:0x21:Completed 450000 out of 500000 steps (90%)
07:34:42:WU01:FS00:0x21:Completed 455000 out of 500000 steps (91%)
07:35:17:WU01:FS00:0x21:Completed 460000 out of 500000 steps (92%)
07:35:55:WU01:FS00:0x21:Completed 465000 out of 500000 steps (93%)
07:35:56:WU02:FS00:Connecting to 65.254.110.245:8080
07:35:57:WU02:FS00:Assigned to work server 155.247.166.220
07:35:57:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 155.247.166.220
07:35:57:WU02:FS00:Connecting to 155.247.166.220:8080
07:35:58:WU02:FS00:Downloading 15.84MiB
07:36:25:WU02:FS00:Download 0.39%
07:36:29:WU01:FS00:0x21:Completed 470000 out of 500000 steps (94%)
07:37:03:WU01:FS00:0x21:Completed 475000 out of 500000 steps (95%)
07:37:37:WU01:FS00:0x21:Completed 480000 out of 500000 steps (96%)
07:38:15:WU01:FS00:0x21:Completed 485000 out of 500000 steps (97%)
07:38:48:WU01:FS00:0x21:Completed 490000 out of 500000 steps (98%)
07:39:22:WU01:FS00:0x21:Completed 495000 out of 500000 steps (99%)
07:39:56:WU01:FS00:0x21:Completed 500000 out of 500000 steps (100%)
07:40:01:WU01:FS00:0x21:Saving result file logfile_01.txt
07:40:01:WU01:FS00:0x21:Saving result file checkpointState.xml
07:40:01:WU01:FS00:0x21:Saving result file checkpt.crc
07:40:01:WU01:FS00:0x21:Saving result file log.txt
07:40:01:WU01:FS00:0x21:Saving result file positions.xtc
07:40:01:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
07:40:01:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:40:01:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14230 run:374 clone:0 gen:46 core:0x21 unit:0x0000004080fccb0a5d654ddf126ec99d
07:40:01:WU01:FS00:Uploading 51.57MiB to 128.252.203.10
07:40:01:WU01:FS00:Connecting to 128.252.203.10:8080
07:40:07:WU01:FS00:Upload 8.12%
07:40:13:WU01:FS00:Upload 32.72%
07:40:19:WU01:FS00:Upload 51.26%
07:40:25:WU01:FS00:Upload 69.08%
07:40:31:WU01:FS00:Upload 90.53%
07:40:35:WU01:FS00:Upload complete
07:40:35:WU01:FS00:Server responded WORK_ACK (400)
07:40:35:WU01:FS00:Final credit estimate, 46915.00 points
07:40:35:WU01:FS00:Cleaning up

[HaloJones]

155.247.166.220 is the same server causing my servers to just


stop

[FaaR]

Also seeing cases of downloading simply stopping, forever. Seemingly for no reason. Like here. I already quit F@H just prior because it wouldn't fetch new WUs, and then it completes just 1 additional WU and again stops... *shrug*

Never seen this behavior before until now. Not sure what's going on.

Code: Select all

*********************** Log Started 2019-09-26T08:30:35Z ***********************
08:31:46:FS01:Unpaused
08:31:51:WU00:FS01:Connecting to 65.254.110.245:80
08:31:53:WU00:FS01:Assigned to work server 128.252.203.10
08:31:53:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:[Radeon Rx vega] from 128.252.203.10
08:31:53:WU00:FS01:Connecting to 128.252.203.10:8080
08:31:54:WU00:FS01:Downloading 70.39MiB
08:32:00:WU00:FS01:Download 71.83%
08:32:01:WU00:FS01:Download complete
08:32:01:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14230 run:579 clone:1 gen:12 core:0x21 unit:0x0000001080fccb0a5d654bd98ca3fd7b
08:32:02:WU00:FS01:Starting
08:32:02:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/faarg/AppData/Roaming/FAHClient/cores/cores.foldingathome.org/Win32/AMD64/ATI/R600/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 20236 -checkpoint 15 -gpu 0 -gpu-vendor ati
08:32:02:WU00:FS01:Started FahCore on PID 6016
08:32:02:WU00:FS01:Core PID:13868
08:32:02:WU00:FS01:FahCore 0x21 started
08:32:02:WU00:FS01:0x21:*********************** Log Started 2019-09-26T08:32:02Z ***********************
08:32:02:WU00:FS01:0x21:Project: 14230 (Run 579, Clone 1, Gen 12)
08:32:02:WU00:FS01:0x21:Unit: 0x0000001080fccb0a5d654bd98ca3fd7b
08:32:02:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
08:32:02:WU00:FS01:0x21:Machine: 1
08:32:02:WU00:FS01:0x21:Reading tar file core.xml
08:32:02:WU00:FS01:0x21:Reading tar file integrator.xml
08:32:02:WU00:FS01:0x21:Reading tar file state.xml
08:32:03:WU00:FS01:0x21:Reading tar file system.xml
08:32:04:WU00:FS01:0x21:Digital signatures verified
08:32:04:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
08:32:04:WU00:FS01:0x21:Version 0.0.20
08:32:29:WU00:FS01:0x21:Completed 0 out of 500000 steps (0%)
08:32:29:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
08:33:02:WU00:FS01:0x21:Completed 5000 out of 500000 steps (1%)
08:33:36:WU00:FS01:0x21:Completed 10000 out of 500000 steps (2%)
08:34:09:WU00:FS01:0x21:Completed 15000 out of 500000 steps (3%)
08:34:42:WU00:FS01:0x21:Completed 20000 out of 500000 steps (4%)
08:35:23:WU00:FS01:0x21:Completed 25000 out of 500000 steps (5%)
08:35:56:WU00:FS01:0x21:Completed 30000 out of 500000 steps (6%)
08:36:29:WU00:FS01:0x21:Completed 35000 out of 500000 steps (7%)
08:37:04:WU00:FS01:0x21:Completed 40000 out of 500000 steps (8%)
08:37:44:WU00:FS01:0x21:Completed 45000 out of 500000 steps (9%)
08:38:19:WU00:FS01:0x21:Completed 50000 out of 500000 steps (10%)
08:38:52:WU00:FS01:0x21:Completed 55000 out of 500000 steps (11%)
08:39:27:WU00:FS01:0x21:Completed 60000 out of 500000 steps (12%)
08:40:07:WU00:FS01:0x21:Completed 65000 out of 500000 steps (13%)
08:40:41:WU00:FS01:0x21:Completed 70000 out of 500000 steps (14%)
08:41:15:WU00:FS01:0x21:Completed 75000 out of 500000 steps (15%)
08:41:49:WU00:FS01:0x21:Completed 80000 out of 500000 steps (16%)
08:42:30:WU00:FS01:0x21:Completed 85000 out of 500000 steps (17%)
08:43:04:WU00:FS01:0x21:Completed 90000 out of 500000 steps (18%)
08:43:38:WU00:FS01:0x21:Completed 95000 out of 500000 steps (19%)
08:44:12:WU00:FS01:0x21:Completed 100000 out of 500000 steps (20%)
08:44:53:WU00:FS01:0x21:Completed 105000 out of 500000 steps (21%)
08:45:28:WU00:FS01:0x21:Completed 110000 out of 500000 steps (22%)
08:46:01:WU00:FS01:0x21:Completed 115000 out of 500000 steps (23%)
08:46:36:WU00:FS01:0x21:Completed 120000 out of 500000 steps (24%)
08:47:17:WU00:FS01:0x21:Completed 125000 out of 500000 steps (25%)
08:47:51:WU00:FS01:0x21:Completed 130000 out of 500000 steps (26%)
08:48:25:WU00:FS01:0x21:Completed 135000 out of 500000 steps (27%)
08:48:59:WU00:FS01:0x21:Completed 140000 out of 500000 steps (28%)
08:49:40:WU00:FS01:0x21:Completed 145000 out of 500000 steps (29%)
08:50:14:WU00:FS01:0x21:Completed 150000 out of 500000 steps (30%)
08:50:48:WU00:FS01:0x21:Completed 155000 out of 500000 steps (31%)
08:51:22:WU00:FS01:0x21:Completed 160000 out of 500000 steps (32%)
08:52:03:WU00:FS01:0x21:Completed 165000 out of 500000 steps (33%)
08:52:37:WU00:FS01:0x21:Completed 170000 out of 500000 steps (34%)
08:53:11:WU00:FS01:0x21:Completed 175000 out of 500000 steps (35%)
08:53:46:WU00:FS01:0x21:Completed 180000 out of 500000 steps (36%)
08:54:27:WU00:FS01:0x21:Completed 185000 out of 500000 steps (37%)
08:55:01:WU00:FS01:0x21:Completed 190000 out of 500000 steps (38%)
08:55:35:WU00:FS01:0x21:Completed 195000 out of 500000 steps (39%)
08:56:09:WU00:FS01:0x21:Completed 200000 out of 500000 steps (40%)
08:56:50:WU00:FS01:0x21:Completed 205000 out of 500000 steps (41%)
08:57:25:WU00:FS01:0x21:Completed 210000 out of 500000 steps (42%)
08:57:58:WU00:FS01:0x21:Completed 215000 out of 500000 steps (43%)
08:58:32:WU00:FS01:0x21:Completed 220000 out of 500000 steps (44%)
08:59:13:WU00:FS01:0x21:Completed 225000 out of 500000 steps (45%)
08:59:48:WU00:FS01:0x21:Completed 230000 out of 500000 steps (46%)
09:00:21:WU00:FS01:0x21:Completed 235000 out of 500000 steps (47%)
09:00:56:WU00:FS01:0x21:Completed 240000 out of 500000 steps (48%)
09:01:37:WU00:FS01:0x21:Completed 245000 out of 500000 steps (49%)
09:02:11:WU00:FS01:0x21:Completed 250000 out of 500000 steps (50%)
09:02:45:WU00:FS01:0x21:Completed 255000 out of 500000 steps (51%)
09:03:19:WU00:FS01:0x21:Completed 260000 out of 500000 steps (52%)
09:04:00:WU00:FS01:0x21:Completed 265000 out of 500000 steps (53%)
09:04:34:WU00:FS01:0x21:Completed 270000 out of 500000 steps (54%)
09:05:08:WU00:FS01:0x21:Completed 275000 out of 500000 steps (55%)
09:05:42:WU00:FS01:0x21:Completed 280000 out of 500000 steps (56%)
09:06:23:WU00:FS01:0x21:Completed 285000 out of 500000 steps (57%)
09:06:57:WU00:FS01:0x21:Completed 290000 out of 500000 steps (58%)
09:07:31:WU00:FS01:0x21:Completed 295000 out of 500000 steps (59%)
09:08:05:WU00:FS01:0x21:Completed 300000 out of 500000 steps (60%)
09:08:46:WU00:FS01:0x21:Completed 305000 out of 500000 steps (61%)
09:09:21:WU00:FS01:0x21:Completed 310000 out of 500000 steps (62%)
09:09:54:WU00:FS01:0x21:Completed 315000 out of 500000 steps (63%)
09:10:29:WU00:FS01:0x21:Completed 320000 out of 500000 steps (64%)
09:11:10:WU00:FS01:0x21:Completed 325000 out of 500000 steps (65%)
09:11:44:WU00:FS01:0x21:Completed 330000 out of 500000 steps (66%)
09:12:18:WU00:FS01:0x21:Completed 335000 out of 500000 steps (67%)
09:12:52:WU00:FS01:0x21:Completed 340000 out of 500000 steps (68%)
09:13:33:WU00:FS01:0x21:Completed 345000 out of 500000 steps (69%)
09:14:07:WU00:FS01:0x21:Completed 350000 out of 500000 steps (70%)
09:14:41:WU00:FS01:0x21:Completed 355000 out of 500000 steps (71%)
09:15:15:WU00:FS01:0x21:Completed 360000 out of 500000 steps (72%)
09:15:56:WU00:FS01:0x21:Completed 365000 out of 500000 steps (73%)
09:16:30:WU00:FS01:0x21:Completed 370000 out of 500000 steps (74%)
09:17:04:WU00:FS01:0x21:Completed 375000 out of 500000 steps (75%)
09:17:38:WU00:FS01:0x21:Completed 380000 out of 500000 steps (76%)
09:18:19:WU00:FS01:0x21:Completed 385000 out of 500000 steps (77%)
09:18:54:WU00:FS01:0x21:Completed 390000 out of 500000 steps (78%)
09:19:27:WU00:FS01:0x21:Completed 395000 out of 500000 steps (79%)
09:20:02:WU00:FS01:0x21:Completed 400000 out of 500000 steps (80%)
09:20:43:WU00:FS01:0x21:Completed 405000 out of 500000 steps (81%)
09:21:17:WU00:FS01:0x21:Completed 410000 out of 500000 steps (82%)
09:21:50:WU00:FS01:0x21:Completed 415000 out of 500000 steps (83%)
09:22:25:WU00:FS01:0x21:Completed 420000 out of 500000 steps (84%)
09:23:05:WU00:FS01:0x21:Completed 425000 out of 500000 steps (85%)
09:23:39:WU00:FS01:0x21:Completed 430000 out of 500000 steps (86%)
09:24:13:WU00:FS01:0x21:Completed 435000 out of 500000 steps (87%)
09:24:47:WU00:FS01:0x21:Completed 440000 out of 500000 steps (88%)
09:25:28:WU00:FS01:0x21:Completed 445000 out of 500000 steps (89%)
09:26:02:WU00:FS01:0x21:Completed 450000 out of 500000 steps (90%)
09:26:36:WU00:FS01:0x21:Completed 455000 out of 500000 steps (91%)
09:27:11:WU00:FS01:0x21:Completed 460000 out of 500000 steps (92%)
09:27:52:WU00:FS01:0x21:Completed 465000 out of 500000 steps (93%)
09:28:26:WU00:FS01:0x21:Completed 470000 out of 500000 steps (94%)
09:29:00:WU00:FS01:0x21:Completed 475000 out of 500000 steps (95%)
09:29:34:WU00:FS01:0x21:Completed 480000 out of 500000 steps (96%)
09:30:15:WU00:FS01:0x21:Completed 485000 out of 500000 steps (97%)
09:30:49:WU00:FS01:0x21:Completed 490000 out of 500000 steps (98%)
09:31:23:WU00:FS01:0x21:Completed 495000 out of 500000 steps (99%)
09:31:23:WU04:FS01:Connecting to 65.254.110.245:80
09:31:23:WU04:FS01:Assigned to work server 155.247.166.220
09:31:23:WU04:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:[Radeon Rx vega] from 155.247.166.220
09:31:23:WU04:FS01:Connecting to 155.247.166.220:8080
09:31:24:WU04:FS01:Downloading 6.06MiB
09:31:32:WU04:FS01:Download 3.09%
09:31:57:WU00:FS01:0x21:Completed 500000 out of 500000 steps (100%)
09:32:05:WU00:FS01:0x21:Saving result file logfile_01.txt
09:32:06:WU00:FS01:0x21:Saving result file checkpointState.xml
09:32:06:WU00:FS01:0x21:Saving result file checkpt.crc
09:32:06:WU00:FS01:0x21:Saving result file log.txt
09:32:06:WU00:FS01:0x21:Saving result file positions.xtc
09:32:06:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
09:32:07:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:32:07:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14230 run:579 clone:1 gen:12 core:0x21 unit:0x0000001080fccb0a5d654bd98ca3fd7b
09:32:07:WU00:FS01:Uploading 52.01MiB to 128.252.203.10
09:32:07:WU00:FS01:Connecting to 128.252.203.10:8080
09:32:13:WU00:FS01:Upload 2.52%
09:32:19:WU00:FS01:Upload 5.89%
09:32:19:WU04:FS01:Download 4.12%
09:32:25:WU00:FS01:Upload 8.53%
09:32:27:WU04:FS01:Download 6.19%
09:32:31:WU00:FS01:Upload 11.42%
09:32:37:WU00:FS01:Upload 13.70%
09:32:38:WU04:FS01:Download 9.28%
09:32:43:WU00:FS01:Upload 16.46%
09:32:45:WU04:FS01:Download 11.34%
09:32:49:WU00:FS01:Upload 20.19%
09:32:51:WU04:FS01:Download 13.40%
09:32:55:WU00:FS01:Upload 23.31%
09:33:01:WU00:FS01:Upload 27.28%
09:33:01:WU04:FS01:Download 15.46%
09:33:07:WU00:FS01:Upload 32.33%
09:33:09:WU04:FS01:Download 17.53%
09:33:13:WU00:FS01:Upload 35.93%
09:33:19:WU00:FS01:Upload 39.66%
09:33:25:WU00:FS01:Upload 43.14%
09:33:31:WU00:FS01:Upload 49.75%
09:33:37:WU00:FS01:Upload 53.72%
09:33:38:WU04:FS01:Download 18.56%
09:33:43:WU00:FS01:Upload 56.24%
09:33:49:WU00:FS01:Upload 58.88%
09:33:55:WU00:FS01:Upload 63.09%
09:34:01:WU00:FS01:Upload 67.41%
09:34:07:WU00:FS01:Upload 73.54%
09:34:08:WU04:FS01:Download 19.59%
09:34:13:WU00:FS01:Upload 77.63%
09:34:18:WU04:FS01:Download 21.65%
09:34:19:WU00:FS01:Upload 81.47%
09:34:25:WU00:FS01:Upload 84.84%
09:34:27:WU04:FS01:Download 23.71%
09:34:31:WU00:FS01:Upload 88.92%
09:34:37:WU00:FS01:Upload 93.97%
09:34:43:WU00:FS01:Upload 96.98%
09:34:48:WU00:FS01:Upload complete
09:34:49:WU00:FS01:Server responded WORK_ACK (400)
09:34:49:WU00:FS01:Final credit estimate, 45726.00 points
09:34:49:WU00:FS01:Cleaning up
09:35:19:WU04:FS01:Download 25.77%
09:35:27:WU04:FS01:Download 27.84%
09:35:58:WU04:FS01:Download 28.87%
09:42:29:FS01:Finishing

[Shirty]

I too am seeing 15 meg downloads taking >10 minutes from time to time. Not normal, nothing wrong with download speeds elsewhere.

[goodyca]

I am seeing the same problem from server 155.247.166.219.

[JimF]

I have been out of work for two hours. The last log entry just says "Cleaning up".

[Sven]

Same here, very slow download, sometimes even get stuck. Have to restart the client.

This on different PC's since yesterday.

Code: Select all

12:58:55:WU02:FS01:Assigned to work server 155.247.166.220
12:58:55:WU01:FS00:Connecting to 155.247.166.220:8080
12:58:55:WU02:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 155.247.166.220
12:58:55:WU02:FS01:Connecting to 155.247.166.220:8080
12:58:56:WU02:FS01:Downloading 15.60MiB
12:58:57:ERROR:WU01:FS00:Exception: Server did not assign work unit
12:58:57:WU01:FS00:Connecting to 65.254.110.245:8080
12:58:58:WU01:FS00:Assigned to work server 128.252.203.10
12:58:58:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 128.252.203.10
12:58:58:WU01:FS00:Connecting to 128.252.203.10:8080
12:58:59:WU01:FS00:Downloading 70.42MiB
12:59:02:WU02:FS01:Download 0.80%
12:59:05:WU01:FS00:Download 89.20%
12:59:05:WU01:FS00:Download complete
12:59:05:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14250 run:397 clone:1 gen:18 core:0x21 unit:0x0000001880fccb0a5d6ed218425849ab
12:59:07:WU00:FS00:0x21:Completed 980000 out of 1000000 steps (98%)
12:59:07:WU00:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:59:09:WU02:FS01:Download 1.60%
12:59:18:WU02:FS01:Download 2.40%
12:59:26:WU02:FS01:Download 3.20%
12:59:32:WU02:FS01:Download 3.61%
12:59:39:WU02:FS01:Download 4.41%
12:59:45:WU02:FS01:Download 5.21%
12:59:52:WU02:FS01:Download 5.61%
12:59:59:WU02:FS01:Download 6.41%
13:00:08:WU02:FS01:Download 7.21%
13:00:15:WU02:FS01:Download 8.01%

[Catalina588]

Found an ERROR. Request assistance.

Overnight, 7 of 8 clients stalled. I cannot remember how to bracket log files, so forgive me ...

Note "Exception: Transfer Failure" at 13:44:45. This happened after a reboot this morning. Note, this is the first time I have observed an attempt and success at connecting to an alternate server.

13:38:17:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
13:38:17:WU00:FS00:Connecting to 155.247.166.220:8080
13:38:17:WU01:FS01:Assigned to work server 155.247.166.220
13:38:17:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
13:38:17:WU01:FS01:Connecting to 155.247.166.220:8080
13:38:18:WU00:FS00:Downloading 6.06MiB
13:38:18:WU01:FS01:Downloading 15.58MiB
13:38:24:WU00:FS00:Download 4.12%
13:38:24:WU01:FS01:Download 2.01%
13:38:31:WU00:FS00:Download 6.18%
13:38:32:WU01:FS01:Download 3.61%
13:38:38:WU01:FS01:Download 4.81%
13:38:41:WU00:FS00:Download 8.25%
13:38:44:WU01:FS01:Download 5.61%
13:38:53:WU01:FS01:Download 6.02%
13:39:07:WU01:FS01:Download 6.82%
13:40:09:WU01:FS01:Download 7.22%
13:40:28:WU01:FS01:Download 7.62%
13:40:35:WU01:FS01:Download 8.42%
13:40:41:WU01:FS01:Download 8.82%
13:44:45:WU01:FS01:Download 9.20%
13:44:45:ERROR:WU01:FS01:Exception: Transfer failed
13:44:45:WU01:FS01:Connecting to 65.254.110.245:8080
13:44:45:WU01:FS01:Assigned to work server 128.252.203.10
13:44:45:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 128.252.203.10
13:44:45:WU01:FS01:Connecting to 128.252.203.10:8080
13:44:46:WU01:FS01:Downloading 69.38MiB
13:44:50:WU01:FS01:Download complete

[Catalina588]

Here's an overnight session that ended with the completed work units uploading--proving comms is working--but downloads fail to get beyond 0.4% from 155.247.166.220.

Code: Select all

 01:47:13:WU01:FS00:0x21:Completed 65000 out of 500000 steps (13%)
2:FS01:0x21:Completed 460000 out of 500000 steps (92%)
02:28:04:WU01:FS00:0x21:Completed 320000 out of 500000 steps (64%)
02:28:07:WU02:FS01:0x21:Completed 465000 out of 500000 steps (93%)
02:28:54:WU02:FS01:0x21:Completed 470000 out of 500000 steps (94%)
02:28:54:WU01:FS00:0x21:Completed 325000 out of 500000 steps (65%)
02:29:41:WU02:FS01:0x21:Completed 475000 out of 500000 steps (95%)
02:29:42:WU01:FS00:0x21:Completed 330000 out of 500000 steps (66%)
02:30:28:WU02:FS01:0x21:Completed 480000 out of 500000 steps (96%)
02:30:29:WU01:FS00:0x21:Completed 335000 out of 500000 steps (67%)
02:31:16:WU01:FS00:0x21:Completed 340000 out of 500000 steps (68%)
02:31:19:WU02:FS01:0x21:Completed 485000 out of 500000 steps (97%)
02:32:07:WU02:FS01:0x21:Completed 490000 out of 500000 steps (98%)
02:32:07:WU01:FS00:0x21:Completed 345000 out of 500000 steps (69%)
02:32:54:WU02:FS01:0x21:Completed 495000 out of 500000 steps (99%)
02:32:54:WU01:FS00:0x21:Completed 350000 out of 500000 steps (70%)
02:33:41:WU02:FS01:0x21:Completed 500000 out of 500000 steps (100%)
02:33:41:WU00:FS01:Connecting to 65.254.110.245:8080
02:33:41:WU00:FS01:Assigned to work server 155.247.166.220
02:33:41:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
02:33:41:WU00:FS01:Connecting to 155.247.166.220:8080
02:33:41:WU00:FS01:Downloading 3.41MiB
02:33:41:WU01:FS00:0x21:Completed 355000 out of 500000 steps (71%)
02:33:45:WU02:FS01:0x21:Saving result file logfile_01.txt
02:33:45:WU02:FS01:0x21:Saving result file checkpointState.xml
02:33:45:WU02:FS01:0x21:Saving result file checkpt.crc
02:33:45:WU02:FS01:0x21:Saving result file log.txt
02:33:45:WU02:FS01:0x21:Saving result file positions.xtc
02:33:45:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
02:33:45:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:33:45:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14229 run:1308 clone:0 gen:4 core:0x21 unit:0x0000000480fccb0a5d6553db4d6a57f3
02:33:45:WU02:FS01:Uploading 51.45MiB to 128.252.203.10
02:33:45:WU02:FS01:Connecting to 128.252.203.10:8080
02:33:48:WU00:FS01:Download 10.99%
02:33:51:WU02:FS01:Upload 16.04%
02:33:54:WU00:FS01:Download 16.48%
02:33:57:WU02:FS01:Upload 33.04%
02:34:01:WU00:FS01:Download 23.81%
02:34:03:WU02:FS01:Upload 49.93%
02:34:08:WU00:FS01:Download 31.13%
02:34:09:WU02:FS01:Upload 65.96%
02:34:15:WU02:FS01:Upload 82.73%
02:34:15:WU00:FS01:Download 38.45%
02:34:21:WU00:FS01:Download 43.95%
02:34:21:WU02:FS01:Upload 98.76%
02:34:22:WU02:FS01:Upload complete
02:34:22:WU02:FS01:Server responded WORK_ACK (400)
02:34:22:WU02:FS01:Final credit estimate, 40280.00 points
02:34:22:WU02:FS01:Cleaning up
02:34:27:WU00:FS01:Download 49.44%
02:34:29:WU01:FS00:0x21:Completed 360000 out of 500000 steps (72%)
02:34:33:WU00:FS01:Download 54.94%
02:34:40:WU00:FS01:Download 64.09%
02:34:46:WU00:FS01:Download 69.59%
02:34:52:WU00:FS01:Download 75.08%
02:35:07:WU00:FS01:Download 82.40%
02:35:14:WU00:FS01:Download 89.73%
02:35:19:WU01:FS00:0x21:Completed 365000 out of 500000 steps (73%)
02:35:20:WU00:FS01:Download 91.56%
02:35:27:WU00:FS01:Download 98.88%
02:35:27:WU00:FS01:Download complete
02:35:27:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14187 run:3 clone:79 gen:63 core:0x21 unit:0x000000740002894c5d3897505a4f75b5
02:35:27:WU00:FS01:Starting
02:35:27:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 705 -lifeline 1877 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
02:35:27:WU00:FS01:Started FahCore on PID 3832
02:35:27:WU00:FS01:Core PID:3836
02:35:27:WU00:FS01:FahCore 0x21 started
02:35:28:WU00:FS01:0x21:*********************** Log Started 2019-09-26T02:35:27Z ***********************
02:35:28:WU00:FS01:0x21:Project: 14187 (Run 3, Clone 79, Gen 63)
02:35:28:WU00:FS01:0x21:Unit: 0x000000740002894c5d3897505a4f75b5
02:35:28:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
02:35:28:WU00:FS01:0x21:Machine: 1
02:35:28:WU00:FS01:0x21:Reading tar file core.xml
02:35:28:WU00:FS01:0x21:Reading tar file integrator.xml
02:35:28:WU00:FS01:0x21:Reading tar file state.xml
02:35:28:WU00:FS01:0x21:Reading tar file system.xml
02:35:28:WU00:FS01:0x21:Digital signatures verified
02:35:28:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
02:35:28:WU00:FS01:0x21:Version 0.0.20
02:35:29:WU00:FS01:0x21:Completed 0 out of 12500000 steps (0%)
02:35:29:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:36:06:WU01:FS00:0x21:Completed 370000 out of 500000 steps (74%)
02:36:31:WU00:FS01:0x21:Completed 125000 out of 12500000 steps (1%)

02:55:21:WU01:FS00:0x21:Completed 490000 out of 500000 steps (98%)
02:56:08:WU01:FS00:0x21:Completed 495000 out of 500000 steps (99%)
02:56:18:WU00:FS01:0x21:Completed 2500000 out of 12500000 steps (20%)
02:56:55:WU01:FS00:0x21:Completed 500000 out of 500000 steps (100%)
02:56:56:WU02:FS00:Connecting to 65.254.110.245:8080
02:56:56:WU02:FS00:Assigned to work server 128.252.203.10
02:56:56:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP104 [GeForce GTX 1080] 8873 from 128.252.203.10
02:56:56:WU02:FS00:Connecting to 128.252.203.10:8080
02:56:57:WU02:FS00:Downloading 69.75MiB
02:56:59:WU01:FS00:0x21:Saving result file logfile_01.txt
02:56:59:WU01:FS00:0x21:Saving result file checkpointState.xml
02:56:59:WU01:FS00:0x21:Saving result file checkpt.crc
02:56:59:WU01:FS00:0x21:Saving result file log.txt
02:56:59:WU01:FS00:0x21:Saving result file positions.xtc
02:56:59:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
02:57:00:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:57:00:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14229 run:1381 clone:0 gen:3 core:0x21 unit:0x0000000380fccb0a5d65531861005052
02:57:00:WU01:FS00:Uploading 51.48MiB to 128.252.203.10
02:57:00:WU01:FS00:Connecting to 128.252.203.10:8080
02:57:03:WU02:FS00:Download 32.26%
02:57:04:WU02:FS00:Download complete
02:57:04:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14229 run:222 clone:0 gen:83 core:0x21 unit:0x0000007e80fccb0a5d6553d9c785fd51
02:57:04:WU02:FS00:Starting
02:57:04:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 705 -lifeline 1877 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
02:57:04:WU02:FS00:Started FahCore on PID 3865
02:57:04:WU02:FS00:Core PID:3869
02:57:04:WU02:FS00:FahCore 0x21 started
02:57:05:WU02:FS00:0x21:*********************** Log Started 2019-09-26T02:57:04Z ***********************
02:57:05:WU02:FS00:0x21:Project: 14229 (Run 222, Clone 0, Gen 83)
02:57:05:WU02:FS00:0x21:Unit: 0x0000007e80fccb0a5d6553d9c785fd51
02:57:05:WU02:FS00:0x21:CPU: 0x00000000000000000000000000000000
02:57:05:WU02:FS00:0x21:Machine: 0
02:57:05:WU02:FS00:0x21:Reading tar file core.xml
02:57:05:WU02:FS00:0x21:Reading tar file integrator.xml
02:57:05:WU02:FS00:0x21:Reading tar file state.xml
02:57:05:WU02:FS00:0x21:Reading tar file system.xml
02:57:05:WU02:FS00:0x21:Digital signatures verified
02:57:05:WU02:FS00:0x21:Folding@home GPU Core21 Folding@home Core
02:57:05:WU02:FS00:0x21:Version 0.0.20
02:57:06:WU01:FS00:Upload 15.42%
02:57:12:WU01:FS00:Upload 32.41%
02:57:15:WU02:FS00:0x21:Completed 0 out of 500000 steps (0%)
02:57:15:WU02:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:57:18:WU01:FS00:Upload 48.68%
02:57:21:WU00:FS01:0x21:Completed 2625000 out of 12500000 steps (21%)
02:57:24:WU01:FS00:Upload 64.95%
02:57:30:WU01:FS00:Upload 82.07%
02:57:36:WU01:FS00:Upload 98.34%
02:57:37:WU01:FS00:Upload complete
02:57:37:WU01:FS00:Server responded WORK_ACK (400)
02:57:37:WU01:FS00:Final credit estimate, 40288.00 points
02:57:37:WU01:FS00:Cleaning up
02:58:03:WU02:FS00:0x21:Completed 5000 out of 500000 steps (1%)
02:58:23:WU00:FS01:0x21:Completed 2750000 out of 12500000 steps (22%)
02:58:50:WU02:FS00:0x21:Completed 10000 out of 500000 steps (2%)
04:16:05:WU02:FS00:0x21:Completed 490000 out of 500000 steps (98%)
04:16:34:WU00:FS01:0x21:Completed 12125000 out of 12500000 steps (97%)
04:16:52:WU02:FS00:0x21:Completed 495000 out of 500000 steps (99%)
04:17:36:WU00:FS01:0x21:Completed 12250000 out of 12500000 steps (98%)
04:17:39:WU02:FS00:0x21:Completed 500000 out of 500000 steps (100%)
04:17:40:WU01:FS00:Connecting to 65.254.110.245:8080
04:17:40:WU01:FS00:Assigned to work server 128.252.203.10
04:17:40:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP104 [GeForce GTX 1080] 8873 from 128.252.203.10
04:17:40:WU01:FS00:Connecting to 128.252.203.10:8080
04:17:40:WU01:FS00:Downloading 70.41MiB
04:17:43:WU02:FS00:0x21:Saving result file logfile_01.txt
04:17:43:WU02:FS00:0x21:Saving result file checkpointState.xml
04:17:43:WU02:FS00:0x21:Saving result file checkpt.crc
04:17:43:WU02:FS00:0x21:Saving result file log.txt
04:17:43:WU02:FS00:0x21:Saving result file positions.xtc
04:17:44:WU02:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
04:17:44:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:17:44:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14229 run:222 clone:0 gen:83 core:0x21 unit:0x0000007e80fccb0a5d6553d9c785fd51
04:17:44:WU02:FS00:Uploading 51.43MiB to 128.252.203.10
04:17:44:WU02:FS00:Connecting to 128.252.203.10:8080
04:17:46:WU01:FS00:Download 22.19%
04:17:48:WU01:FS00:Download complete
04:17:48:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14250 run:74 clone:1 gen:23 core:0x21 unit:0x0000001d80fccb0a5d6ed2170ecf9bfa
04:17:48:WU01:FS00:Starting
04:17:48:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 1877 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
04:17:48:WU01:FS00:Started FahCore on PID 5036
04:17:48:WU01:FS00:Core PID:5040
04:17:48:WU01:FS00:FahCore 0x21 started
04:17:48:WU01:FS00:0x21:*********************** Log Started 2019-09-26T04:17:48Z ***********************
04:17:48:WU01:FS00:0x21:Project: 14250 (Run 74, Clone 1, Gen 23)
04:17:48:WU01:FS00:0x21:Unit: 0x0000001d80fccb0a5d6ed2170ecf9bfa
04:17:48:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
04:17:48:WU01:FS00:0x21:Machine: 0
04:17:48:WU01:FS00:0x21:Reading tar file core.xml
04:17:48:WU01:FS00:0x21:Reading tar file integrator.xml
04:17:48:WU01:FS00:0x21:Reading tar file state.xml
04:17:48:WU01:FS00:0x21:Reading tar file system.xml
04:17:48:WU01:FS00:0x21:Digital signatures verified
04:17:48:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
04:17:48:WU01:FS00:0x21:Version 0.0.20
04:17:50:WU02:FS00:Upload 16.89%
04:17:56:WU02:FS00:Upload 33.79%
04:17:59:WU01:FS00:0x21:Completed 0 out of 1000000 steps (0%)
04:17:59:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:18:02:WU02:FS00:Upload 50.80%
04:18:08:WU02:FS00:Upload 67.69%
04:18:14:WU02:FS00:Upload 83.37%
04:18:21:WU02:FS00:Upload complete
04:18:21:WU02:FS00:Server responded WORK_ACK (400)
04:18:21:WU02:FS00:Final credit estimate, 40202.00 points
04:18:21:WU02:FS00:Cleaning up
04:18:39:WU00:FS01:0x21:Completed 12375000 out of 12500000 steps (99%)
04:19:34:WU01:FS00:0x21:Completed 10000 out of 1000000 steps (1%)
04:19:41:WU00:FS01:0x21:Completed 12500000 out of 12500000 steps (100%)
04:19:41:WU00:FS01:0x21:Saving result file logfile_01.txt
04:19:41:WU00:FS01:0x21:Saving result file checkpointState.xml
04:19:41:WU00:FS01:0x21:Saving result file checkpt.crc
04:19:41:WU00:FS01:0x21:Saving result file log.txt
04:19:41:WU00:FS01:0x21:Saving result file positions.xtc
04:19:41:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
04:19:42:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:19:42:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14187 run:3 clone:79 gen:63 core:0x21 unit:0x000000740002894c5d3897505a4f75b5
04:19:42:WU00:FS01:Uploading 3.63MiB to 155.247.166.220
04:19:42:WU00:FS01:Connecting to 155.247.166.220:8080
04:19:42:WU02:FS01:Connecting to 65.254.110.245:8080
04:19:42:WU02:FS01:Assigned to work server 155.247.166.220
04:19:42:WU02:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
04:19:42:WU02:FS01:Connecting to 155.247.166.220:8080
04:19:43:WU02:FS01:Downloading 15.61MiB
04:19:45:WU00:FS01:Upload complete
04:19:45:WU00:FS01:Server responded WORK_ACK (400)
04:19:45:WU00:FS01:Final credit estimate, 56649.00 points
04:19:45:WU00:FS01:Cleaning up
04:19:50:WU02:FS01:Download 2.00%
04:19:57:WU02:FS01:Download 2.80%
04:20:03:WU02:FS01:Download 3.20%
04:20:09:WU02:FS01:Download 4.80%
04:20:16:WU02:FS01:Download 6.81%
04:20:22:WU02:FS01:Download 9.21%
04:20:28:WU02:FS01:Download 11.61%
04:20:34:WU02:FS01:Download 14.01%
04:20:40:WU02:FS01:Download 16.41%
04:20:47:WU02:FS01:Download 18.82%
04:20:54:WU02:FS01:Download 21.22%
04:21:00:WU02:FS01:Download 22.82%
04:21:06:WU02:FS01:Download 24.42%
04:21:09:WU01:FS00:0x21:Completed 20000 out of 1000000 steps (2%)
04:21:12:WU02:FS01:Download 26.02%
04:21:18:WU02:FS01:Download 28.02%
04:21:25:WU02:FS01:Download 30.03%
04:21:31:WU02:FS01:Download 32.03%
04:21:37:WU02:FS01:Download 33.23%
04:21:43:WU02:FS01:Download 33.63%
04:21:50:WU02:FS01:Download 34.43%
04:21:56:WU02:FS01:Download 35.63%
04:22:09:WU02:FS01:Download 36.83%
04:22:27:WU02:FS01:Download 37.23%
04:22:34:WU02:FS01:Download 37.63%
04:22:41:WU02:FS01:Download 39.63%
04:22:47:WU01:FS00:0x21:Completed 30000 out of 1000000 steps (3%)
04:22:49:WU02:FS01:Download 41.23%
04:22:55:WU02:FS01:Download 42.44%
04:23:01:WU02:FS01:Download 44.44%
04:23:07:WU02:FS01:Download 46.04%
04:23:15:WU02:FS01:Download 47.64%
04:23:21:WU02:FS01:Download 48.84%
04:23:31:WU02:FS01:Download 50.44%
04:23:37:WU02:FS01:Download 51.64%
04:23:43:WU02:FS01:Download 53.24%
04:23:49:WU02:FS01:Download 55.25%
04:23:56:WU02:FS01:Download 57.25%
04:24:03:WU02:FS01:Download 58.45%
04:24:10:WU02:FS01:Download 60.05%
04:24:16:WU02:FS01:Download 62.05%
04:24:22:WU02:FS01:Download 63.65%
04:24:22:WU01:FS00:0x21:Completed 40000 out of 1000000 steps (4%)
04:24:31:WU02:FS01:Download 64.45%
04:24:37:WU02:FS01:Download 66.06%
04:24:43:WU02:FS01:Download 68.06%
04:24:51:WU02:FS01:Download 70.06%
04:24:57:WU02:FS01:Download 72.06%
04:25:03:WU02:FS01:Download 74.06%
04:25:09:WU02:FS01:Download 76.06%
04:25:15:WU02:FS01:Download 77.67%
04:25:21:WU02:FS01:Download 79.67%
04:25:27:WU02:FS01:Download 81.27%
04:25:34:WU02:FS01:Download 82.87%
04:25:42:WU02:FS01:Download 84.87%
04:25:53:WU02:FS01:Download 86.07%
04:25:59:WU02:FS01:Download 88.07%
04:26:00:WU01:FS00:0x21:Completed 50000 out of 1000000 steps (5%)
04:26:05:WU02:FS01:Download 89.68%
04:26:13:WU02:FS01:Download 90.48%
04:26:21:WU02:FS01:Download 91.68%
04:26:29:WU02:FS01:Download 93.28%
04:26:35:WU02:FS01:Download 95.28%
04:26:41:WU02:FS01:Download 96.88%
04:26:47:WU02:FS01:Download 98.08%
04:26:53:WU02:FS01:Download 100.00%
04:26:53:WU02:FS01:Download complete
04:26:53:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14180 run:9 clone:311 gen:36 core:0x21 unit:0x0000003b0002894c5d3b55bd16035ea7
04:26:53:WU02:FS01:Starting
04:26:53:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 705 -lifeline 1877 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
04:26:53:WU02:FS01:Started FahCore on PID 5065
04:26:53:WU02:FS01:Core PID:5069
04:26:53:WU02:FS01:FahCore 0x21 started
04:26:54:WU02:FS01:0x21:*********************** Log Started 2019-09-26T04:26:53Z ***********************
04:26:54:WU02:FS01:0x21:Project: 14180 (Run 9, Clone 311, Gen 36)
04:26:54:WU02:FS01:0x21:Unit: 0x0000003b0002894c5d3b55bd16035ea7
04:26:54:WU02:FS01:0x21:CPU: 0x00000000000000000000000000000000
04:26:54:WU02:FS01:0x21:Machine: 1
04:26:54:WU02:FS01:0x21:Reading tar file core.xml
04:26:54:WU02:FS01:0x21:Reading tar file integrator.xml
04:26:54:WU02:FS01:0x21:Reading tar file state.xml
04:26:54:WU02:FS01:0x21:Reading tar file system.xml
04:26:54:WU02:FS01:0x21:Digital signatures verified
04:26:54:WU02:FS01:0x21:Folding@home GPU Core21 Folding@home Core
04:26:54:WU02:FS01:0x21:Version 0.0.20
04:26:57:WU02:FS01:0x21:Completed 0 out of 12500000 steps (0%)
04:26:57:WU02:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:27:35:WU01:FS00:0x21:Completed 60000 out of 1000000 steps (6%)
04:29:13:WU01:FS00:0x21:Completed 70000 out of 1000000 steps (7%)
04:30:03:WU02:FS01:0x21:Completed 125000 out of 12500000 steps (1%)
04:30:48:WU01:FS00:0x21:Completed 80000 out of 1000000 steps (8%)
04:32:26:WU01:FS00:0x21:Completed 90000 out of 1000000 steps (9%)
04:33:09:WU02:FS01:0x21:Completed 250000 out of 12500000 steps (2%)
04:34:01:WU01:FS00:0x21:Completed 100000 out of 1000000 steps (10%)
04:35:40:WU01:FS00:0x21:Completed 110000 out of 1000000 steps (11%)
04:36:17:WU02:FS01:0x21:Completed 375000 out of 12500000 steps (3%)
04:37:14:WU01:FS00:0x21:Completed 120000 out of 1000000 steps (12%)
04:38:53:WU01:FS00:0x21:Completed 130000 out of 1000000 steps (13%)
04:39:23:WU02:FS01:0x21:Completed 500000 out of 12500000 steps (4%)
04:40:28:WU01:FS00:0x21:Completed 140000 out of 1000000 steps (14%)
04:42:06:WU01:FS00:0x21:Completed 150000 out of 1000000 steps (15%)
04:42:31:WU02:FS01:0x21:Completed 625000 out of 12500000 steps (5%)
04:43:41:WU01:FS00:0x21:Completed 160000 out of 1000000 steps (16%)
04:45:19:WU01:FS00:0x21:Completed 170000 out of 1000000 steps (17%)
04:45:38:WU02:FS01:0x21:Completed 750000 out of 12500000 steps (6%)
04:46:55:WU01:FS00:0x21:Completed 180000 out of 1000000 steps (18%)
04:48:33:WU01:FS00:0x21:Completed 190000 out of 1000000 steps (19%)
04:48:46:WU02:FS01:0x21:Completed 875000 out of 12500000 steps (7%)
04:50:08:WU01:FS00:0x21:Completed 200000 out of 1000000 steps (20%)
04:51:46:WU01:FS00:0x21:Completed 210000 out of 1000000 steps (21%)
04:51:53:WU02:FS01:0x21:Completed 1000000 out of 12500000 steps (8%)
04:53:21:WU01:FS00:0x21:Completed 220000 out of 1000000 steps (22%)
04:55:00:WU01:FS00:0x21:Completed 230000 out of 1000000 steps (23%)
04:55:01:WU02:FS01:0x21:Completed 1125000 out of 12500000 steps (9%)
04:56:35:WU01:FS00:0x21:Completed 240000 out of 1000000 steps (24%)
04:58:08:WU02:FS01:0x21:Completed 1250000 out of 12500000 steps (10%)
04:58:13:WU01:FS00:0x21:Completed 250000 out of 1000000 steps (25%)
04:59:48:WU01:FS00:0x21:Completed 260000 out of 1000000 steps (26%)
05:01:16:WU02:FS01:0x21:Completed 1375000 out of 12500000 steps (11%)
05:01:26:WU01:FS00:0x21:Completed 270000 out of 1000000 steps (27%)
05:03:02:WU01:FS00:0x21:Completed 280000 out of 1000000 steps (28%)
05:04:23:WU02:FS01:0x21:Completed 1500000 out of 12500000 steps (12%)
05:04:40:WU01:FS00:0x21:Completed 290000 out of 1000000 steps (29%)
05:06:15:WU01:FS00:0x21:Completed 300000 out of 1000000 steps (30%)
05:07:31:WU02:FS01:0x21:Completed 1625000 out of 12500000 steps (13%)
05:07:54:WU01:FS00:0x21:Completed 310000 out of 1000000 steps (31%)
05:09:28:WU01:FS00:0x21:Completed 320000 out of 1000000 steps (32%)
05:10:38:WU02:FS01:0x21:Completed 1750000 out of 12500000 steps (14%)
05:11:07:WU01:FS00:0x21:Completed 330000 out of 1000000 steps (33%)
05:12:42:WU01:FS00:0x21:Completed 340000 out of 1000000 steps (34%)
05:13:46:WU02:FS01:0x21:Completed 1875000 out of 12500000 steps (15%)
05:14:20:WU01:FS00:0x21:Completed 350000 out of 1000000 steps (35%)
05:15:55:WU01:FS00:0x21:Completed 360000 out of 1000000 steps (36%)
05:16:53:WU02:FS01:0x21:Completed 2000000 out of 12500000 steps (16%)
05:17:34:WU01:FS00:0x21:Completed 370000 out of 1000000 steps (37%)
******************************* Date: 2019-09-26 *******************************
05:19:09:WU01:FS00:0x21:Completed 380000 out of 1000000 steps (38%)
06:56:57:WU02:FS01:0x21:Completed 6000000 out of 12500000 steps (48%)
06:57:33:WU01:FS00:0x21:Completed 990000 out of 1000000 steps (99%)
06:59:09:WU01:FS00:0x21:Completed 1000000 out of 1000000 steps (100%)
06:59:09:WU00:FS00:Connecting to 65.254.110.245:8080
06:59:09:WU00:FS00:Assigned to work server 155.247.166.220
06:59:09:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
06:59:09:WU00:FS00:Connecting to 155.247.166.220:8080
06:59:09:WU00:FS00:Downloading 3.41MiB
06:59:13:WU01:FS00:0x21:Saving result file logfile_01.txt
06:59:13:WU01:FS00:0x21:Saving result file checkpointState.xml
06:59:13:WU01:FS00:0x21:Saving result file checkpt.crc
06:59:13:WU01:FS00:0x21:Saving result file log.txt
06:59:13:WU01:FS00:0x21:Saving result file positions.xtc
06:59:13:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
06:59:13:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:59:13:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14250 run:74 clone:1 gen:23 core:0x21 unit:0x0000001d80fccb0a5d6ed2170ecf9bfa
06:59:13:WU01:FS00:Uploading 66.93MiB to 128.252.203.10
06:59:13:WU01:FS00:Connecting to 128.252.203.10:8080
06:59:15:WU00:FS00:Download 7.32%
06:59:19:WU01:FS00:Upload 13.26%
06:59:21:WU00:FS00:Download 12.82%
06:59:25:WU01:FS00:Upload 25.96%
06:59:27:WU00:FS00:Download 18.31%
06:59:31:WU01:FS00:Upload 38.19%
06:59:35:WU00:FS00:Download 23.80%
06:59:37:WU01:FS00:Upload 51.64%
06:59:41:WU00:FS00:Download 25.63%
06:59:43:WU01:FS00:Upload 63.69%
06:59:48:WU00:FS00:Download 31.13%
06:59:49:WU01:FS00:Upload 76.85%
06:59:55:WU01:FS00:Upload 89.37%
06:59:56:WU00:FS00:Download 34.79%
07:00:01:WU01:FS00:Upload complete
07:00:01:WU01:FS00:Server responded WORK_ACK (400)
07:00:01:WU01:FS00:Final credit estimate, 80693.00 points
07:00:01:WU01:FS00:Cleaning up
07:00:04:WU00:FS00:Download 38.45%
07:00:06:WU02:FS01:0x21:Completed 6125000 out of 12500000 steps (49%)
07:00:10:WU00:FS00:Download 47.60%
07:00:16:WU00:FS00:Download 54.93%
07:00:23:WU00:FS00:Download 64.08%
07:00:29:WU00:FS00:Download 73.24%
07:00:36:WU00:FS00:Download 78.73%
07:00:43:WU00:FS00:Download 80.56%
07:00:49:WU00:FS00:Download 84.22%
07:00:55:WU00:FS00:Download 87.88%
07:03:13:WU02:FS01:0x21:Completed 6250000 out of 12500000 steps (50%)
07:06:21:WU02:FS01:0x21:Completed 6375000 out of 12500000 steps (51%)
07:09:28:WU02:FS01:0x21:Completed 6500000 out of 12500000 steps (52%)
07:12:36:WU02:FS01:0x21:Completed 6625000 out of 12500000 steps (53%)
07:15:42:WU02:FS01:0x21:Completed 6750000 out of 12500000 steps (54%)
07:18:49:WU02:FS01:0x21:Completed 6875000 out of 12500000 steps (55%)
07:21:56:WU02:FS01:0x21:Completed 7000000 out of 12500000 steps (56%)
07:25:03:WU02:FS01:0x21:Completed 7125000 out of 12500000 steps (57%)
07:28:10:WU02:FS01:0x21:Completed 7250000 out of 12500000 steps (58%)
07:31:17:WU02:FS01:0x21:Completed 7375000 out of 12500000 steps (59%)
07:34:24:WU02:FS01:0x21:Completed 7500000 out of 12500000 steps (60%)
07:37:31:WU02:FS01:0x21:Completed 7625000 out of 12500000 steps (61%)
07:40:38:WU02:FS01:0x21:Completed 7750000 out of 12500000 steps (62%)
07:43:45:WU02:FS01:0x21:Completed 7875000 out of 12500000 steps (63%)
07:46:51:WU02:FS01:0x21:Completed 8000000 out of 12500000 steps (64%)
07:49:59:WU02:FS01:0x21:Completed 8125000 out of 12500000 steps (65%)
07:53:05:WU02:FS01:0x21:Completed 8250000 out of 12500000 steps (66%)
07:56:13:WU02:FS01:0x21:Completed 8375000 out of 12500000 steps (67%)
07:59:19:WU02:FS01:0x21:Completed 8500000 out of 12500000 steps (68%)
08:02:26:WU02:FS01:0x21:Completed 8625000 out of 12500000 steps (69%)
08:05:33:WU02:FS01:0x21:Completed 8750000 out of 12500000 steps (70%)
08:08:40:WU02:FS01:0x21:Completed 8875000 out of 12500000 steps (71%)
08:11:47:WU02:FS01:0x21:Completed 9000000 out of 12500000 steps (72%)
08:14:54:WU02:FS01:0x21:Completed 9125000 out of 12500000 steps (73%)
08:18:01:WU02:FS01:0x21:Completed 9250000 out of 12500000 steps (74%)
08:21:08:WU02:FS01:0x21:Completed 9375000 out of 12500000 steps (75%)
08:24:15:WU02:FS01:0x21:Completed 9500000 out of 12500000 steps (76%)
08:27:22:WU02:FS01:0x21:Completed 9625000 out of 12500000 steps (77%)
08:30:28:WU02:FS01:0x21:Completed 9750000 out of 12500000 steps (78%)
08:33:36:WU02:FS01:0x21:Completed 9875000 out of 12500000 steps (79%)
08:36:42:WU02:FS01:0x21:Completed 10000000 out of 12500000 steps (80%)
08:39:50:WU02:FS01:0x21:Completed 10125000 out of 12500000 steps (81%)
08:42:56:WU02:FS01:0x21:Completed 10250000 out of 12500000 steps (82%)
08:46:03:WU02:FS01:0x21:Completed 10375000 out of 12500000 steps (83%)
08:49:10:WU02:FS01:0x21:Completed 10500000 out of 12500000 steps (84%)
08:52:18:WU02:FS01:0x21:Completed 10625000 out of 12500000 steps (85%)
08:55:24:WU02:FS01:0x21:Completed 10750000 out of 12500000 steps (86%)
08:58:31:WU02:FS01:0x21:Completed 10875000 out of 12500000 steps (87%)
09:01:38:WU02:FS01:0x21:Completed 11000000 out of 12500000 steps (88%)
09:04:45:WU02:FS01:0x21:Completed 11125000 out of 12500000 steps (89%)
09:07:52:WU02:FS01:0x21:Completed 11250000 out of 12500000 steps (90%)
09:10:59:WU02:FS01:0x21:Completed 11375000 out of 12500000 steps (91%)
09:14:06:WU02:FS01:0x21:Completed 11500000 out of 12500000 steps (92%)
09:17:13:WU02:FS01:0x21:Completed 11625000 out of 12500000 steps (93%)
09:20:19:WU02:FS01:0x21:Completed 11750000 out of 12500000 steps (94%)
09:23:27:WU02:FS01:0x21:Completed 11875000 out of 12500000 steps (95%)
09:26:33:WU02:FS01:0x21:Completed 12000000 out of 12500000 steps (96%)
09:29:41:WU02:FS01:0x21:Completed 12125000 out of 12500000 steps (97%)
09:32:47:WU02:FS01:0x21:Completed 12250000 out of 12500000 steps (98%)
09:35:55:WU02:FS01:0x21:Completed 12375000 out of 12500000 steps (99%)
09:39:01:WU02:FS01:0x21:Completed 12500000 out of 12500000 steps (100%)
09:39:02:WU01:FS01:Connecting to 65.254.110.245:8080
09:39:02:WU01:FS01:Assigned to work server 155.247.166.220
09:39:02:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
09:39:02:WU01:FS01:Connecting to 155.247.166.220:8080
09:39:02:WU01:FS01:Downloading 3.41MiB
09:39:02:WU02:FS01:0x21:Saving result file logfile_01.txt
09:39:02:WU02:FS01:0x21:Saving result file checkpointState.xml
09:39:02:WU02:FS01:0x21:Saving result file checkpt.crc
09:39:02:WU02:FS01:0x21:Saving result file log.txt
09:39:02:WU02:FS01:0x21:Saving result file positions.xtc
09:39:02:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
09:39:03:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:39:03:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14180 run:9 clone:311 gen:36 core:0x21 unit:0x0000003b0002894c5d3b55bd16035ea7
09:39:03:WU02:FS01:Uploading 14.83MiB to 155.247.166.220
09:39:03:WU02:FS01:Connecting to 155.247.166.220:8080
09:39:09:WU02:FS01:Upload 60.25%
09:39:12:WU01:FS01:Download 3.66%
09:39:14:WU02:FS01:Upload complete
09:39:14:WU02:FS01:Server responded WORK_ACK (400)
09:39:14:WU02:FS01:Final credit estimate, 168597.00 points
09:39:14:WU02:FS01:Cleaning up
09:39:18:WU01:FS01:Download 5.50%
09:39:24:WU01:FS01:Download 7.33%
09:39:32:WU01:FS01:Download 12.82%
09:39:49:WU01:FS01:Download 14.65%
09:40:03:WU01:FS01:Download 18.32%
09:40:10:WU01:FS01:Download 23.81%
09:40:16:WU01:FS01:Download 31.14%
09:40:22:WU01:FS01:Download 36.63%
09:40:29:WU01:FS01:Download 45.79%
09:40:35:WU01:FS01:Download 49.46%
09:40:41:WU01:FS01:Download 54.95%
09:40:47:WU01:FS01:Download 60.45%
09:40:53:WU01:FS01:Download 64.11%
09:42:20:WU01:FS01:Download 67.31%
09:42:20:ERROR:WU01:FS01:Exception: Transfer failed
09:42:21:WU01:FS01:Connecting to 65.254.110.245:8080
09:42:21:WU01:FS01:Assigned to work server 128.252.203.10
09:42:21:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 128.252.203.10
09:42:21:WU01:FS01:Connecting to 128.252.203.10:8080
09:42:22:WU01:FS01:Downloading 69.71MiB
09:42:27:WU01:FS01:Download complete
09:42:27:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14249 run:228 clone:1 gen:1 core:0x21 unit:0x0000000180fccb0a5d716f900886ecde
09:42:27:WU01:FS01:Starting
09:42:27:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 1877 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
09:42:27:WU01:FS01:Started FahCore on PID 7260
09:42:27:WU01:FS01:Core PID:7264
09:42:27:WU01:FS01:FahCore 0x21 started
09:42:27:WU01:FS01:0x21:*********************** Log Started 2019-09-26T09:42:27Z ***********************
09:42:27:WU01:FS01:0x21:Project: 14249 (Run 228, Clone 1, Gen 1)
09:42:27:WU01:FS01:0x21:Unit: 0x0000000180fccb0a5d716f900886ecde
09:42:27:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
09:42:27:WU01:FS01:0x21:Machine: 1
09:42:27:WU01:FS01:0x21:Reading tar file core.xml
09:42:27:WU01:FS01:0x21:Reading tar file integrator.xml
09:42:27:WU01:FS01:0x21:Reading tar file state.xml
09:42:27:WU01:FS01:0x21:Reading tar file system.xml
09:42:27:WU01:FS01:0x21:Digital signatures verified
09:42:27:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
09:42:27:WU01:FS01:0x21:Version 0.0.20
09:42:37:WU01:FS01:0x21:Completed 0 out of 1000000 steps (0%)
09:42:37:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:44:11:WU01:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
09:45:45:WU01:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
09:47:22:WU01:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
12:17:26:WU01:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
12:19:00:WU01:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
12:20:38:WU01:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
12:22:12:WU01:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
12:22:13:WU02:FS01:Connecting to 65.254.110.245:8080
12:22:13:WU02:FS01:Assigned to work server 155.247.166.220
12:22:13:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.220
12:22:13:WU02:FS01:Connecting to 155.247.166.220:8080
12:22:14:WU02:FS01:Downloading 15.58MiB
12:22:16:WU01:FS01:0x21:Saving result file logfile_01.txt
12:22:16:WU01:FS01:0x21:Saving result file checkpointState.xml
12:22:16:WU01:FS01:0x21:Saving result file checkpt.crc
12:22:16:WU01:FS01:0x21:Saving result file log.txt
12:22:16:WU01:FS01:0x21:Saving result file positions.xtc
12:22:16:WU01:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
12:22:16:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:22:16:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:228 clone:1 gen:1 core:0x21 unit:0x0000000180fccb0a5d716f900886ecde
12:22:16:WU01:FS01:Uploading 66.22MiB to 128.252.203.10
12:22:16:WU01:FS01:Connecting to 128.252.203.10:8080
12:22:22:WU01:FS01:Upload 12.08%
12:22:28:WU01:FS01:Upload 25.20%
12:22:30:WU02:FS01:Download 0.40%
12:22:34:WU01:FS01:Upload 38.04%
12:22:40:WU01:FS01:Upload 50.97%
12:22:46:WU01:FS01:Upload 63.99%
12:22:52:WU01:FS01:Upload 76.83%
12:22:58:WU01:FS01:Upload 89.66%
12:23:04:WU01:FS01:Upload complete
12:23:04:WU01:FS01:Server responded WORK_ACK (400)
12:23:04:WU01:FS01:Final credit estimate, 81106.00 points
12:23:04:WU01:FS01:Cleaning up

[Catalina588]

This thread exhibits the same symptoms and problems as the following thread. Treat them as related/same problem. viewtopic.php?f=106&t=31874&p=308979#p308979