Multiple WU's Fail downld/upld to 155.247.166.*

[MeeLee]

Just FYI,
The server drops aren't affecting faster GPUs only.
it's affecting nearly ALL GPUs.
You may see your 2080 is affected, because it is assigned to a specific server address for a WU. However in my case all cards on random are affected, whenever they're assigned to the broken server.

[bruce]

Here's what i THINK I know:

* Something is getting overloaded at temple.edu. (Communications gets very slow)
* Several things have been tried, including rebooting their WS/CS (and possibly the campus router). This drops all connections so all clients can try connecting again (at least until the next overload occurs).
* FAHClient does not recover that connection, nor does it recognize that it needs to drop that connection and restart it. Restarting FAHClient is required to do that.

Obviously you can't predict whether the server based problem has been cleared when you restart your FAHClient. Dumping WUs rarely helps except if enough people do it concurrently to contributes to a reduction in the server overload problem.

All WUs downloaded from any WS must eventually be returned to that WS. If your results happen to go to a CS (Collection Server) it will be timestamped as returned and eventually forwarded to the WS (WorkServer), so that'a good thing. I suspect that communications between the CS and the WS can recover from communications failures more effectively that your local Client.

[MeeLee]

155.247.166.220:8080 is at it again.
Can't they take the server offline, or just assign it to the smaller CPU WUs only?
Even 16MB WUs getting only as far as 17% download before timeout!

[Celso Azevedo]

It started again after almost two weeks running without problems. Yesterday both slots got stuck, today it happened again to one of them.

I have this machine in a shed outdoors, it's getting cold and I rely on the heat to keep the place at acceptable temperatures. This forces me to stop folding.

Code: Select all

19:46:51:WU02:FS00:Connecting to 65.254.110.245:8080
19:46:51:WU02:FS00:Assigned to work server 128.252.203.10
19:46:51:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 128.252.203.10
19:46:51:WU02:FS00:Connecting to 128.252.203.10:8080
19:46:52:ERROR:WU02:FS00:Exception: Server did not assign work unit
19:46:52:WU02:FS00:Connecting to 65.254.110.245:8080
19:46:52:WU02:FS00:Assigned to work server 155.247.166.220
19:46:52:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 155.247.166.220
19:46:53:WU02:FS00:Connecting to 155.247.166.220:8080
19:46:54:WU02:FS00:Downloading 15.61MiB
19:47:06:WU02:FS00:Download 1.20%
19:47:34:WU02:FS00:Download 2.00%
19:47:42:WU02:FS00:Download 2.80%
19:47:49:WU02:FS00:Download 3.20%
19:47:55:WU02:FS00:Download 4.00%
19:48:40:WU02:FS00:Download 4.40%
19:48:46:WU02:FS00:Download 5.20%
19:48:53:WU02:FS00:Download 6.01%
19:48:59:WU02:FS00:Download 6.81%
19:49:09:WU02:FS00:Download 8.01%
19:49:17:WU02:FS00:Download 8.81%
19:49:23:WU02:FS00:Download 9.21%
19:49:29:WU02:FS00:Download 9.61%
19:49:35:WU02:FS00:Download 10.41%
19:49:43:WU02:FS00:Download 11.21%
19:49:57:WU02:FS00:Download 11.61%
19:50:44:WU02:FS00:Download 12.41%
19:50:50:WU02:FS00:Download 13.61%
19:50:59:WU02:FS00:Download 14.81%
19:51:06:WU02:FS00:Download 16.01%
19:51:15:WU02:FS00:Download 17.22%
19:51:26:WU02:FS00:Download 18.02%
19:51:33:WU02:FS00:Download 18.42%
19:51:41:WU02:FS00:Download 18.82%
19:51:48:WU02:FS00:Download 19.22%
19:52:09:WU02:FS00:Download 19.62%
19:52:37:WU02:FS00:Download 20.02%
19:52:46:WU02:FS00:Download 20.42%
19:53:02:WU02:FS00:Download 20.82%
19:53:14:WU02:FS00:Download 21.22%
19:53:26:WU02:FS00:Download 21.62%

[MeeLee]

155.247.166.220:8080

Small Wus make it through. Large WUs don't (unless a rare occasion where barely anyone else is downloading WUs from that server).

We really need a way to get an alarm, and remotely reboot the WUs when this happens.

[JohnJohn]

After having minimal issues the last two days, we're currently again having UL/DL problems with projects involving the Temple servers; the faulty server is pushing over 2 days without a reboot, and we're having predictable/repeated problems. At this point, it is irresponsible of the project owner/admin to not have addressed this issue on any of the available forums and formats.

I will take my GPUs to other projects if this is not addressed by the end of the week. I speak for many donors.

In no way is this a critique of Bruce or the other forum admins; they have done what they can at this point. We respect and value your time as volunteers, but this is completely ridiculous.

[JimF]

JohnJohn,
Thanks for the update. It is too bad we have to rely on other volunteers for the status. There should be some announcement.

[bruce]

The fundamental problem is too many projects running on only two servers (and too few running elsewhere). The solution is to buy more servers, but they're very expensive so bureaucracy (outside of FAH) raises it's ugly head. There is a lot of red tape getting all the required signatures of people who can spend that much money.

[gbowman]

Sorry for the troubles. We're looking into whether there is a network issue and balancing the project load.

[bruce]

When possible, I'm trying to fold new P142xx projects instead of P141xx although it's not always something I can control.

[gordonbb]

Just an update to report that I’ve observed no problems with the servers at Temple over the last two days but Monday night after a brief flurry of stuck slots between 18:00 and 21:00 things improved.

What I’ve observed over the last couple of weeks is the whole gamut of problems reported here.

I’m assigned to a Work Server which doesn’t respond but the client lay gracefully times out and is assigned to another Work Server.

Downloads from either of the two Temple Servers but predominately .220 (80-90%) will on occasion start a download and stall. When this occurs only occasionally will no progress indication be seen more frequently the downloads will start slow with 5-10% per 6-8 second interval the gradually the download rate will decrease until finally all progress indications stop reporting usually after the frequency between reports drops to a minute or so and the rate below a couple of percent per interval.

The download failures seem independent of file size. I have observed 6, 13 and 30MB files fail.

Multiple slots connecting to the same server in the same timeframe will fail.

Rebooting the system will restart the download and it will complete successfully from the Temple servers only occasionally. Usually no WU can be successfully be downloaded until assigned to another Work Server.

The issue will sometimes correct itself with no observed change in the Server uptime on the stats page (I.e. the server was not rebooted)

I have NOT observed Uploads to the same server of the prior WU failing so you see the strange behaviour occasionally of a download to the server failing while an upload to the same server completes and at a much faster rate like the transmit buffers somewhere being full but the receive buffers being OK.

My sympathy to the staff at Temple U who are trying to figure this one out. Intermittent problems that can’t easily be reproduced are the worst.

[bruce]

Thank you for your observations. Narratives like that one often contribute useful information to those who are attempting to correct the problem.

[gbowman]

We're still digging into this issue. More at
https://foldingathome.org/2019/10/25/upload-issues/

[JohnJohn]

Thank you for your feedback, Bruce, Jim, GordonBB, GBowman/Pande lab. It is much appreciated.

[biodoc]

This WU upload fails at about 73% multiple times. No success yet.
linux mint 19.2 Tina
Client version 7.5.1
RTX2070 and RTX2070 Super
Driver Version: 430.50
Ethernet connection to router.

Any suggestion on what I should try to help diagnose the problem?

Code: Select all

*********************** Log Started 2019-10-29T21:26:40Z ***********************
02:33:53:WU01:FS00:Connecting to 65.254.110.245:8080
02:33:53:WU01:FS00:Assigned to work server 128.252.203.10
02:33:53:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TBC [RTX 2070 Super] from 128.252.203.10
02:33:53:WU01:FS00:Connecting to 128.252.203.10:8080
02:33:54:WU01:FS00:Downloading 72.54MiB
02:34:00:WU01:FS00:Download 95.81%
02:34:00:WU01:FS00:Download complete
02:34:00:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14280 run:0 clone:80 gen:23 core:0x21 unit:0x0000001c80fccb0a5d9e115a686914ab
02:34:00:WU01:FS00:Starting
02:34:00:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 6682 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
02:34:00:WU01:FS00:Started FahCore on PID 9130
02:34:00:WU01:FS00:Core PID:9134
02:34:00:WU01:FS00:FahCore 0x21 started
02:34:00:WU01:FS00:0x21:*********************** Log Started 2019-10-30T02:34:00Z ***********************
02:34:00:WU01:FS00:0x21:Project: 14280 (Run 0, Clone 80, Gen 23)
02:34:00:WU01:FS00:0x21:Unit: 0x0000001c80fccb0a5d9e115a686914ab
02:34:00:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
02:34:00:WU01:FS00:0x21:Machine: 0
02:34:00:WU01:FS00:0x21:Reading tar file core.xml
02:34:00:WU01:FS00:0x21:Reading tar file integrator.xml
02:34:00:WU01:FS00:0x21:Reading tar file state.xml
02:34:00:WU01:FS00:0x21:Reading tar file system.xml
02:34:00:WU01:FS00:0x21:Digital signatures verified
02:34:00:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
02:34:00:WU01:FS00:0x21:Version 0.0.20
02:34:17:WU01:FS00:0x21:Completed 0 out of 1000000 steps (0%)
02:34:17:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:35:18:WU01:FS00:0x21:Completed 10000 out of 1000000 steps (1%)
02:36:18:WU01:FS00:0x21:Completed 20000 out of 1000000 steps (2%)
02:38:24:WU01:FS00:0x21:Completed 40000 out of 1000000 steps (4%)
02:39:30:WU01:FS00:0x21:Completed 50000 out of 1000000 steps (5%)
02:40:30:WU01:FS00:0x21:Completed 60000 out of 1000000 steps (6%)
02:41:36:WU01:FS00:0x21:Completed 70000 out of 1000000 steps (7%)
02:42:37:WU01:FS00:0x21:Completed 80000 out of 1000000 steps (8%)
02:43:42:WU01:FS00:0x21:Completed 90000 out of 1000000 steps (9%)
02:44:43:WU01:FS00:0x21:Completed 100000 out of 1000000 steps (10%)
02:45:48:WU01:FS00:0x21:Completed 110000 out of 1000000 steps (11%)
02:46:49:WU01:FS00:0x21:Completed 120000 out of 1000000 steps (12%)
02:47:54:WU01:FS00:0x21:Completed 130000 out of 1000000 steps (13%)
02:48:54:WU01:FS00:0x21:Completed 140000 out of 1000000 steps (14%)
02:50:00:WU01:FS00:0x21:Completed 150000 out of 1000000 steps (15%)
02:51:00:WU01:FS00:0x21:Completed 160000 out of 1000000 steps (16%)
02:52:04:WU01:FS00:0x21:Completed 170000 out of 1000000 steps (17%)
02:53:04:WU01:FS00:0x21:Completed 180000 out of 1000000 steps (18%)
02:54:09:WU01:FS00:0x21:Completed 190000 out of 1000000 steps (19%)
02:55:09:WU01:FS00:0x21:Completed 200000 out of 1000000 steps (20%)
02:56:14:WU01:FS00:0x21:Completed 210000 out of 1000000 steps (21%)
02:57:14:WU01:FS00:0x21:Completed 220000 out of 1000000 steps (22%)
02:58:19:WU01:FS00:0x21:Completed 230000 out of 1000000 steps (23%)
02:59:19:WU01:FS00:0x21:Completed 240000 out of 1000000 steps (24%)
03:00:24:WU01:FS00:0x21:Completed 250000 out of 1000000 steps (25%)
03:01:25:WU01:FS00:0x21:Completed 260000 out of 1000000 steps (26%)
03:02:30:WU01:FS00:0x21:Completed 270000 out of 1000000 steps (27%)
03:03:31:WU01:FS00:0x21:Completed 280000 out of 1000000 steps (28%)
03:04:36:WU01:FS00:0x21:Completed 290000 out of 1000000 steps (29%)
03:05:36:WU01:FS00:0x21:Completed 300000 out of 1000000 steps (30%)
03:06:42:WU01:FS00:0x21:Completed 310000 out of 1000000 steps (31%)
03:07:42:WU01:FS00:0x21:Completed 320000 out of 1000000 steps (32%)
03:08:48:WU01:FS00:0x21:Completed 330000 out of 1000000 steps (33%)
03:09:48:WU01:FS00:0x21:Completed 340000 out of 1000000 steps (34%)
03:10:54:WU01:FS00:0x21:Completed 350000 out of 1000000 steps (35%)
03:11:54:WU01:FS00:0x21:Completed 360000 out of 1000000 steps (36%)
03:13:00:WU01:FS00:0x21:Completed 370000 out of 1000000 steps (37%)
03:14:00:WU01:FS00:0x21:Completed 380000 out of 1000000 steps (38%)
03:15:05:WU01:FS00:0x21:Completed 390000 out of 1000000 steps (39%)
03:16:05:WU01:FS00:0x21:Completed 400000 out of 1000000 steps (40%)
03:17:10:WU01:FS00:0x21:Completed 410000 out of 1000000 steps (41%)
03:18:11:WU01:FS00:0x21:Completed 420000 out of 1000000 steps (42%)
03:19:15:WU01:FS00:0x21:Completed 430000 out of 1000000 steps (43%)
03:20:16:WU01:FS00:0x21:Completed 440000 out of 1000000 steps (44%)
03:21:21:WU01:FS00:0x21:Completed 450000 out of 1000000 steps (45%)
03:22:21:WU01:FS00:0x21:Completed 460000 out of 1000000 steps (46%)
03:23:26:WU01:FS00:0x21:Completed 470000 out of 1000000 steps (47%)
03:24:27:WU01:FS00:0x21:Completed 480000 out of 1000000 steps (48%)
03:25:32:WU01:FS00:0x21:Completed 490000 out of 1000000 steps (49%)
03:26:33:WU01:FS00:0x21:Completed 500000 out of 1000000 steps (50%)
******************************* Date: 2019-10-30 *******************************
03:27:38:WU01:FS00:0x21:Completed 510000 out of 1000000 steps (51%)
03:28:39:WU01:FS00:0x21:Completed 520000 out of 1000000 steps (52%)
03:29:44:WU01:FS00:0x21:Completed 530000 out of 1000000 steps (53%)
03:30:44:WU01:FS00:0x21:Completed 540000 out of 1000000 steps (54%)
03:31:49:WU01:FS00:0x21:Completed 550000 out of 1000000 steps (55%)
03:32:50:WU01:FS00:0x21:Completed 560000 out of 1000000 steps (56%)
03:33:55:WU01:FS00:0x21:Completed 570000 out of 1000000 steps (57%)
03:34:56:WU01:FS00:0x21:Completed 580000 out of 1000000 steps (58%)
03:36:01:WU01:FS00:0x21:Completed 590000 out of 1000000 steps (59%)
03:37:01:WU01:FS00:0x21:Completed 600000 out of 1000000 steps (60%)
03:38:06:WU01:FS00:0x21:Completed 610000 out of 1000000 steps (61%)
03:39:07:WU01:FS00:0x21:Completed 620000 out of 1000000 steps (62%)
03:40:11:WU01:FS00:0x21:Completed 630000 out of 1000000 steps (63%)
03:41:11:WU01:FS00:0x21:Completed 640000 out of 1000000 steps (64%)
03:42:16:WU01:FS00:0x21:Completed 650000 out of 1000000 steps (65%)
03:43:17:WU01:FS00:0x21:Completed 660000 out of 1000000 steps (66%)
03:44:21:WU01:FS00:0x21:Completed 670000 out of 1000000 steps (67%)
03:45:22:WU01:FS00:0x21:Completed 680000 out of 1000000 steps (68%)
03:46:27:WU01:FS00:0x21:Completed 690000 out of 1000000 steps (69%)
03:47:28:WU01:FS00:0x21:Completed 700000 out of 1000000 steps (70%)
03:48:32:WU01:FS00:0x21:Completed 710000 out of 1000000 steps (71%)
03:49:32:WU01:FS00:0x21:Completed 720000 out of 1000000 steps (72%)
03:50:37:WU01:FS00:0x21:Completed 730000 out of 1000000 steps (73%)
03:51:38:WU01:FS00:0x21:Completed 740000 out of 1000000 steps (74%)
03:52:43:WU01:FS00:0x21:Completed 750000 out of 1000000 steps (75%)
03:53:44:WU01:FS00:0x21:Completed 760000 out of 1000000 steps (76%)
03:54:49:WU01:FS00:0x21:Completed 770000 out of 1000000 steps (77%)
03:55:50:WU01:FS00:0x21:Completed 780000 out of 1000000 steps (78%)
03:56:55:WU01:FS00:0x21:Completed 790000 out of 1000000 steps (79%)
03:57:55:WU01:FS00:0x21:Completed 800000 out of 1000000 steps (80%)
03:59:01:WU01:FS00:0x21:Completed 810000 out of 1000000 steps (81%)
04:00:01:WU01:FS00:0x21:Completed 820000 out of 1000000 steps (82%)
04:01:07:WU01:FS00:0x21:Completed 830000 out of 1000000 steps (83%)
04:02:07:WU01:FS00:0x21:Completed 840000 out of 1000000 steps (84%)
04:03:12:WU01:FS00:0x21:Completed 850000 out of 1000000 steps (85%)
04:04:13:WU01:FS00:0x21:Completed 860000 out of 1000000 steps (86%)
04:05:18:WU01:FS00:0x21:Completed 870000 out of 1000000 steps (87%)
04:06:19:WU01:FS00:0x21:Completed 880000 out of 1000000 steps (88%)
04:07:24:WU01:FS00:0x21:Completed 890000 out of 1000000 steps (89%)
04:08:25:WU01:FS00:0x21:Completed 900000 out of 1000000 steps (90%)
04:09:30:WU01:FS00:0x21:Completed 910000 out of 1000000 steps (91%)
04:10:31:WU01:FS00:0x21:Completed 920000 out of 1000000 steps (92%)
04:11:36:WU01:FS00:0x21:Completed 930000 out of 1000000 steps (93%)
04:12:37:WU01:FS00:0x21:Completed 940000 out of 1000000 steps (94%)
04:13:42:WU01:FS00:0x21:Completed 950000 out of 1000000 steps (95%)
04:14:42:WU01:FS00:0x21:Completed 960000 out of 1000000 steps (96%)
04:15:47:WU01:FS00:0x21:Completed 970000 out of 1000000 steps (97%)
04:16:47:WU01:FS00:0x21:Completed 980000 out of 1000000 steps (98%)
04:17:52:WU01:FS00:0x21:Completed 990000 out of 1000000 steps (99%)
04:18:52:WU01:FS00:0x21:Completed 1000000 out of 1000000 steps (100%)
04:18:57:WU01:FS00:0x21:Saving result file logfile_01.txt
04:18:57:WU01:FS00:0x21:Saving result file checkpointState.xml
04:18:57:WU01:FS00:0x21:Saving result file checkpt.crc
04:18:57:WU01:FS00:0x21:Saving result file log.txt
04:18:57:WU01:FS00:0x21:Saving result file positions.xtc
04:18:57:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
04:18:58:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:18:58:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14280 run:0 clone:80 gen:23 core:0x21 unit:0x0000001c80fccb0a5d9e115a686914ab
04:18:58:WU01:FS00:Uploading 68.58MiB to 128.252.203.10
04:18:58:WU01:FS00:Connecting to 128.252.203.10:8080
04:19:03:WU01:FS00:Upload complete
04:19:04:WU01:FS00:Server responded WORK_ACK (400)
04:19:04:WU01:FS00:Final credit estimate, 106617.00 points
04:19:04:WU01:FS00:Cleaning up
07:23:39:WU01:FS00:Connecting to 65.254.110.245:8080
07:23:39:WU01:FS00:Assigned to work server 155.247.166.220
07:23:39:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TBC [RTX 2070 Super] from 155.247.166.220
07:23:39:WU01:FS00:Connecting to 155.247.166.220:8080
07:23:39:WU01:FS00:Downloading 2.34MiB
07:23:40:WU01:FS00:Download complete
07:23:40:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14184 run:11 clone:99 gen:45 core:0x21 unit:0x000000640002894c5d38948eec207f7e
07:23:44:WU01:FS00:Starting
07:23:44:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 6682 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
07:23:44:WU01:FS00:Started FahCore on PID 13932
07:23:44:WU01:FS00:Core PID:13936
07:23:44:WU01:FS00:FahCore 0x21 started
07:23:45:WU01:FS00:0x21:*********************** Log Started 2019-10-30T07:23:44Z ***********************
07:23:45:WU01:FS00:0x21:Project: 14184 (Run 11, Clone 99, Gen 45)
07:23:45:WU01:FS00:0x21:Unit: 0x000000640002894c5d38948eec207f7e
07:23:45:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
07:23:45:WU01:FS00:0x21:Machine: 0
07:23:45:WU01:FS00:0x21:Reading tar file core.xml
07:23:45:WU01:FS00:0x21:Reading tar file integrator.xml
07:23:45:WU01:FS00:0x21:Reading tar file state.xml
07:23:45:WU01:FS00:0x21:Reading tar file system.xml
07:23:45:WU01:FS00:0x21:Digital signatures verified
07:23:45:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
07:23:45:WU01:FS00:0x21:Version 0.0.20
07:23:46:WU01:FS00:0x21:Completed 0 out of 12500000 steps (0%)
07:23:46:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
07:24:22:WU01:FS00:0x21:Completed 125000 out of 12500000 steps (1%)
07:24:58:WU01:FS00:0x21:Completed 250000 out of 12500000 steps (2%)
07:25:34:WU01:FS00:0x21:Completed 375000 out of 12500000 steps (3%)
07:26:10:WU01:FS00:0x21:Completed 500000 out of 12500000 steps (4%)
07:26:46:WU01:FS00:0x21:Completed 625000 out of 12500000 steps (5%)
07:27:22:WU01:FS00:0x21:Completed 750000 out of 12500000 steps (6%)
07:27:58:WU01:FS00:0x21:Completed 875000 out of 12500000 steps (7%)
07:28:34:WU01:FS00:0x21:Completed 1000000 out of 12500000 steps (8%)
07:29:10:WU01:FS00:0x21:Completed 1125000 out of 12500000 steps (9%)
07:29:46:WU01:FS00:0x21:Completed 1250000 out of 12500000 steps (10%)
07:30:22:WU01:FS00:0x21:Completed 1375000 out of 12500000 steps (11%)
07:30:58:WU01:FS00:0x21:Completed 1500000 out of 12500000 steps (12%)
07:31:34:WU01:FS00:0x21:Completed 1625000 out of 12500000 steps (13%)
07:32:10:WU01:FS00:0x21:Completed 1750000 out of 12500000 steps (14%)
07:32:46:WU01:FS00:0x21:Completed 1875000 out of 12500000 steps (15%)
07:33:22:WU01:FS00:0x21:Completed 2000000 out of 12500000 steps (16%)
07:33:58:WU01:FS00:0x21:Completed 2125000 out of 12500000 steps (17%)
07:34:34:WU01:FS00:0x21:Completed 2250000 out of 12500000 steps (18%)
07:35:10:WU01:FS00:0x21:Completed 2375000 out of 12500000 steps (19%)
07:35:46:WU01:FS00:0x21:Completed 2500000 out of 12500000 steps (20%)
07:36:22:WU01:FS00:0x21:Completed 2625000 out of 12500000 steps (21%)
07:36:58:WU01:FS00:0x21:Completed 2750000 out of 12500000 steps (22%)
07:37:34:WU01:FS00:0x21:Completed 2875000 out of 12500000 steps (23%)
07:38:10:WU01:FS00:0x21:Completed 3000000 out of 12500000 steps (24%)
07:38:46:WU01:FS00:0x21:Completed 3125000 out of 12500000 steps (25%)
07:39:22:WU01:FS00:0x21:Completed 3250000 out of 12500000 steps (26%)
07:39:58:WU01:FS00:0x21:Completed 3375000 out of 12500000 steps (27%)
07:40:34:WU01:FS00:0x21:Completed 3500000 out of 12500000 steps (28%)
07:41:10:WU01:FS00:0x21:Completed 3625000 out of 12500000 steps (29%)
07:41:46:WU01:FS00:0x21:Completed 3750000 out of 12500000 steps (30%)
07:42:22:WU01:FS00:0x21:Completed 3875000 out of 12500000 steps (31%)
07:42:58:WU01:FS00:0x21:Completed 4000000 out of 12500000 steps (32%)
07:43:34:WU01:FS00:0x21:Completed 4125000 out of 12500000 steps (33%)
07:44:10:WU01:FS00:0x21:Completed 4250000 out of 12500000 steps (34%)
07:44:46:WU01:FS00:0x21:Completed 4375000 out of 12500000 steps (35%)
07:45:22:WU01:FS00:0x21:Completed 4500000 out of 12500000 steps (36%)
07:45:58:WU01:FS00:0x21:Completed 4625000 out of 12500000 steps (37%)
07:46:34:WU01:FS00:0x21:Completed 4750000 out of 12500000 steps (38%)
07:47:10:WU01:FS00:0x21:Completed 4875000 out of 12500000 steps (39%)
07:47:46:WU01:FS00:0x21:Completed 5000000 out of 12500000 steps (40%)
07:48:22:WU01:FS00:0x21:Completed 5125000 out of 12500000 steps (41%)
07:48:58:WU01:FS00:0x21:Completed 5250000 out of 12500000 steps (42%)
07:49:34:WU01:FS00:0x21:Completed 5375000 out of 12500000 steps (43%)
07:50:10:WU01:FS00:0x21:Completed 5500000 out of 12500000 steps (44%)
07:50:46:WU01:FS00:0x21:Completed 5625000 out of 12500000 steps (45%)
07:51:22:WU01:FS00:0x21:Completed 5750000 out of 12500000 steps (46%)
07:51:58:WU01:FS00:0x21:Completed 5875000 out of 12500000 steps (47%)
07:52:34:WU01:FS00:0x21:Completed 6000000 out of 12500000 steps (48%)
07:53:10:WU01:FS00:0x21:Completed 6125000 out of 12500000 steps (49%)
07:53:46:WU01:FS00:0x21:Completed 6250000 out of 12500000 steps (50%)
07:54:22:WU01:FS00:0x21:Completed 6375000 out of 12500000 steps (51%)
07:54:58:WU01:FS00:0x21:Completed 6500000 out of 12500000 steps (52%)
07:55:34:WU01:FS00:0x21:Completed 6625000 out of 12500000 steps (53%)
07:56:09:WU01:FS00:0x21:Completed 6750000 out of 12500000 steps (54%)
07:56:46:WU01:FS00:0x21:Completed 6875000 out of 12500000 steps (55%)
07:57:21:WU01:FS00:0x21:Completed 7000000 out of 12500000 steps (56%)
07:57:58:WU01:FS00:0x21:Completed 7125000 out of 12500000 steps (57%)
07:58:34:WU01:FS00:0x21:Completed 7250000 out of 12500000 steps (58%)
07:59:10:WU01:FS00:0x21:Completed 7375000 out of 12500000 steps (59%)
07:59:46:WU01:FS00:0x21:Completed 7500000 out of 12500000 steps (60%)
08:00:21:WU01:FS00:0x21:Completed 7625000 out of 12500000 steps (61%)
08:00:57:WU01:FS00:0x21:Completed 7750000 out of 12500000 steps (62%)
08:01:33:WU01:FS00:0x21:Completed 7875000 out of 12500000 steps (63%)
08:02:09:WU01:FS00:0x21:Completed 8000000 out of 12500000 steps (64%)
08:02:45:WU01:FS00:0x21:Completed 8125000 out of 12500000 steps (65%)
08:03:21:WU01:FS00:0x21:Completed 8250000 out of 12500000 steps (66%)
08:03:57:WU01:FS00:0x21:Completed 8375000 out of 12500000 steps (67%)
08:04:33:WU01:FS00:0x21:Completed 8500000 out of 12500000 steps (68%)
08:05:09:WU01:FS00:0x21:Completed 8625000 out of 12500000 steps (69%)
08:05:45:WU01:FS00:0x21:Completed 8750000 out of 12500000 steps (70%)
08:06:21:WU01:FS00:0x21:Completed 8875000 out of 12500000 steps (71%)
08:06:57:WU01:FS00:0x21:Completed 9000000 out of 12500000 steps (72%)
08:07:33:WU01:FS00:0x21:Completed 9125000 out of 12500000 steps (73%)
08:08:09:WU01:FS00:0x21:Completed 9250000 out of 12500000 steps (74%)
08:08:45:WU01:FS00:0x21:Completed 9375000 out of 12500000 steps (75%)
08:09:21:WU01:FS00:0x21:Completed 9500000 out of 12500000 steps (76%)
08:09:57:WU01:FS00:0x21:Completed 9625000 out of 12500000 steps (77%)
08:10:33:WU01:FS00:0x21:Completed 9750000 out of 12500000 steps (78%)
08:11:09:WU01:FS00:0x21:Completed 9875000 out of 12500000 steps (79%)
08:11:45:WU01:FS00:0x21:Completed 10000000 out of 12500000 steps (80%)
08:12:21:WU01:FS00:0x21:Completed 10125000 out of 12500000 steps (81%)
08:12:57:WU01:FS00:0x21:Completed 10250000 out of 12500000 steps (82%)
08:13:33:WU01:FS00:0x21:Completed 10375000 out of 12500000 steps (83%)
08:14:08:WU01:FS00:0x21:Completed 10500000 out of 12500000 steps (84%)
08:14:44:WU01:FS00:0x21:Completed 10625000 out of 12500000 steps (85%)
08:15:21:WU01:FS00:0x21:Completed 10750000 out of 12500000 steps (86%)
08:15:57:WU01:FS00:0x21:Completed 10875000 out of 12500000 steps (87%)
08:16:33:WU01:FS00:0x21:Completed 11000000 out of 12500000 steps (88%)
08:17:09:WU01:FS00:0x21:Completed 11125000 out of 12500000 steps (89%)
08:17:45:WU01:FS00:0x21:Completed 11250000 out of 12500000 steps (90%)
08:18:21:WU01:FS00:0x21:Completed 11375000 out of 12500000 steps (91%)
08:18:57:WU01:FS00:0x21:Completed 11500000 out of 12500000 steps (92%)
08:19:33:WU01:FS00:0x21:Completed 11625000 out of 12500000 steps (93%)
08:20:09:WU01:FS00:0x21:Completed 11750000 out of 12500000 steps (94%)
08:20:45:WU01:FS00:0x21:Completed 11875000 out of 12500000 steps (95%)
08:21:21:WU01:FS00:0x21:Completed 12000000 out of 12500000 steps (96%)
08:21:57:WU01:FS00:0x21:Completed 12125000 out of 12500000 steps (97%)
08:22:33:WU01:FS00:0x21:Completed 12250000 out of 12500000 steps (98%)
08:23:09:WU01:FS00:0x21:Completed 12375000 out of 12500000 steps (99%)
08:23:45:WU01:FS00:0x21:Completed 12500000 out of 12500000 steps (100%)
08:23:45:WU01:FS00:0x21:Saving result file logfile_01.txt
08:23:45:WU01:FS00:0x21:Saving result file checkpointState.xml
08:23:45:WU01:FS00:0x21:Saving result file checkpt.crc
08:23:45:WU01:FS00:0x21:Saving result file log.txt
08:23:45:WU01:FS00:0x21:Saving result file positions.xtc
08:23:45:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
08:23:45:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:23:45:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14184 run:11 clone:99 gen:45 core:0x21 unit:0x000000640002894c5d38948eec207f7e
08:23:45:WU01:FS00:Uploading 2.44MiB to 155.247.166.220
08:23:45:WU01:FS00:Connecting to 155.247.166.220:8080
08:23:46:WU01:FS00:Upload complete
08:23:46:WU01:FS00:Server responded WORK_ACK (400)
08:23:46:WU01:FS00:Final credit estimate, 48319.00 points
08:23:46:WU01:FS00:Cleaning up
******************************* Date: 2019-10-30 *******************************
11:53:26:WU01:FS00:Connecting to 65.254.110.245:8080
11:53:26:WU01:FS00:Assigned to work server 128.252.203.10
11:53:26:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TBC [RTX 2070 Super] from 128.252.203.10
11:53:26:WU01:FS00:Connecting to 128.252.203.10:8080
11:53:26:ERROR:WU01:FS00:Exception: Server did not assign work unit
11:53:26:WU01:FS00:Connecting to 65.254.110.245:8080
11:53:27:WU01:FS00:Assigned to work server 128.252.203.10
11:53:27:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TBC [RTX 2070 Super] from 128.252.203.10
11:53:27:WU01:FS00:Connecting to 128.252.203.10:8080
11:53:27:WU01:FS00:Downloading 72.92MiB
11:53:33:WU01:FS00:Download 85.03%
11:53:33:WU01:FS00:Download complete
11:53:33:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
11:53:33:WU01:FS00:Starting
11:53:33:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 705 -lifeline 6682 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
11:53:33:WU01:FS00:Started FahCore on PID 17002
11:53:33:WU01:FS00:Core PID:17006
11:53:33:WU01:FS00:FahCore 0x21 started
11:53:34:WU01:FS00:0x21:*********************** Log Started 2019-10-30T11:53:33Z ***********************
11:53:34:WU01:FS00:0x21:Project: 14288 (Run 0, Clone 65, Gen 22)
11:53:34:WU01:FS00:0x21:Unit: 0x0000001780fccb0a5da629b6dcc14d16
11:53:34:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
11:53:34:WU01:FS00:0x21:Machine: 0
11:53:34:WU01:FS00:0x21:Reading tar file core.xml
11:53:34:WU01:FS00:0x21:Reading tar file integrator.xml
11:53:34:WU01:FS00:0x21:Reading tar file state.xml
11:53:34:WU01:FS00:0x21:Reading tar file system.xml
11:53:34:WU01:FS00:0x21:Digital signatures verified
11:53:34:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
11:53:34:WU01:FS00:0x21:Version 0.0.20
11:53:47:WU01:FS00:0x21:Completed 0 out of 1000000 steps (0%)
11:53:47:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:54:48:WU01:FS00:0x21:Completed 10000 out of 1000000 steps (1%)
11:55:48:WU01:FS00:0x21:Completed 20000 out of 1000000 steps (2%)
11:56:53:WU01:FS00:0x21:Completed 30000 out of 1000000 steps (3%)
11:57:54:WU01:FS00:0x21:Completed 40000 out of 1000000 steps (4%)
11:59:00:WU01:FS00:0x21:Completed 50000 out of 1000000 steps (5%)
12:00:00:WU01:FS00:0x21:Completed 60000 out of 1000000 steps (6%)
12:01:06:WU01:FS00:0x21:Completed 70000 out of 1000000 steps (7%)
12:02:07:WU01:FS00:0x21:Completed 80000 out of 1000000 steps (8%)
12:03:12:WU01:FS00:0x21:Completed 90000 out of 1000000 steps (9%)
12:04:13:WU01:FS00:0x21:Completed 100000 out of 1000000 steps (10%)
12:05:18:WU01:FS00:0x21:Completed 110000 out of 1000000 steps (11%)
12:06:19:WU01:FS00:0x21:Completed 120000 out of 1000000 steps (12%)
12:07:24:WU01:FS00:0x21:Completed 130000 out of 1000000 steps (13%)
12:08:25:WU01:FS00:0x21:Completed 140000 out of 1000000 steps (14%)
12:09:30:WU01:FS00:0x21:Completed 150000 out of 1000000 steps (15%)
12:10:31:WU01:FS00:0x21:Completed 160000 out of 1000000 steps (16%)
12:11:36:WU01:FS00:0x21:Completed 170000 out of 1000000 steps (17%)
12:12:36:WU01:FS00:0x21:Completed 180000 out of 1000000 steps (18%)
12:13:41:WU01:FS00:0x21:Completed 190000 out of 1000000 steps (19%)
12:14:41:WU01:FS00:0x21:Completed 200000 out of 1000000 steps (20%)
12:15:46:WU01:FS00:0x21:Completed 210000 out of 1000000 steps (21%)
12:16:47:WU01:FS00:0x21:Completed 220000 out of 1000000 steps (22%)
12:17:51:WU01:FS00:0x21:Completed 230000 out of 1000000 steps (23%)
12:18:51:WU01:FS00:0x21:Completed 240000 out of 1000000 steps (24%)
12:19:57:WU01:FS00:0x21:Completed 250000 out of 1000000 steps (25%)
12:20:57:WU01:FS00:0x21:Completed 260000 out of 1000000 steps (26%)
12:22:02:WU01:FS00:0x21:Completed 270000 out of 1000000 steps (27%)
12:23:03:WU01:FS00:0x21:Completed 280000 out of 1000000 steps (28%)
12:24:08:WU01:FS00:0x21:Completed 290000 out of 1000000 steps (29%)
12:25:09:WU01:FS00:0x21:Completed 300000 out of 1000000 steps (30%)
12:26:14:WU01:FS00:0x21:Completed 310000 out of 1000000 steps (31%)
12:27:15:WU01:FS00:0x21:Completed 320000 out of 1000000 steps (32%)
12:28:21:WU01:FS00:0x21:Completed 330000 out of 1000000 steps (33%)
12:29:21:WU01:FS00:0x21:Completed 340000 out of 1000000 steps (34%)
12:30:27:WU01:FS00:0x21:Completed 350000 out of 1000000 steps (35%)
12:31:27:WU01:FS00:0x21:Completed 360000 out of 1000000 steps (36%)
12:32:33:WU01:FS00:0x21:Completed 370000 out of 1000000 steps (37%)
12:33:34:WU01:FS00:0x21:Completed 380000 out of 1000000 steps (38%)
12:34:39:WU01:FS00:0x21:Completed 390000 out of 1000000 steps (39%)
12:35:39:WU01:FS00:0x21:Completed 400000 out of 1000000 steps (40%)
12:36:45:WU01:FS00:0x21:Completed 410000 out of 1000000 steps (41%)
12:37:45:WU01:FS00:0x21:Completed 420000 out of 1000000 steps (42%)
12:38:50:WU01:FS00:0x21:Completed 430000 out of 1000000 steps (43%)
12:39:51:WU01:FS00:0x21:Completed 440000 out of 1000000 steps (44%)
12:40:56:WU01:FS00:0x21:Completed 450000 out of 1000000 steps (45%)
12:41:56:WU01:FS00:0x21:Completed 460000 out of 1000000 steps (46%)
12:43:02:WU01:FS00:0x21:Completed 470000 out of 1000000 steps (47%)
12:44:03:WU01:FS00:0x21:Completed 480000 out of 1000000 steps (48%)
12:45:07:WU01:FS00:0x21:Completed 490000 out of 1000000 steps (49%)
12:46:08:WU01:FS00:0x21:Completed 500000 out of 1000000 steps (50%)
12:47:13:WU01:FS00:0x21:Completed 510000 out of 1000000 steps (51%)
12:48:13:WU01:FS00:0x21:Completed 520000 out of 1000000 steps (52%)
12:49:18:WU01:FS00:0x21:Completed 530000 out of 1000000 steps (53%)
12:50:18:WU01:FS00:0x21:Completed 540000 out of 1000000 steps (54%)
12:51:23:WU01:FS00:0x21:Completed 550000 out of 1000000 steps (55%)
12:52:23:WU01:FS00:0x21:Completed 560000 out of 1000000 steps (56%)
12:53:27:WU01:FS00:0x21:Completed 570000 out of 1000000 steps (57%)
12:54:28:WU01:FS00:0x21:Completed 580000 out of 1000000 steps (58%)
12:55:33:WU01:FS00:0x21:Completed 590000 out of 1000000 steps (59%)
12:56:33:WU01:FS00:0x21:Completed 600000 out of 1000000 steps (60%)
12:57:37:WU01:FS00:0x21:Completed 610000 out of 1000000 steps (61%)
12:58:38:WU01:FS00:0x21:Completed 620000 out of 1000000 steps (62%)
12:59:43:WU01:FS00:0x21:Completed 630000 out of 1000000 steps (63%)
13:00:43:WU01:FS00:0x21:Completed 640000 out of 1000000 steps (64%)
13:01:48:WU01:FS00:0x21:Completed 650000 out of 1000000 steps (65%)
13:02:49:WU01:FS00:0x21:Completed 660000 out of 1000000 steps (66%)
13:03:54:WU01:FS00:0x21:Completed 670000 out of 1000000 steps (67%)
13:04:55:WU01:FS00:0x21:Completed 680000 out of 1000000 steps (68%)
13:06:00:WU01:FS00:0x21:Completed 690000 out of 1000000 steps (69%)
13:07:01:WU01:FS00:0x21:Completed 700000 out of 1000000 steps (70%)
13:08:06:WU01:FS00:0x21:Completed 710000 out of 1000000 steps (71%)
13:09:06:WU01:FS00:0x21:Completed 720000 out of 1000000 steps (72%)
13:10:12:WU01:FS00:0x21:Completed 730000 out of 1000000 steps (73%)
13:11:12:WU01:FS00:0x21:Completed 740000 out of 1000000 steps (74%)
13:12:17:WU01:FS00:0x21:Completed 750000 out of 1000000 steps (75%)
13:13:17:WU01:FS00:0x21:Completed 760000 out of 1000000 steps (76%)
13:14:22:WU01:FS00:0x21:Completed 770000 out of 1000000 steps (77%)
13:15:22:WU01:FS00:0x21:Completed 780000 out of 1000000 steps (78%)
13:16:27:WU01:FS00:0x21:Completed 790000 out of 1000000 steps (79%)
13:17:28:WU01:FS00:0x21:Completed 800000 out of 1000000 steps (80%)
13:18:33:WU01:FS00:0x21:Completed 810000 out of 1000000 steps (81%)
13:19:34:WU01:FS00:0x21:Completed 820000 out of 1000000 steps (82%)
13:20:39:WU01:FS00:0x21:Completed 830000 out of 1000000 steps (83%)
13:21:40:WU01:FS00:0x21:Completed 840000 out of 1000000 steps (84%)
13:22:45:WU01:FS00:0x21:Completed 850000 out of 1000000 steps (85%)
13:23:46:WU01:FS00:0x21:Completed 860000 out of 1000000 steps (86%)
13:24:51:WU01:FS00:0x21:Completed 870000 out of 1000000 steps (87%)
13:25:52:WU01:FS00:0x21:Completed 880000 out of 1000000 steps (88%)
13:26:57:WU01:FS00:0x21:Completed 890000 out of 1000000 steps (89%)
13:27:58:WU01:FS00:0x21:Completed 900000 out of 1000000 steps (90%)
13:29:03:WU01:FS00:0x21:Completed 910000 out of 1000000 steps (91%)
13:30:04:WU01:FS00:0x21:Completed 920000 out of 1000000 steps (92%)
13:31:09:WU01:FS00:0x21:Completed 930000 out of 1000000 steps (93%)
13:32:10:WU01:FS00:0x21:Completed 940000 out of 1000000 steps (94%)
13:33:15:WU01:FS00:0x21:Completed 950000 out of 1000000 steps (95%)
13:34:16:WU01:FS00:0x21:Completed 960000 out of 1000000 steps (96%)
13:35:21:WU01:FS00:0x21:Completed 970000 out of 1000000 steps (97%)
13:36:22:WU01:FS00:0x21:Completed 980000 out of 1000000 steps (98%)
13:37:27:WU01:FS00:0x21:Completed 990000 out of 1000000 steps (99%)
13:38:28:WU01:FS00:0x21:Completed 1000000 out of 1000000 steps (100%)
13:38:33:WU01:FS00:0x21:Saving result file logfile_01.txt
13:38:33:WU01:FS00:0x21:Saving result file checkpointState.xml
13:38:33:WU01:FS00:0x21:Saving result file checkpt.crc
13:38:33:WU01:FS00:0x21:Saving result file log.txt
13:38:33:WU01:FS00:0x21:Saving result file positions.xtc
13:38:33:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
13:38:33:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:38:33:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
13:38:33:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
13:38:33:WU01:FS00:Connecting to 128.252.203.10:8080
13:55:16:WU01:FS00:Upload 72.59%
13:55:16:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
13:55:16:WU01:FS00:Trying to send results to collection server
13:55:16:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
13:55:16:WU01:FS00:Connecting to 155.247.166.220:8080
13:55:16:ERROR:WU01:FS00:Exception: Transfer failed
13:55:16:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
13:55:16:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
13:55:16:WU01:FS00:Connecting to 128.252.203.10:8080
14:11:59:WU01:FS00:Upload 74.49%
14:11:59:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
14:11:59:WU01:FS00:Trying to send results to collection server
14:11:59:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
14:11:59:WU01:FS00:Connecting to 155.247.166.220:8080
14:11:59:ERROR:WU01:FS00:Exception: Transfer failed
14:12:00:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
14:12:00:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
14:12:00:WU01:FS00:Connecting to 128.252.203.10:8080
14:28:38:WU01:FS00:Upload 74.04%
14:28:39:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
14:28:39:WU01:FS00:Trying to send results to collection server
14:28:39:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
14:28:39:WU01:FS00:Connecting to 155.247.166.220:8080
14:28:39:ERROR:WU01:FS00:Exception: Transfer failed
14:28:39:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
14:28:39:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
14:28:39:WU01:FS00:Connecting to 128.252.203.10:8080
14:45:18:WU01:FS00:Upload 72.95%
14:45:18:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
14:45:18:WU01:FS00:Trying to send results to collection server
14:45:18:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
14:45:18:WU01:FS00:Connecting to 155.247.166.220:8080
14:45:18:ERROR:WU01:FS00:Exception: Transfer failed
14:45:18:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
14:45:18:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
14:45:18:WU01:FS00:Connecting to 128.252.203.10:8080
15:02:01:WU01:FS00:Upload 74.13%
15:02:01:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
15:02:01:WU01:FS00:Trying to send results to collection server
15:02:01:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
15:02:01:WU01:FS00:Connecting to 155.247.166.220:8080
15:02:02:ERROR:WU01:FS00:Exception: Transfer failed
15:02:02:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
15:02:02:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
15:02:02:WU01:FS00:Connecting to 128.252.203.10:8080
15:18:41:WU01:FS00:Upload 72.77%
15:18:41:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
15:18:41:WU01:FS00:Trying to send results to collection server
15:18:41:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
15:18:41:WU01:FS00:Connecting to 155.247.166.220:8080
15:18:41:ERROR:WU01:FS00:Exception: Transfer failed
15:18:41:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
15:18:41:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
15:18:41:WU01:FS00:Connecting to 128.252.203.10:8080
******************************* Date: 2019-10-30 *******************************
15:35:24:WU01:FS00:Upload 73.13%
15:35:24:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
15:35:24:WU01:FS00:Trying to send results to collection server
15:35:24:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
15:35:24:WU01:FS00:Connecting to 155.247.166.220:8080
15:35:25:ERROR:WU01:FS00:Exception: Transfer failed
15:35:25:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
15:35:25:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
15:35:25:WU01:FS00:Connecting to 128.252.203.10:8080
15:52:08:WU01:FS00:Upload 74.04%
15:52:08:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
15:52:08:WU01:FS00:Trying to send results to collection server
15:52:08:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
15:52:08:WU01:FS00:Connecting to 155.247.166.220:8080
15:52:08:ERROR:WU01:FS00:Exception: Transfer failed
15:53:22:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
15:53:22:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
15:53:22:WU01:FS00:Connecting to 128.252.203.10:8080
16:10:07:WU01:FS00:Upload 73.04%
16:10:07:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
16:10:07:WU01:FS00:Trying to send results to collection server
16:10:07:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
16:10:07:WU01:FS00:Connecting to 155.247.166.220:8080
16:10:08:ERROR:WU01:FS00:Exception: Transfer failed
16:22:24:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14288 run:0 clone:65 gen:22 core:0x21 unit:0x0000001780fccb0a5da629b6dcc14d16
16:22:24:WU01:FS00:Uploading 68.97MiB to 128.252.203.10
16:22:24:WU01:FS00:Connecting to 128.252.203.10:8080
16:39:33:WU01:FS00:Upload 73.50%
16:39:33:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
16:39:33:WU01:FS00:Trying to send results to collection server
16:39:33:WU01:FS00:Uploading 68.97MiB to 155.247.166.220
16:39:33:WU01:FS00:Connecting to 155.247.166.220:8080
16:39:33:ERROR:WU01:FS00:Exception: Transfer failed