Work unit upload failures

[aetch]

Starting around the 18th March I have been experiencing multiple upload transfer failures from my systems.
The uploads do eventually go but sometimes bounce multiple times between work server and collection server before they go.

Since the 18th I have logged 54 failures with about half of them (26 off) on the 128.252.203.10-14 server range alone.
Another server that has given me a number of failures is 206.223.170.146 (7 off).

Normally I wouldn't worry about the occasional upload failure but this is quite an escalation.

Could this be an issue with overloaded servers or collateral from current world events?

[toTOW]

I didn't notice any major issues ... but some temporary server overloads or downtimes might happen ...

Are you still seeing issues ?

[aetch]

Yes.
Since the original post I've had another 16 failures, 6 off to 128.252.203.11 alone. The rest of the new failures were spread among 7 other servers.

TBH, most are single failures where the work server fails the upload so a collection servers picks it up instead.
The ones that bother me are the double/triple failures where the work server fails bouncing it to the collection server, the collection server then fails bouncing it back to the work server.

My most recent examples of double failures

Code: Select all

******************************* Date: 2022-03-27 *******************************
08:02:51:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18201 run:14037 clone:1 gen:26 core:0x22 unit:0x000000010000001a00004719000036d5
08:02:51:WU01:FS01:Uploading 27.50MiB to 128.252.203.11
08:02:51:WU01:FS01:Connecting to 128.252.203.11:8080
08:03:07:WU01:FS01:Upload 0.68%
08:03:21:WU01:FS01:Upload 0.91%
08:03:30:WU01:FS01:Upload 1.14%
08:03:49:WU01:FS01:Upload 1.36%
08:03:55:WU01:FS01:Upload 1.59%
08:04:14:WU01:FS01:Upload 1.82%
08:04:37:WU01:FS01:Upload 2.05%
08:05:19:WU01:FS01:Upload 2.27%
08:05:19:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
08:05:19:WU01:FS01:Trying to send results to collection server
08:05:19:WU01:FS01:Uploading 27.50MiB to 128.252.203.14
08:05:19:WU01:FS01:Connecting to 128.252.203.14:8080
08:05:39:WU01:FS01:Upload 0.68%
08:05:47:WU01:FS01:Upload 1.14%
08:06:05:WU01:FS01:Upload 1.36%
08:06:31:WU01:FS01:Upload 1.59%
08:06:49:WU01:FS01:Upload 1.82%
08:06:58:WU01:FS01:Upload 2.05%
08:07:14:WU01:FS01:Upload 2.27%
08:07:32:WU01:FS01:Upload 2.50%
08:07:48:WU01:FS01:Upload 2.73%
08:08:01:WU01:FS01:Upload 2.95%
08:08:12:WU01:FS01:Upload 3.18%
08:08:21:WU01:FS01:Upload 3.41%
08:08:29:WU01:FS01:Upload 3.64%
08:08:38:WU01:FS01:Upload 3.86%
08:08:44:WU01:FS01:Upload 4.09%
08:08:57:WU01:FS01:Upload 4.32%
08:09:04:WU01:FS01:Upload 4.55%
08:09:50:WU01:FS01:Upload 4.77%
08:10:56:WU01:FS01:Upload 5.00%
08:10:56:ERROR:WU01:FS01:Exception: Transfer failed
08:10:56:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18201 run:14037 clone:1 gen:26 core:0x22 unit:0x000000010000001a00004719000036d5
08:10:56:WU01:FS01:Uploading 27.50MiB to 128.252.203.11
08:10:56:WU01:FS01:Connecting to 128.252.203.11:8080
08:11:13:WU01:FS01:Upload 0.68%
08:11:28:WU01:FS01:Upload 0.91%
08:11:34:WU01:FS01:Upload 5.23%
08:11:40:WU01:FS01:Upload 15.23%
08:11:46:WU01:FS01:Upload 25.46%
08:11:52:WU01:FS01:Upload 35.46%
08:11:58:WU01:FS01:Upload 45.69%
08:12:04:WU01:FS01:Upload 55.69%
08:12:10:WU01:FS01:Upload 65.92%
08:12:16:WU01:FS01:Upload 75.92%
08:12:22:WU01:FS01:Upload 86.15%
08:12:28:WU01:FS01:Upload 96.15%
08:12:32:WU01:FS01:Upload complete
08:12:32:WU01:FS01:Server responded WORK_ACK (400)
08:12:32:WU01:FS01:Final credit estimate, 366752.00 points

Code: Select all

******************************* Date: 2022-03-26 *******************************
05:06:59:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18408 run:58 clone:1 gen:38 core:0xa8 unit:0x0000000100000026000047e80000003a
05:06:59:WU01:FS00:Uploading 24.74MiB to 129.32.209.203
05:06:59:WU01:FS00:Connecting to 129.32.209.203:8080
05:07:23:WU01:FS00:Upload 0.51%
05:07:53:WU01:FS00:Upload 0.76%
05:07:53:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
05:07:53:WU01:FS00:Trying to send results to collection server
05:07:53:WU01:FS00:Uploading 24.74MiB to 129.32.209.206
05:07:53:WU01:FS00:Connecting to 129.32.209.206:8080
05:07:59:WU01:FS00:Upload 12.88%
05:08:05:WU01:FS00:Upload 26.52%
05:08:11:WU01:FS00:Upload 39.91%
05:08:18:WU01:FS00:Upload 45.98%
05:08:28:WU01:FS00:Upload 46.23%
05:08:45:WU01:FS00:Upload 46.48%
05:08:57:WU01:FS00:Upload 46.73%
05:09:12:WU01:FS00:Upload 46.99%
05:09:24:WU01:FS00:Upload 47.24%
05:10:18:WU01:FS00:Upload 48.00%
05:10:18:ERROR:WU01:FS00:Exception: Transfer failed
05:10:18:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:18408 run:58 clone:1 gen:38 core:0xa8 unit:0x0000000100000026000047e80000003a
05:10:18:WU01:FS00:Uploading 24.74MiB to 129.32.209.203
05:10:18:WU01:FS00:Connecting to 129.32.209.203:8080
05:10:36:WU01:FS00:Upload 0.51%
05:10:56:WU01:FS00:Upload 0.76%
05:11:09:WU01:FS00:Upload 1.01%
05:11:15:WU01:FS00:Upload 3.28%
05:11:21:WU01:FS00:Upload 6.06%
05:11:38:WU01:FS00:Upload 6.82%
05:11:44:WU01:FS00:Upload 8.84%
05:11:50:WU01:FS00:Upload 11.62%
05:11:56:WU01:FS00:Upload 14.15%
05:12:02:WU01:FS00:Upload 17.18%
05:12:08:WU01:FS00:Upload 20.21%
05:12:14:WU01:FS00:Upload 22.99%
05:12:20:WU01:FS00:Upload 26.27%
05:12:26:WU01:FS00:Upload 29.05%
05:12:32:WU01:FS00:Upload 32.84%
05:12:38:WU01:FS00:Upload 36.88%
05:12:44:WU01:FS00:Upload 40.92%
05:12:50:WU01:FS00:Upload 44.97%
05:12:56:WU01:FS00:Upload 50.27%
05:13:02:WU01:FS00:Upload 54.31%
05:13:08:WU01:FS00:Upload 59.11%
05:13:14:WU01:FS00:Upload 67.20%
05:13:22:WU01:FS00:Upload 75.53%
05:13:30:WU01:FS00:Upload 75.78%
05:13:45:WU01:FS00:Upload 76.04%
05:13:56:WU01:FS00:Upload 76.29%
05:14:09:WU01:FS00:Upload 76.54%
05:14:21:WU01:FS00:Upload 76.79%
05:14:33:WU01:FS00:Upload 77.05%
05:14:46:WU01:FS00:Upload 77.30%
05:14:52:WU01:FS00:Upload 82.86%
05:14:58:WU01:FS00:Upload 96.75%
05:14:59:WU01:FS00:Upload complete
05:14:59:WU01:FS00:Server responded WORK_ACK (400)
05:14:59:WU01:FS00:Final credit estimate, 431503.00 points

Code: Select all

******************************* Date: 2022-03-25 *******************************
15:34:22:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18213 run:2652 clone:0 gen:5 core:0x22 unit:0x00000000000000050000472500000a5c
15:34:22:WU00:FS01:Uploading 27.50MiB to 206.223.170.146
15:34:22:WU00:FS01:Connecting to 206.223.170.146:8080
15:34:35:WU00:FS01:Upload 0.68%
15:34:46:WU00:FS01:Upload 0.91%
15:34:58:WU00:FS01:Upload 1.14%
15:35:10:WU00:FS01:Upload 1.36%
15:35:23:WU00:FS01:Upload 1.59%
15:35:33:WU00:FS01:Upload 1.82%
15:35:44:WU00:FS01:Upload 2.05%
15:37:10:WU00:FS01:Upload 2.27%
15:40:58:WU00:FS01:Upload 2.50%
15:40:59:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
15:40:59:WU00:FS01:Trying to send results to collection server
15:40:59:WU00:FS01:Uploading 27.50MiB to 128.252.203.14
15:40:59:WU00:FS01:Connecting to 128.252.203.14:8080
15:41:05:WU00:FS01:Upload 0.45%
15:41:28:WU00:FS01:Upload 0.68%
15:41:58:WU00:FS01:Upload 0.91%
15:41:58:ERROR:WU00:FS01:Exception: Transfer failed
15:41:58:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18213 run:2652 clone:0 gen:5 core:0x22 unit:0x00000000000000050000472500000a5c
15:41:58:WU00:FS01:Uploading 27.50MiB to 206.223.170.146
15:41:58:WU00:FS01:Connecting to 206.223.170.146:8080
15:42:17:WU00:FS01:Upload 0.45%
15:42:36:WU00:FS01:Upload 0.68%
15:42:48:WU00:FS01:Upload 0.91%
15:43:03:WU00:FS01:Upload 1.14%
15:43:15:WU00:FS01:Upload 1.36%
15:43:28:WU00:FS01:Upload 1.59%
15:43:39:WU00:FS01:Upload 1.82%
15:43:53:WU00:FS01:Upload 2.05%
15:44:06:WU00:FS01:Upload 2.27%
15:44:19:WU00:FS01:Upload 2.50%
15:44:30:WU00:FS01:Upload 2.73%
15:44:44:WU00:FS01:Upload 2.95%
15:44:57:WU00:FS01:Upload 3.18%
15:45:12:WU00:FS01:Upload 3.41%
15:45:27:WU00:FS01:Upload 3.64%
15:45:43:WU00:FS01:Upload 3.86%
15:45:55:WU00:FS01:Upload 4.09%
15:46:10:WU00:FS01:Upload 4.32%
15:46:26:WU00:FS01:Upload 4.55%
15:46:40:WU00:FS01:Upload 4.77%
15:46:54:WU00:FS01:Upload 5.00%
15:47:08:WU00:FS01:Upload 5.23%
15:47:22:WU00:FS01:Upload 5.46%
15:47:37:WU00:FS01:Upload 5.68%
15:47:52:WU00:FS01:Upload 5.91%
15:48:07:WU00:FS01:Upload 6.14%
15:48:20:WU00:FS01:Upload 6.36%
15:48:34:WU00:FS01:Upload 6.59%
15:48:49:WU00:FS01:Upload 6.82%
15:49:03:WU00:FS01:Upload 7.05%
15:49:18:WU00:FS01:Upload 7.27%
15:49:28:WU00:FS01:Upload 7.50%
15:49:34:WU00:FS01:Upload 13.87%
15:49:40:WU00:FS01:Upload 25.23%
15:49:46:WU00:FS01:Upload 37.28%
15:49:52:WU00:FS01:Upload 49.10%
15:49:58:WU00:FS01:Upload 60.92%
15:50:04:WU00:FS01:Upload 72.96%
15:50:10:WU00:FS01:Upload 84.78%
15:50:16:WU00:FS01:Upload 96.83%
15:50:19:WU00:FS01:Upload complete
15:50:19:WU00:FS01:Server responded WORK_ACK (400)
15:50:19:WU00:FS01:Final credit estimate, 372266.00 points

An example of a triple failure

Code: Select all

******************************* Date: 2022-03-24 *******************************
17:58:51:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18201 run:4179 clone:1 gen:26 core:0x22 unit:0x000000010000001a0000471900001053
17:58:51:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
17:58:51:WU02:FS01:Connecting to 128.252.203.11:8080
17:59:25:WU02:FS01:Upload 0.45%
18:01:01:WU02:FS01:Upload 0.68%
18:01:14:WU02:FS01:Upload 1.14%
18:01:24:WU02:FS01:Upload 1.36%
18:01:39:WU02:FS01:Upload 1.59%
18:01:49:WU02:FS01:Upload 1.82%
18:02:11:WU02:FS01:Upload 2.05%
18:03:20:WU02:FS01:Upload 2.27%
18:03:56:WU02:FS01:Upload 2.50%
18:03:57:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
18:03:57:WU02:FS01:Trying to send results to collection server
18:03:57:WU02:FS01:Uploading 27.50MiB to 128.252.203.14
18:03:57:WU02:FS01:Connecting to 128.252.203.14:8080
18:04:03:WU02:FS01:Upload 3.41%
18:04:09:WU02:FS01:Upload 13.41%
18:04:15:WU02:FS01:Upload 23.63%
18:04:21:WU02:FS01:Upload 33.40%
18:04:27:WU02:FS01:Upload 42.04%
18:04:38:WU02:FS01:Upload 42.27%
18:06:02:WU02:FS01:Upload 42.49%
18:07:51:WU02:FS01:Upload 42.72%
18:08:21:WU02:FS01:Upload 42.95%
18:08:21:ERROR:WU02:FS01:Exception: Transfer failed
18:08:21:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18201 run:4179 clone:1 gen:26 core:0x22 unit:0x000000010000001a0000471900001053
18:08:21:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
18:08:21:WU02:FS01:Connecting to 128.252.203.11:8080
18:08:33:WU02:FS01:Upload 0.45%
18:08:42:WU02:FS01:Upload 0.68%
18:08:50:WU02:FS01:Upload 0.91%
18:09:50:WU02:FS01:Upload 1.14%
18:11:25:WU02:FS01:Upload 1.36%
18:11:25:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
18:11:25:WU02:FS01:Trying to send results to collection server
18:11:25:WU02:FS01:Uploading 27.50MiB to 128.252.203.14
18:11:25:WU02:FS01:Connecting to 128.252.203.14:8080
18:11:36:WU02:FS01:Upload 0.45%
18:11:55:WU02:FS01:Upload 0.68%
18:12:09:WU02:FS01:Upload 0.91%
18:12:24:WU02:FS01:Upload 1.14%
18:12:39:WU02:FS01:Upload 1.36%
18:12:55:WU02:FS01:Upload 1.59%
18:13:10:WU02:FS01:Upload 1.82%
18:13:16:WU02:FS01:Upload 3.18%
18:13:22:WU02:FS01:Upload 12.73%
18:13:28:WU02:FS01:Upload 22.95%
18:13:34:WU02:FS01:Upload 32.95%
18:13:40:WU02:FS01:Upload 43.17%
18:13:46:WU02:FS01:Upload 53.40%
18:13:52:WU02:FS01:Upload 63.40%
18:13:58:WU02:FS01:Upload 73.62%
18:14:04:WU02:FS01:Upload 83.85%
18:14:10:WU02:FS01:Upload 93.85%
18:14:15:WU02:FS01:Upload complete
18:14:15:WU02:FS01:Server responded WORK_ACK (400)
18:14:15:WU02:FS01:Final credit estimate, 361038.00 points

Like I said before, I don't know if this is a server issue or a current world events issue.