Send Errors - 155.247.164.213 & .214

[InDebtBigTime]

Same here, 3 days multiple WU's completed. All have failed to upload.

[davidcoton]

Hey guys, there is no point in commenting about upload failures without identifying the projects and servers concerned, preferably by posting a full log. This topic is about two particular servers, please start a different topic for different servers.
You can get some server information here.

[mimora]

23:46:42:WU02:Sending unit results: id:02 state:SEND error:NO_ERROR project:14371 run:657 clone:4 gen:7 core:0xa7 unit:0x000000089bf7a4d55e84afec3b8a3403
23:46:42:WU02:Uploading 8.25MiB to 155.247.164.213
23:46:42:WU02:Connecting to 155.247.164.213:8080
23:47:16:WU02:Upload 1.51%
23:47:25:WU02:Upload 3.03%
23:53:39:WU02:Upload 3.79%
23:53:39:WARNING:WU02:Exception: Failed to send results to work server: Transfer failed
23:53:39:WU02:Trying to send results to collection server
23:53:39:WU02:Uploading 8.25MiB to 155.247.166.219
23:53:39:WU02:Connecting to 155.247.166.219:8080
23:53:46:WU02:Upload 1.51%
23:57:52:WU02:Upload 2.27%
23:57:52:ERROR:WU02:Exception: Transfer failed

Project: 14371
Attempts: 13
Assigned: 2020-04-04T22:25:09Z
Timeout: 2020-04-06T22:25:09Z
Expiration: 2020-04-09T17:37:09Z
Work Server: 155.247.164.213
Collection Server: 155.247.166.219

[PantherX]

Welcome to the F@H Forum mimora,

It seems that the CS (155.247.166.219) was restated(?) about 15 minutes ago so hopefully, it will start receiving WUs.

[Eagle]

Yet, this doesn't answer both questions I've raised just half a page ago...

[PantherX]

First and foremost, the science part: what's the actual official word about dropped work units due to not being submitted in time due to server problems? We're losing valuable scientific data or we're getting a chance of submitting eventually? I mean, we're talking COVID-19 here, guys - the more WUs we solve, the sooner we get a cure!...

Completed WUs can be returned to the Servers before the Expiration deadline. Once a WU (completed or otherwise) has reached the expiration, it will be deleted.
Completed WUs submitted before the timeout period will not be reassigned and science can progress.
Completed WUs submitted between the timeout period and expiration time will be on the re-assign list but since can progress the minute a completed WU is received from any donor.

...Secondly, what's the actual official word about dropped (bonus) points for our machine(s)'s hard work which they tried to submit and either partly or even completely failed to do so due to those server problems? Do they get those points or is it all lost in the middle of nowhere?

Historically, points lost due to technical issues we just that, lost. However, they were pretty rare and we generally a fraction of WUs. AFAIK, any completed WU successfully received by the server will be credited sooner or later. If you view this graph, you will notice a massive spikes in the number of credits being entered in the database: https://apps.foldingathome.org/credit-log
WUs completed and successfully uploaded before the timeout period will bet bonus points
WUs completed and successfully uploaded after the timeout period but before the expiration will get base points.
WUs expired will not receive any points.
A completed WU that never reaches a server will not be credited.

All the above is simply based on observation and unless there's an official statement that says otherwise, the above is most likely what's going to happen.

I am aware that the F@H Team is working very hard to increase the F@H infrastructure to meet the enormous demand and they are making progress as fast as humanly possible during this global pandemic situation.

[Eagle]

Thanks for the clarification, PantherX. Regarding (bonus) points, I'll sum it up this way: any lost (bonus) points are probably simply that - lost, i. e. no restore, no compensation later.
That's said, but then again we're in this all together and if FAH does its part for finding a cure, I'm still happy.

[TheWolf]

I'm having the same issue with these two servers.

Code: Select all

04:06:09:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:06:09:WU01:FS00:Connecting to 155.247.164.213:8080
04:06:31:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
04:06:31:WU01:FS00:Connecting to 155.247.164.213:80
04:07:01:WU01:FS00:Upload 0.75%
04:07:01:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:07:01:WU01:FS00:Trying to send results to collection server
04:07:01:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:07:01:WU01:FS00:Connecting to 155.247.166.219:8080
04:07:20:WU01:FS00:Upload 0.75%
04:07:20:ERROR:WU01:FS00:Exception: Transfer failed

[PantherX]

Given that it is able to upload data, I would think that it is a bandwidth issue/computing limitation. Thus, while the Server is functional, there's a massive load on it. Give it a few hours and I am sure it will be able to process your WU.

[TheWolf]

Well its going on the 2nd day for 1 WU and now there is a 2nd hung as well.

[TheWolf]

Code: Select all

16:49:11:WU00:FS00:0xa7:Calling: mdrun -s frame22.tpr -o frame22.trr -x frame22.xtc -e frame22.edr -cpi state.cpt -cpt 3 -nt 4
16:49:11:WU00:FS00:0xa7:Steps: first=11000000 total=500000
16:49:14:WU00:FS00:0xa7:Completed 356722 out of 500000 steps (71%)
16:49:31:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
16:49:31:WU01:FS00:Connecting to 155.247.164.213:80
16:49:34:WU02:FS00:Upload 0.75%
16:49:34:ERROR:WU02:FS00:Exception: Transfer failed
16:49:34:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
16:49:34:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
16:49:34:WU02:FS00:Connecting to 155.247.164.213:8080
16:49:52:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 155.247.164.213:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
16:49:52:WU01:FS00:Trying to send results to collection server
16:49:52:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
16:49:52:WU01:FS00:Connecting to 155.247.166.219:8080
16:49:55:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
16:49:55:WU02:FS00:Connecting to 155.247.164.213:80
16:50:16:WU01:FS00:Upload 0.75%
16:50:16:ERROR:WU01:FS00:Exception: Transfer failed
16:50:18:WU02:FS00:Upload 0.75%
16:50:18:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
16:50:18:WU02:FS00:Trying to send results to collection server
16:50:18:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
16:50:18:WU02:FS00:Connecting to 155.247.166.219:8080
16:50:42:WU02:FS00:Upload 0.75%
16:50:42:ERROR:WU02:FS00:Exception: Transfer failed
16:50:47:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
16:50:47:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
16:50:47:WU01:FS00:Connecting to 155.247.164.213:8080
16:51:12:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
16:51:12:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
16:51:12:WU02:FS00:Connecting to 155.247.164.213:8080
16:51:44:WU02:FS00:Upload 0.75%
16:51:44:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
16:51:44:WU02:FS00:Trying to send results to collection server
16:51:44:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
16:51:44:WU02:FS00:Connecting to 155.247.166.219:8080
16:52:03:WU02:FS00:Upload 0.75%
16:52:03:ERROR:WU02:FS00:Exception: Transfer failed

[TheWolf]

[I still have both WU's still stuck in a loop trying to upload. Lucky my new WU's hve been from a different work server and those have had no problems returning/uploading.

Code: Select all

20:23:11:WU03:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
20:26:14:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
20:26:14:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
20:26:14:WU02:FS00:Connecting to 155.247.164.213:8080
20:26:35:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
20:26:35:WU02:FS00:Connecting to 155.247.164.213:80
20:26:44:WU03:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
20:26:56:WARNING:WU02:FS00:Exception: Failed to send results to work server: Failed to connect to 155.247.164.213:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
20:26:56:WU02:FS00:Trying to send results to collection server
20:26:56:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
20:26:56:WU02:FS00:Connecting to 155.247.166.219:8080
20:27:17:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
20:27:17:WU02:FS00:Connecting to 155.247.166.219:80
20:27:38:ERROR:WU02:FS00:Exception: Failed to connect to 155.247.166.219:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
20:30:18:WU03:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
20:33:22:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
20:33:22:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
20:33:22:WU01:FS00:Connecting to 155.247.164.213:8080
20:33:43:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
20:33:43:WU01:FS00:Connecting to 155.247.164.213:80
20:33:52:WU03:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
20:34:03:WU01:FS00:Upload 0.75%
20:34:03:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
20:34:03:WU01:FS00:Trying to send results to collection server
20:34:03:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
20:34:03:WU01:FS00:Connecting to 155.247.166.219:8080
20:34:23:WU01:FS00:Upload 0.75%
20:34:23:ERROR:WU01:FS00:Exception: Transfer failed
20:37:28:WU03:FS00:0xa7:Completed 75000 out of 500000 steps (15%)

Mod Edit: Fixed Code Tags - PantherX

[TheWolf]

Another 8 hours has passed and neither of the WU has uploaded..This is a unacceptable long time frame to have for returns of such important work and will have to be reworked by someone else if they are not excepted at the servers soon. My next option is deleting the slot an is starting to look like the only way to get these messages off my screen, a last resort.

Code: Select all

 04:15:23:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
04:15:23:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
04:15:23:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:15:23:WU02:FS00:Connecting to 155.247.164.213:8080
04:15:23:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:15:23:WU01:FS00:Connecting to 155.247.164.213:8080
04:15:52:WU01:FS00:Upload 0.75%
04:15:52:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:15:52:WU01:FS00:Trying to send results to collection server
04:15:52:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:15:52:WU01:FS00:Connecting to 155.247.166.219:8080
04:16:00:WU02:FS00:Upload 0.75%
04:16:00:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
04:16:00:WU02:FS00:Trying to send results to collection server
04:16:00:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
04:16:00:WU02:FS00:Connecting to 155.247.166.219:8080
04:16:11:WU01:FS00:Upload 0.75%
04:16:11:ERROR:WU01:FS00:Exception: Transfer failed
04:16:12:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:16:12:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:16:12:WU01:FS00:Connecting to 155.247.164.213:8080
04:16:33:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
04:16:33:WU01:FS00:Connecting to 155.247.164.213:80
04:16:54:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 155.247.164.213:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
04:16:54:WU01:FS00:Trying to send results to collection server
04:16:54:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:16:54:WU01:FS00:Connecting to 155.247.166.219:8080
04:17:00:WU01:FS00:Upload 0.75%
04:17:48:WU02:FS00:Upload 0.75%
04:17:48:ERROR:WU02:FS00:Exception: Transfer failed
04:17:48:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
04:17:48:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
04:17:48:WU02:FS00:Connecting to 155.247.164.213:8080
04:18:08:WU02:FS00:Upload 0.75%
04:18:08:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
04:18:08:WU02:FS00:Trying to send results to collection server
04:18:08:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
04:18:08:WU02:FS00:Connecting to 155.247.166.219:8080
04:18:43:WU02:FS00:Upload 0.75%
04:18:44:WU01:FS00:Upload 1.50%
04:18:44:ERROR:WU01:FS00:Exception: Transfer failed
04:18:44:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:18:44:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:18:44:WU01:FS00:Connecting to 155.247.164.213:8080
04:20:20:WU01:FS00:Upload 0.75%
04:20:20:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:20:20:WU01:FS00:Trying to send results to collection server
04:20:20:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:20:20:WU01:FS00:Connecting to 155.247.166.219:8080
04:20:40:WU01:FS00:Upload 0.75%
04:20:41:ERROR:WU01:FS00:Exception: Transfer failed
04:20:41:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:20:41:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:20:41:WU01:FS00:Connecting to 155.247.164.213:8080
04:21:19:WU01:FS00:Upload 0.75%
04:21:19:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:21:19:WU01:FS00:Trying to send results to collection server
04:21:19:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:21:19:WU01:FS00:Connecting to 155.247.166.219:8080
04:22:05:WU02:FS00:Upload 1.50%
04:22:06:ERROR:WU02:FS00:Exception: Transfer failed
04:22:06:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
04:22:06:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
04:22:06:WU02:FS00:Connecting to 155.247.164.213:8080
04:22:27:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
04:22:27:WU02:FS00:Connecting to 155.247.164.213:80
04:22:47:WU01:FS00:Upload 0.75%
04:22:47:ERROR:WU01:FS00:Exception: Transfer failed
04:22:49:WU02:FS00:Upload 0.75%
04:22:49:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
04:22:49:WU02:FS00:Trying to send results to collection server
04:22:49:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
04:22:49:WU02:FS00:Connecting to 155.247.166.219:8080
04:23:18:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:23:18:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:23:18:WU01:FS00:Connecting to 155.247.164.213:8080
04:24:17:WU02:FS00:Upload 0.75%
04:24:17:ERROR:WU02:FS00:Exception: Transfer failed
04:24:17:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
04:24:17:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
04:24:17:WU02:FS00:Connecting to 155.247.164.213:8080
04:24:38:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
04:24:38:WU02:FS00:Connecting to 155.247.164.213:80
04:25:29:WU01:FS00:Upload 0.75%
04:25:30:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:25:30:WU01:FS00:Trying to send results to collection server
04:25:30:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:25:30:WU01:FS00:Connecting to 155.247.166.219:8080
04:26:03:WU01:FS00:Upload 0.75%
04:26:03:ERROR:WU01:FS00:Exception: Transfer failed
04:27:32:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14643 run:403 clone:1 gen:0 core:0xa7 unit:0x000000009bf7a4d55e8b9f634ceb52d4
04:27:32:WU01:FS00:Uploading 8.33MiB to 155.247.164.213
04:27:32:WU01:FS00:Connecting to 155.247.164.213:8080
04:27:40:WU02:FS00:Upload 0.75%
04:27:40:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
04:27:40:WU02:FS00:Trying to send results to collection server
04:27:40:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
04:27:40:WU02:FS00:Connecting to 155.247.166.219:8080
04:27:53:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
04:27:53:WU01:FS00:Connecting to 155.247.164.213:80
04:28:05:WU02:FS00:Upload 0.75%
04:28:05:ERROR:WU02:FS00:Exception: Transfer failed
04:28:06:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14638 run:272 clone:1 gen:3 core:0xa7 unit:0x000000039bf7a4d55e8b1f06003b13dc
04:28:06:WU02:FS00:Uploading 8.31MiB to 155.247.164.213
04:28:06:WU02:FS00:Connecting to 155.247.164.213:8080
04:28:34:WU01:FS00:Upload 0.75%
04:28:34:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:28:34:WU01:FS00:Trying to send results to collection server
04:28:34:WU01:FS00:Uploading 8.33MiB to 155.247.166.219
04:28:34:WU01:FS00:Connecting to 155.247.166.219:8080
04:28:37:WU02:FS00:Upload 0.75%
04:28:37:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
04:28:37:WU02:FS00:Trying to send results to collection server
04:28:37:WU02:FS00:Uploading 8.31MiB to 155.247.166.219
04:28:37:WU02:FS00:Connecting to 155.247.166.219:8080
04:29:02:WU02:FS00:Upload 0.75%
04:29:02:ERROR:WU02:FS00:Exception: Transfer failed
04:29:50:WU01:FS00:Upload 0.75%
04:29:50:ERROR:WU01:FS00:Exception: Transfer failed

[astronomyrat]

Same problem here processed WU (2) are now waiting for more than 2 days for the oldest one to be uploaded, the PC is powered on 24h/day so it's not a question of bad time frame.
So please act on this server or tell us how we can configure our client software to avoid WU coming for 155.247.167.214/155.247166.220 as the upload server seems to be totally overloaded and assigning more WU than it can get back.

[Neil-B]

Both servers appear up https://apps.foldingathome.org/serverstats and have a fair bit of space free, they are both in full assign (so both assigning and accepting) … It does look simply like a bandwith issue on the server - though I am sure someone will check.

If you are having problems returning this it is perfectly feasible that the WU reassignment at Timeout to someone else may be delayed by the same issues, and they may well have to queue as you are doing … Honestly at this point the best thing for the science is to just let your client keep trying to send the WU - if it does eventually reach the Expiration Deadline it will regrettably be dumped by your client so there should be no need to take specific action for this yourself … The team have been working really hard to get all the old/new servers up to speed but there are still areas being worked on - Please bear with them, as they will be at least as frustrated as you are (if not very much more) as this will be impacting their projects and they will be trying as hard as they can to get everything working as it should.