128.252.203.11 is not giving work units

[scott@bjorn3d]

This server has been causing grief for the last few days. Can someone check on it?

[jjmiller]

Hi Scott, Thanks for the report. Last week 128.252.203.11 had very high load over a few days. We then ran into a storage allocation issue causing issues with returning points to folders through the weekend and well into Monday. As of Tuesday ~noon CST we thought we had things ironed out but have recently seen some reports of WUs being dumped. Could you provide a few more details on what you're experiencing so we can get a full sense of the problem? Thanks!

[scott@bjorn3d]

It is telling me it is the work server but never receiving jobs from it.

[Defkon1]

Same problem here:

Code: Select all

08:51:01:WU01:FS02:Connecting to assign1.foldingathome.org:80
08:51:02:WU01:FS02:Assigned to work server 128.252.203.11
08:51:02:WU01:FS02:Requesting new work unit for slot 02: gpu:1:0 [...] from 128.252.203.11
08:51:02:WU01:FS02:Connecting to 128.252.203.11:8080
08:51:30:ERROR:Receive error: 10054: Connection reset by peer.
08:51:34:ERROR:WU01:FS02:Exception: 10002: Received short response, expected 512 bytes, got 0

other slots are working fine... there's a way to force to bypass this server and get a WU from a different one?

[gunnarre]

It's one of only two servers with GPU jobs available at the moment, so bypassing it would not solve the problem.

It is telling me it is the work server but never receiving jobs from it.

Could you please post the relevant parts of your log?

[mgetz]

Code: Select all

10:36:37:WU00:FS01:Connecting to assign1.foldingathome.org:80
10:36:38:WU00:FS01:Assigned to work server 128.252.203.11
10:36:38:WU00:FS01:Requesting new work unit for slot 01: gpu:10:0 TU104 [GeForce RTX 2080 Rev. A] 10068 from 128.252.203.11
10:36:38:WU00:FS01:Connecting to 128.252.203.11:8080
10:37:08:ERROR:WU00:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
10:37:08:WU00:FS01:Connecting to assign1.foldingathome.org:80
10:37:09:WU00:FS01:Assigned to work server 128.252.203.11
10:37:09:WU00:FS01:Requesting new work unit for slot 01: gpu:10:0 TU104 [GeForce RTX 2080 Rev. A] 10068 from 128.252.203.11
10:37:09:WU00:FS01:Connecting to 128.252.203.11:8080
10:37:39:ERROR:WU00:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0

Yep seeing the same.... it appears overloaded

[aetch]

We all have those "Received short response" warnings in our logs.
I'd seen them many times and rarely read much into them beyond "we're too busy just now".
Sometimes I'll use the lack of assignments as a trigger to do some maintenance on my computer(s). We don't always get to choose those moments.
On a personal level I'd much rather have a server fail to assign me a task than to fail/refuse to accept my completed task.

[jjmiller]

Hi all,

Thanks for the notes on this. As gunnarre highlights, the bulk of GPU jobs available on FAH are currently housed on this server, meaning that a lot of the FAH GPU folders are currently bearing down on this server. To manage this load, we have set a relatively strict assignment cap on this server to try and prevent it from becoming overloaded. We've also added several collection servers to make sure that WUs make it back to us and aren't denied as was the case ~2 weeks ago.

I'll poke around and see if there's another WS that we can move some of these GPU jobs unto to expedite jobs going out.

UPDATE- it looks like we may actually be limited with our I/O from the university. We have dialed back our assignment rate on our WUSTL servers-hopefully this clears up all of the issues with receiving jobs.

Thanks!

[autogrog]

Continuing failure to issue CPU work units; error msg is:
ERROR:WU01:FS00:Exception: Could not get an assignment.

System:
Ryzen 9 3950X running Fedora 32

No WU's for last 24 hours, and very few since July 03.
Is there any resolution pending?

[jjmiller]

Hi autogrog,

Thank you for posting about this and apologies for the delayed response as I'm current OOO. At the moment as far as we can tell this is an issue with the amount of load going to 128.252.203.11(12) as these two servers currently house the bulk of the available FAH jobs. To prevent server overload, we need to set a pretty heavy assignment cap on these two servers and they're currently being requested for jobs more than they're allowed to give out jobs. We've played around with increasing the assignment cap but that quickly leads to other issues with data transfer between the server and the outside world. We have some folks poking around to see if we can increase our I/O capacity, but this isn't likely to be resolved soon so we're stuck for the moment with our current assignment rate.

At our institution we're currently working on putting together a few new servers to house these jobs. Unfortunately this will be a delayed fix (possible ETA for public jobs ~2-3 weeks) as we want to put them through some internal testing before they get rolled out. In the meantime I see several large CPU projects in beta testing. Beta testing typically lasts ~1 week, though this varies from project to project. As I'm not involved with these projects and don't know when they entered beta testing I can't provide an ETA for when those jobs will become publicly available, but hopefully they will be available for folding soon.

[mgetz]

And now this server isn't accepting WU's apparently...

Code: Select all

02:52:04:WU01:FS01:Uploading 27.51MiB to 128.252.203.11
02:52:04:WU01:FS01:Connecting to 128.252.203.11:8080
02:52:35:WU01:FS01:Upload 1.14%
02:52:35:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
02:52:35:WU01:FS01:Trying to send results to collection server
02:52:35:WU01:FS01:Uploading 27.51MiB to 128.252.203.1
02:52:35:WU01:FS01:Connecting to 128.252.203.1:8080
02:52:41:WU01:FS01:Upload 56.58%
02:52:45:WU01:FS01:Upload complete
02:52:45:WU01:FS01:Server responded PLEASE_WAIT (464)
02:52:45:WARNING:WU01:FS01:Failed to send results, will try again later
02:53:04:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18202 run:18143 clone:1 gen:11 core:0x22 unit:0x000000010000000b0000471a000046df
02:53:04:WU01:FS01:Uploading 27.51MiB to 128.252.203.11
02:53:04:WU01:FS01:Connecting to 128.252.203.11:8080
02:53:36:WARNING:WU01:FS01:Exception: Failed to send results to work server: Not connected
02:53:36:WU01:FS01:Trying to send results to collection server
02:53:36:WU01:FS01:Uploading 27.51MiB to 128.252.203.1
02:53:36:WU01:FS01:Connecting to 128.252.203.1:8080
02:53:42:WU01:FS01:Upload 57.49%
02:53:46:WU01:FS01:Upload complete
02:53:46:WU01:FS01:Server responded PLEASE_WAIT (464)
02:53:46:WARNING:WU01:FS01:Failed to send results, will try again later
02:54:41:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18202 run:18143 clone:1 gen:11 core:0x22 unit:0x000000010000000b0000471a000046df
02:54:41:WU01:FS01:Uploading 27.51MiB to 128.252.203.11
02:54:41:WU01:FS01:Connecting to 128.252.203.11:8080
02:54:48:WU01:FS01:Upload 0.23%
02:55:19:WU01:FS01:Upload 1.36%
02:55:25:WU01:FS01:Upload 57.03%
02:55:29:WU01:FS01:Upload complete
02:55:29:WU01:FS01:Server responded PLEASE_WAIT (464)
02:55:29:WARNING:WU01:FS01:Failed to send results, will try again later

[Frontiers]

Same problem as mgetz's problem with 128.252.203.11, multiple send retries ended with WARNING:WU01:FS01:Exception: Failed to send results to work server: Not connected...

Code: Select all

06:01:35:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18202 run:14645 clone:1 gen:11 core:0x22 unit:0x000000010000000b0000471a00003935
06:01:35:WU01:FS01:Uploading 27.51MiB to 128.252.203.11
06:01:35:WU01:FS01:Connecting to 128.252.203.11:8080
06:02:07:WARNING:WU01:FS01:Exception: Failed to send results to work server: Not connected
06:02:07:WU01:FS01:Trying to send results to collection server
06:02:07:WU01:FS01:Uploading 27.51MiB to 128.252.203.10
06:02:07:WU01:FS01:Connecting to 128.252.203.10:8080
06:02:20:WU00:FS01:0x22:Completed 925000 out of 2500000 steps (37%)
06:02:29:WU01:FS01:Upload 1.36%
06:02:35:WU01:FS01:Upload 8.41%
06:02:41:WU01:FS01:Upload 15.91%
06:02:47:WU01:FS01:Upload 23.63%
06:02:53:WU01:FS01:Upload 31.36%
06:02:59:WU01:FS01:Upload 38.85%
06:03:05:WU01:FS01:Upload 46.58%
06:03:11:WU01:FS01:Upload 54.08%
06:03:17:WU01:FS01:Upload 61.80%
06:03:23:WU01:FS01:Upload 69.53%
06:03:29:WU01:FS01:Upload 77.03%
06:03:35:WU01:FS01:Upload 84.75%
06:03:41:WU01:FS01:Upload 92.48%
06:03:47:WU01:FS01:Upload 99.52%
06:03:48:WU01:FS01:Upload complete
06:03:48:WU01:FS01:Server responded PLEASE_WAIT (464)
06:03:48:WARNING:WU01:FS01:Failed to send results, will try again later
06:04:29:WU00:FS01:0x22:Completed 950000 out of 2500000 steps (38%)
06:04:30:WU00:FS01:0x22:Checkpoint completed at step 950000

[zarluk1]

viewtopic.php?f=108&t=37312#p352277

[JeansOn]

One more problem with 128.252.203.11 like above.

uploading is ok, but the server doesn't accept the WU
My client has just an other WU 16947 but I 'm hopefull, so I don't cancel the WU.
Losing ppd does no matter, but not beeing able to send, does.

08:49:12:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16947 run:3 clone:5 gen:208 core:0xa8 unit:0x00000005000000d00000423300000003
08:49:13:WU01:FS00:Uploading 21.71MiB to 129.32.209.203
08:49:13:WU01:FS00:Connecting to 129.32.209.203:8080
08:49:19:WU01:FS00:Upload 10.36%
08:49:25:WU01:FS00:Upload 26.19%
08:49:31:WU01:FS00:Upload 43.46%
08:49:37:WU01:FS00:Upload 59.58%
08:49:43:WU01:FS00:Upload 74.83%
08:49:49:WU01:FS00:Upload 92.10%
08:49:52:WU01:FS00:Upload complete
08:49:52:WU01:FS00:Server responded PLEASE_WAIT (464)
08:49:52:WARNING:WU01:FS00:Failed to send results, will try again later

[NGruia]

Code: Select all

******************************* Date: 2021-06-30 *******************************
08:35:09:ERROR:WU01:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-06-30 *******************************
13:59:22:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
15:39:59:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
17:49:46:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-06-30 *******************************
21:07:28:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
21:07:49:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
22:44:04:ERROR:WU01:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
22:44:26:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
00:21:57:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-01 *******************************
02:02:13:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-01 *******************************
08:34:06:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
10:13:48:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
10:14:09:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
13:31:51:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
13:32:12:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-01 *******************************
******************************* Date: 2021-07-01 *******************************
20:06:02:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
20:06:23:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
20:06:45:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
20:07:06:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
20:07:45:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
20:08:06:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
20:09:22:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
20:09:43:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
22:21:14:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
22:21:35:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-02 *******************************
******************************* Date: 2021-07-03 *******************************
******************************* Date: 2021-07-03 *******************************
08:26:32:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
08:26:53:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:50:36:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
11:50:57:WARNING:WU01:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
13:30:14:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-03 *******************************
16:46:05:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-03 *******************************
******************************* Date: 2021-07-04 *******************************
07:31:51:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
07:33:00:ERROR:WU00:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
******************************* Date: 2021-07-04 *******************************
07:31:51:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
07:33:00:ERROR:WU00:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
******************************* Date: 2021-07-04 *******************************
13:45:02:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
13:46:17:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
******************************* Date: 2021-07-04 *******************************
******************************* Date: 2021-07-04 *******************************
20:14:10:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
20:14:32:ERROR:WU00:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
00:33:48:ERROR:WU00:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-07-05 *******************************
******************************* Date: 2021-07-05 *******************************
09:03:03:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
10:41:24:ERROR:WU00:FS00:Exception: 10002: Received short response, expected 512 bytes, got 0
******************************* Date: 2021-07-05 *******************************
17:28:51:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-05 *******************************
******************************* Date: 2021-07-06 *******************************
******************************* Date: 2021-07-06 *******************************
12:32:37:ERROR:WU00:FS00:Exception: Server did not assign work unit
12:32:38:ERROR:WU00:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-07-06 *******************************
******************************* Date: 2021-07-06 *******************************
01:50:39:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-07 *******************************
05:06:29:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
05:06:50:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-07 *******************************
08:20:10:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
11:41:15:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
11:41:36:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:41:58:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
11:42:19:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:42:22:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
11:42:43:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
11:42:58:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
11:43:19:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-07 *******************************
18:21:01:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
******************************* Date: 2021-07-07 *******************************
00:35:05:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-08 *******************************
******************************* Date: 2021-07-08 *******************************
08:39:57:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
******************************* Date: 2021-07-08 *******************************
14:00:31:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
16:43:52:ERROR:WU01:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-07-08 *******************************
21:03:53:ERROR:WU01:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-07-09 *******************************
04:23:52:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:24:13:ERROR:WU01:FS00:Exception: Transfer failed
04:24:44:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:25:05:ERROR:WU01:FS00:Exception: Transfer failed
04:26:13:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:26:34:ERROR:WU01:FS00:Exception: Transfer failed
04:27:22:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:27:43:ERROR:WU01:FS00:Exception: Transfer failed
04:29:50:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
04:30:12:ERROR:WU01:FS00:Exception: Transfer failed
******************************* Date: 2021-07-09 *******************************
08:08:36:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
08:08:57:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-09 *******************************
18:07:47:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
******************************* Date: 2021-07-09 *******************************
21:29:40:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
21:30:30:ERROR:WU01:FS00:Exception: Transfer failed
01:49:40:ERROR:WU01:FS00:Exception: Server did not assign work unit
******************************* Date: 2021-07-10 *******************************
******************************* Date: 2021-07-10 *******************************
******************************* Date: 2021-07-10 *******************************
******************************* Date: 2021-07-10 *******************************
******************************* Date: 2021-07-11 *******************************
******************************* Date: 2021-07-11 *******************************
******************************* Date: 2021-07-11 *******************************
******************************* Date: 2021-07-11 *******************************
******************************* Date: 2021-07-12 *******************************
05:44:22:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
05:44:43:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
******************************* Date: 2021-07-12 *******************************
08:57:55:ERROR:WU01:FS00:Exception: Server did not assign work unit
08:58:17:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
08:58:38:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
08:59:17:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
08:59:25:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
08:59:39:ERROR:WU01:FS00:Exception: Failed to connect to 128.252.203.11:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
10:35:35:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80