Correct there are not just few but ZERO CPU WUs available?

If you're new to FAH and need help getting started or you have very basic questions, start here.

Moderators: Site Moderators, FAHC Science Team

Yujah
Posts: 41
Joined: Tue Mar 24, 2020 2:36 pm

Correct there are not just few but ZERO CPU WUs available?

Post by Yujah »

One of the many new participants here; as such still ogling things...

Although I've been processing some GPU WU's, for the entire day today, 12 hours certainly, I have not received any CPU WUs; the CPU queue just sits idle. Would this still be expected? I read about the shortage but was expecting to get one or two at least; did get some yesterday.

If it helps; my folding-name is also Yujah (currently on team 223518, and passkey'd) and the CPU "Ready" queue lists 128.252.203.10 as the Work Server.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by bruce »

I was in a discussion last night discussing the problem of having CPU assignments that are not getting enough people to run them.

Give me details.

EDIT: here are some that came on-line recently. viewtopic.php?f=24&t=33290
I'm sure there are others.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Neil-B »

bruce wrote:I was in a discussion last night discussing the problem of having CPU assignments that are not getting enough people to run them.

Give me details.

EDIT: here are some that came on-line recently. viewtopic.php?f=24&t=33290
I'm sure there are others.
Not intending to highjack thread, but this surprises me a bit … was like the OP just letting my server wait for WUs … It has occasionally picked up on but for the most part has been idle … Running two 28 cpu slots on Neck-W, Team 39363 … If there are CPU assignments not getting done then there is a bottleneck somewhere … btw happy to run more smaller cpu slots if that helps - but is nice to see a 14 day WU crunched in 3/4 hrs.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Neil-B »

bad form reply to self … but then as if by magic two WUs drop in :) … You are right they do exist - Thanks.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Yujah
Posts: 41
Joined: Tue Mar 24, 2020 2:36 pm

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Yujah »

My CPU's still as idle as it has been all day...

As a newbie I'm relatively uncertain of which exact details to provide (did follow the link in your sig) but studying the log I just now saw that the last CPU WU request in fact showed an explicit error. Only the last, so this may be a red herring, but, with the error message between [ ] manually translated into English from my native language:

===
20:33:00:WU00:FS00:Connecting to 65.254.110.245:8080
20:33:01:WU00:FS00:Assigned to work server 128.252.203.10
20:33:01:WU00:FS00:Requesting new work unit for slot 00: READY cpu:7 from 128.252.203.10
20:33:01:WU00:FS00:Connecting to 128.252.203.10:8080
20:33:18:WU01:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
20:33:22:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
20:33:22:WU00:FS00:Connecting to 128.252.203.10:80
20:33:43:ERROR:WU00:FS00:Exception: Failed to connect to 128.252.203.10:80: [ A connection attempt has failed due to the connected party answering incorrectly after a certain time, or the established connection has failed due to the connected host not answering ]
===

As said, may be a red herring; before there's just many instances of e.g.

===
20:03:58:WU00:FS00:Connecting to 65.254.110.245:8080
20:03:59:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
20:03:59:WU00:FS00:Connecting to 18.218.241.186:80
20:04:05:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
20:04:05:ERROR:WU00:FS00:Exception: Could not get an assignment
===

Are these the requested details? As said, name Yujah, team 223518; am running GPU WUs --- currently 11762 (0, 3982, 11) --- and the CPU queue is a multi-threaded quad-core, i.e., 8 "CPUs".
Wakkaluba
Posts: 4
Joined: Tue Mar 24, 2020 9:26 pm

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Wakkaluba »

I am facing similar issues.
Normally was at least the GPU working more or less uninterrupted and the CPU slot got 1-2 times a day something to do.
But now? FAH is picking nose since this early morning.

Code: Select all

21:30:57:WU00:FS00:Connecting to 18.218.241.186:80
21:30:57:WU01:FS01:Connecting to 65.254.110.245:8080
21:30:57:WARNING:WU00:FS00:[b]Failed to get assignment[/b] from '18.218.241.186:80': No WUs available for this configuration
21:30:57:ERROR:WU00:FS00:Exception: Could not get an assignment
21:30:57:WARNING:WU01:FS01:[b]Failed to get assignment[/b] from '65.254.110.245:8080': No WUs available for this configuration
21:30:57:WU01:FS01:Connecting to 18.218.241.186:80
21:30:58:WARNING:WU01:FS01:[b]Failed to get assignment[/b] from '18.218.241.186:80': No WUs available for this configuration
21:30:58:ERROR:WU01:FS01:Exception: Could not get an assignment
something that occasionally appeared for the CPU slot also appears for the GPU every now and then..

Code: Select all

21:03:31:WU00:FS00:Connecting to 65.254.110.245:8080
21:03:31:WU01:FS01:Assigned to work server 40.114.52.201
21:03:31:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Tahiti XT [Radeon R9 200/HD 7900/8970] from 40.114.52.201
21:03:31:WU01:FS01:Connecting to 40.114.52.201:8080
21:03:31:WU00:FS00:Assigned to work server 128.252.203.9
21:03:31:WU00:FS00:Requesting new work unit for slot 00: READY cpu:2 from 128.252.203.9
21:03:31:WU00:FS00:Connecting to 128.252.203.9:8080
21:03:43:ERROR:WU01:FS01:Exception: 10002:[b] Received short response, expected 512 bytes, got 0[/b]
21:03:44:ERROR:WU00:FS00:Exception: 10002: [b]Received short response, expected 512 bytes, got 0[/b]
A workserver is assigned but I see the collection server as 0.0.0.0

Not sure if the software fumbles when no collection server is available.
My experience from day one was that something in the infrastructure is causing some race conditions.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Neil-B »

Your logs have a very similar pattern to the ones I have been seeing mostly the "No WUs available …" with the occasional nearly - where the request is assigned to a work server but then no response (variety of failure messages) … I have also been seeing an HTTP error at the same type stage iirc … From other posts I had put it down to 1) no WUs (or at least overloaded assignment server) … 2) Overloaded work server not able to service the assignments passed on by the assignment server … 3) network cards struggling with the throughput … Having said that though I am hoping that Bruce is able to spot something that just needs a tweak to keep CPU WU crunchers happy - If it is just heavy load I'll happily leave kit on until such time as it is called on.

Good luck with the CPU WU allocations
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Jesse_V
Site Moderator
Posts: 2850
Joined: Mon Jul 18, 2011 4:44 am
Hardware configuration: OS: Windows 10, Kubuntu 19.04
CPU: i7-6700k
GPU: GTX 970, GTX 1080 TI
RAM: 24 GB DDR4
Location: Western Washington

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Jesse_V »

I'm definitely not Bruce, but I will refer you to a couple of sticky threads at the top of this board: viewtopic.php?f=61&t=33193 and viewtopic.php?f=61&t=32424

The servers are just absolutely flooded with bandwidth and disk I/O. Nobody here really expected a 20x increase in F@h userbase over a few month. A tweak isn't really going to work, but the teams are getting more servers online and more projects into the queue to keep up with the overwhelming demand.
F@h is now the top computing platform on the planet and nothing unites people like a dedicated fight against a common enemy. This virus affects all of us. Lets end it together.
Joe_H
Site Admin
Posts: 7940
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Joe_H »

Especially with most of that 20x increase just in the last 10-14 days.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Yujah
Posts: 41
Joined: Tue Mar 24, 2020 2:36 pm

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Yujah »

As said, I know about the shortage; was/am specifically asking if 0 rather than just few is still expected. And especially in the context of me easily getting CPU WUs only two days ago when I was completely new (and was for a bit folding under a different name then); believe I got 6 or so in a row that first day. it's the 0 rather than few in 12+ hours now which currently has me wonder if I have a local issue.
uyaem
Posts: 219
Joined: Sat Mar 21, 2020 7:35 pm
Location: Esslingen, Germany

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by uyaem »

Yujah wrote:As said, I know about the shortage; was/am specifically asking if 0 rather than just few is still expected. And especially in the context of me easily getting CPU WUs only two days ago when I was completely new (and was for a bit folding under a different name then); believe I got 6 or so in a row that first day. it's the 0 rather than few in 12+ hours now which currently has me wonder if I have a local issue.
That's quite normal, I've been getting many GPU ones and few CPU ones since... Saturday.
Also note that the retry timer goes up significantly after each attempt (hours).
As they keep saying, they are working on it :)
Image
CPU: Ryzen 9 3900X (1x21 CPUs) ~ GPU: nVidia GeForce GTX 1660 Super (Asus)
jonault
Posts: 216
Joined: Fri Dec 14, 2007 9:53 pm

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by jonault »

Yujah wrote:it's the 0 rather than few in 12+ hours now which currently has me wonder if I have a local issue.
Based on the log file snippets you posted above, you don't have a local issue. 20:03:59:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration means you successfully connected to the work server, it just didn't have anything for you. Nothing you can do except wait for more WUs to become available.
Image
Yujah
Posts: 41
Joined: Tue Mar 24, 2020 2:36 pm

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by Yujah »

Hrmpz. Well, guess I'll exercise patience then.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by bruce »

jonault wrote:
Yujah wrote:it's the 0 rather than few in 12+ hours now which currently has me wonder if I have a local issue.
Based on the log file snippets you posted above, you don't have a local issue. 20:03:59:WARNING:WU00:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration means you successfully connected to the work server, it just didn't have anything for you. Nothing you can do except wait for more WUs to become available.
Not quite. It means you've successfully connected to the Assignment Server, whose job it is to find a Work Server that has WUs for your hardware.

Suppose there are several work servers that do have WUs for you but (A) they're all down for some reason or (B) They all are saturated and currently incapable of accepting any more concurrent requests. It's factually correct that the AS cannot find any Work Server that has work for your configuration but the message doesn't suggest those alternatives.

Actually, (B) is quite common right now. Be patient and sooner or later there will be a gap during which you can get download a WU.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Correct there are not just few but ZERO CPU WUs availabl

Post by bruce »

The new servers we brought online yesterday are now completely saturated. The are (rare) breaks when there's room for a new client to get work, but you have to be lucky to hit one of those moments. Be patient. Supply and Demand, where Demand increases at a faster rate than Supply.
Post Reply