Failed to get assignment - No WUs available [Fixed]

[HaloJones]

Yesterday, my computer was not able to get another work unit after finishing one. I tried pausing and restarting folding, that didn't work. Then I ran junk file clean up and registry clean up, and a computer reboot. After the reboot, I was able to get another work unit.

Today, the same problem. I tried a reboot, that didn't work. Then I ran junk file clean up and registry clean up and paused and restarted folding, and I got another work unit. Here is the log.

Code: Select all

*********************** Log Started 2017-08-20T21:39:53Z ***********************
21:39:55:WU01:FS01:Connecting to 171.67.108.45:80
21:39:57:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': Empty work server assignment
21:39:57:WU01:FS01:Connecting to 171.64.65.35:80

Tried to add a cpu client for the first time in ages and this is what I get, 24 hours now of Empty Work server on 171.64.65.35 and 171.67.108.45

Won't bother then

[Joe_H]

Yesterday, my computer was not able to get another work unit after finishing one. I tried pausing and restarting folding, that didn't work. Then I ran junk file clean up and registry clean up, and a computer reboot. After the reboot, I was able to get another work unit.

Today, the same problem. I tried a reboot, that didn't work. Then I ran junk file clean up and registry clean up and paused and restarted folding, and I got another work unit. Here is the log.

Code: Select all

*********************** Log Started 2017-08-20T21:39:53Z ***********************
21:39:55:WU01:FS01:Connecting to 171.67.108.45:80
21:39:57:WARNING:WU01:FS01:Failed to get assignment from '171.67.108.45:80': Empty work server assignment
21:39:57:WU01:FS01:Connecting to 171.64.65.35:80

Tried to add a cpu client for the first time in ages and this is what I get, 24 hours now of Empty Work server on 171.64.65.35 and 171.67.108.45

Won't bother then

Absent any information as to the settings for the CPU slot you set up, no idea why you did not get an assignment as there are plenty of WU's for CPU folding. First guess is that your CPU setting was a multiple of a prime number greater than 5. That affects setups using the 7.4.4 release, the public beta version 7.4.16 has additional code to negotiate with the AS for WU's available up to the maximum set for the slot.

Also it would help if you actually read through the topic you posted in, this was a problem with GPU WU assignments due to an entirely different issue

[HaloJones]

And yet, the errors are identical with the exact same messages in the logs regarding the exact same servers.

I click Configure, Slots, Add, CPU. That's it. It's a 2600K running Windows10 x64 UK. If doing that doesn't result in work but instead in an error regarding empty work, what do you expect the average Joe to do?

[bruce]

We'd really like Stanford's Development team to put the finishing touches on the beta client V7.4.16+ and release it. This will get rid of the problems inherent in V7.4.4. Unfortunately, we're volunteers, like yourself running a support site and all we can do is tell you how to work around the known issues with the version you're running.

Moving the slider to Medium instead of Full will most likely also work. That should give you a similar configuration to the one provided by the beta client. Unfortunately it's still a guess because you didn't post the essential parts of your log.

[lazyacevw]

In FAHControl I pause individual slots and then fold one at a time and eventually I get a WU.

Just in case anyone reads this and considers trying, by doing this you are doing is resetting the reconnect time-out with the assignment server. The fahclient implements this incrementing time-out feature to prevent clients from overloading/spamming the assignment servers with connection requests.

You could very well be essentially conducting a Denial of Service attack with your over-frequent connection attempts, depending on the load of the server.

Just let the software run and keep an eye on it over time. It works as designed. If you would like to do some troubleshooting/experimenting, check here viewtopic.php?f=18&t=17794 .

[blandyuk]

Is there any sort of protection on the WU servers when clients connect? I have a 6x GPU rig so when it starts, there will be 6x concurrent connections to the WU server. My GPU rig is not running at full capacity at all and I'm lucky to get 2-3 of my 6 GPUs working on WUs.

Now, I have two of these rigs so that's 12 GPUs, then add my personal PC with another GPU so that's 13 GPUs all from the same IP. Then add in a few CPUs on top! There is something wrong as I struggle to get any WUs at all. Can you confirm and if so, relax the SPAM settings on the WU servers?

[Jesse_V]

Is there any sort of protection on the WU servers when clients connect? I have a 6x GPU rig so when it starts, there will be 6x concurrent connections to the WU server. My GPU rig is not running at full capacity at all and I'm lucky to get 2-3 of my 6 GPUs working on WUs.

Now, I have two of these rigs so that's 12 GPUs, then add my personal PC with another GPU so that's 13 GPUs all from the same IP. Then add in a few CPUs on top! There is something wrong as I struggle to get any WUs at all. Can you confirm and if so, relax the SPAM settings on the WU servers?

That's an impressive setup you have there!

The servers are just slammed in general due to the very rapid expansion of new F@h users. Workunits will be available once the teams add new servers and get more projects in the queue. viewtopic.php?f=61&t=32424

[blandyuk]

Awesome thanks. I just wanted to make sure I was not getting blocked due to the number of connections from my side

Cheers.

[bruce]

I've got years of expereience in running the client but I'm constantely discovering new things. I read lazyacevw comments (2 posts up) and agree with what he said. Nevertheless, my recent experiences suggest that there's probably more.

Last night I had a CPU client that has been trying to get work for a long time. (i.e.- the time until it tries again has gotten very long.) I pause the slot and unpaused it. It tried once (got a server error) and went back to waiting. In other words, the timer did not reset. I had the same situation on another machine. Microsoft was telling me it had updates to install so I told it to proceed although that would interrupt the GPU slot before restarting from the checkpoint.

After the reboot, the anti-spam interval was reset so it has been failing to get new work more frequently than the firs one. There was one other difference. Machine 1 is just trying one server. Machine 2 tries two servers before going back to waiting. Both clients are Windows 7.5.1 so there's no identified difference on my end.

After a while, machine 1 got an assignment, completed it, an attempted to upload the result. That upload failed so it went to a CS, where it was dumped. At that point, Machine 1 started trying the same 2 servers that machine 2 has been trying.

I don't expect any special treatment. We're all in this together and if the servers don't have enough WUs to distribute, I presume we all are seeing exactly the same unhappy scenarios. I'm just reporting what I'm seeing.

According to the serverstat app, there are 7 servers which are configured to distribute FAHCore_a7 assignments. My machines are trying two of them and getting errors. According to my understanding, the Assignment Server logic should have the information that (maybe 5 of them) are out of work but it still sending me to the other 2. I wonder: Is that logic working correctly?

[wagneripjr]

16:20:22:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
16:20:22:ERROR:WU01:FS01:Exception: Could not get an assignment

How do I fix this?

[lazyacevw]

16:20:22:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
16:20:22:ERROR:WU01:FS01:Exception: Could not get an assignment

How do I fix this?

This is a known issue. Essentially, your client is looking for work from the assignment server but the assignment doesn't have any work to assign. So many new clients have been brought up, FAH has more horsepower than they've ever had and (A) they are struggling to create enough work to do (a good thing), (B) their software, servers, and networking infrastructure hasn't been stress tested to this level before so things aren't running as smoothly as they used to (not good but not terrible). Read the lastest post @ https://folding.extremeoverclocking.com . FAH is working with some pros to beef up their infrastructure to eliminate any instances (or possibilities) of issues on their end so we can do the business on our end. Right now, it is difficult to tell if the issues that everybody is experiencing is simply an out of work scenario or if there is a problem in the FAH ecosystem. More than likely it is a mixture of both.

My question is, who is going to deliver the redbull and cocaine to the scientist writing the folding problems?

I've got years of expereience in running the client but I'm constantely discovering new things. I read lazyacevw comments (2 posts up) and agree with what he said. Nevertheless, my recent experiences suggest that there's probably more.

According to the serverstat app, there are 7 servers which are configured to distribute FAHCore_a7 assignments. My machines are trying two of them and getting errors. According to my understanding, the Assignment Server logic should have the information that (maybe 5 of them) are out of work but it still sending me to the other 2. I wonder: Is that logic working correctly?

You do have a good line of reasoning that should be investigated, along with the sea of other optimizations that should be done with the FAH infrastructure. Based on my professional experience, they are probably dealing with a wide spectrum of issues that even my company would struggle to address (and we get paid well to do so!). Hopefully they get some great volunteers from Microsoft and GitHub that will sort it all out.

Personally, I think they really need to be contacting Cloudfare with all of the increased network demands. Load balancing and network optimization can go a long way in increasing infrastructure availability.

FYI as of this month, according to some rando who updated wikipedia, FAH is now unofficially the 2nd fastest supercomputer in the world with over 98.7 PFLOPS of processing power. https://en.wikipedia.org/wiki/Folding@home https://en.wikipedia.org/wiki/TOP500 . Not bad for a pieced together system run by smatterings of volunteers.

P.S. I'm not sure what to make of FAH statistics page https://stats.foldingathome.org/os . If I read it correctly, it currently shows 1.431 ExaFLOPS of processing power. Not sure what to think about that...

[lazyacevw]

I also should mention, just in case anyone wants to contribute to the max, that you can also run BOINC with Rosetta@home at the same time as FAH Client/Control. Rosetta@home only has CPU WUs so it can fill the idle time that FAH is doing nothing with your CPU. Rosetta@home is also working on the COVID-19 problem, but in a different computational approach.

BOINC has some neat features that FAHClient doesn't, like the ability to AUTOMATICALLY stop processing its tasks if another specific task starts running on your computer. For instance, I have my BOINC set up so that if it sees a FAHCore_a7 task start, it pauses CPU processing until FAH is done and then automatically resumes! I have FAH running my GPU 24/7 with no shortage of WUs and now FAH and Rosetta@home share my CPU, with FAH having full priority.

[Magnepan]

I also should mention, just in case anyone wants to contribute to the max, that you can also run BOINC with Rosetta@home at the same time as FAH Client/Control. Rosetta@home only has CPU WUs so it can fill the idle time that FAH is doing nothing with your CPU. Rosetta@home is also working on the COVID-19 problem, but in a different computational approach.

BOINC has some neat features that FAHClient doesn't, like the ability to AUTOMATICALLY stop processing its tasks if another specific task starts running on your computer. For instance, I have my BOINC set up so that if it sees a FAHCore_a7 task start, it pauses CPU processing until FAH is done and then automatically resumes! I have FAH running my GPU 24/7 with no shortage of WUs and now FAH and Rosetta@home share my CPU, with FAH having full priority.

Neat, this was really easy to setup Boinc to preempt any GPU tasks for FahCore_21 or FahCore_22. I'll have a lot less hand wringing about feeling like I've donated my time now.

[puuteknikko]

After a while, machine 1 got an assignment, completed it, an attempted to upload the result. That upload failed so it went to a CS, where it was dumped. At that point, Machine 1 started trying the same 2 servers that machine 2 has been trying.[/b]

I saw the same behaviour with my finished CPU assignment. It was a huge one, calculations took over 2 days so the upload size was a bit over 700 megabytes. All gone to waste now.. it just tried hammering the two servers until the deadline day caused it to be dumped

[tpatch54]

9:38:24:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:38:24:WU01:FS01:0x22:Version 0.0.2
19:38:45:WU01:FS01:0x22:Completed 0 out of 2000000 steps (0%)

Mine just started back up, maybe they are back up and working.