New Servers Idle?

Moderators: Site Moderators, FAHC Science Team

PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: New Servers Idle?

Post by PantherX »

UofM.MartinK wrote:...how much computational work was actually completed by how many clients (the "hours clients spent computing" divided by "wall time" mentioned by Joseph Coffland is a good start)...
There's a new number of compute hours logged per hour graph here for you to view: https://apps.foldingathome.org/credit-log

When there's a new Work Server that is being on-boarded, there's a lot of stuff that needs to happen. It is a mixture of human input, waiting for various tasks to finish and then the verification before it goes into production. A blank Work Server will need to be populated with WUs so that it can then assign them to the clients. This step is rather intensive and can take a while depending on how many WUs are gong to be generated.

Moreover, F@H Team works with other teams to decide which proteins to focus on. For sake of argument, let's have a massive team of researches the bottleneck would not be F@H, instead, it would be the teams involved in X-ray crystallography. That unfortunately can't be solved by distributed computing since it requires experimentation and very specialized expensive equipment (not to mention time, money, scientists). It is great to see so many teams collaborating and working with each other to fight the common enemy!

My take is that while significant changes have happened over the last few weeks, the momentum is bound to continue for some more weeks so am looking forward to them :)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: New Servers Idle?

Post by UofM.MartinK »

PantherX wrote: There's a new number of compute hours logged per hour graph here for you to view: https://apps.foldingathome.org/credit-log

When there's a new
That is a great addition to the existing stats! How did you hear about it?

The "Credits" in the top is WUs, I assume?

Next cool stat would be how many assignments did the AS actually hand out per hour :)
JimF
Posts: 651
Joined: Thu Jan 21, 2010 2:03 pm

Re: New Servers Idle?

Post by JimF »

Neil-B wrote:"Conventional" supercomputers for the most part are (unlike a decade or so ago) a very closely connected high bandwidth low latency distributed compute platform (without much distributedness) … They are actually much better for some types of science and calculations than the distributed platforms and as such for the most part are used for purposes that require their performance characteristics - "It is not all about the FLOPs" is fair to say, or even "It is not how much power you have got but how you use it" … They are also fiercely expensive to purchase and support … FAH is probably even more expensive but its model minimises costs to the scientists and effectively crowdsources the costs outwards, by having people donate parts of their electricity bills, pc hardware/software budget, network charges - total all this up and it will be even more scary that the total compute power !! - oh yes and support, guess that is "paid for" by the myriad volunteers who try each to their own ability to help the next folder who encounters problems they have already seen/dealt with - no call centre costs, or salaries for First Line and Deep Technical support, or 24hr call out charges :)
The latency is a key point that I don't think most crunchers appreciate. They purposely created the MSM in order to allow for the high-latency (delay) in distributed computing. That reduces the efficiency as compared to a typical supercomputer, which would not allow for more than a few microseconds of delay, not minutes or hours (or days with BOINC).

However, "efficiency" is a tricky proposition. It depends on what you are comparing it to. If you can treat some diseases with Folding (or other DC projects), it is a lot cheaper than a pandemic.

Crunch on.
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: New Servers Idle?

Post by PantherX »

UofM.MartinK wrote:...That is a great addition to the existing stats! How did you hear about it?...
I actually saw it early last week before the Fire Side Dev chat and then heard Joseph talk about it. There was another Stats dashboard that he was working on but had to disable it since the surge of new donors meant that it wasn't keeping up (as he was still experimenting and it wasn't ready for production). If you know of people who would be able to represent Stats in a better and informative way or have database experience, they would be welcomed to help out: https://www.reddit.com/r/pcmasterrace/c ... s/fkyg9ns/
UofM.MartinK wrote:...The "Credits" in the top is WUs, I assume?...
Yep, how may WUs were processed by the Stats system.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
warrelium
Posts: 6
Joined: Mon Mar 30, 2020 3:23 am

Re: New Servers Idle?

Post by warrelium »

Seems like we're running out of GPU WUs. They've been trickling in for me for the past several days and it looks like even the ec2 spot donations from AWS are having a problem getting work since 4/8: https://folding.extremeoverclocking.com ... =&u=928671
Endgame124
Posts: 52
Joined: Sat Mar 28, 2020 1:22 am

Re: New Servers Idle?

Post by Endgame124 »

warrelium wrote:Seems like we're running out of GPU WUs. They've been trickling in for me for the past several days and it looks like even the ec2 spot donations from AWS are having a problem getting work since 4/8: https://folding.extremeoverclocking.com ... =&u=928671
Is aws having problems with work, or did they pull the plug?
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: New Servers Idle?

Post by UofM.MartinK »

PantherX wrote: https://apps.foldingathome.org/credit-log
UofM.MartinK wrote:...The "Credits" in the top is WUs, I assume?...
Yep, how may WUs were processed by the Stats system.
Do you (or anybody) have a screenshot (or the data) of this plot from two or more days ago?

Suddenly, it looks a LOT more even than yesterday, and I forget to save the dataset from earlier...

I am working on correlating these "credit-log" WU stats with
* WS WU reports from https://apps.foldingathome.org/serverstats (separated by GPU & CPU),
* client logs for statistic of how often clients were successful or failed in retrieving WUs.

I currently use my own client logs from the last 21 days (6 CPU-Only clients), so it is not representative,
but should help putting some numbers into perspective.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: New Servers Idle?

Post by Neil-B »

A couple of days back the log had a big spike on it that related to one of the stats catch ups - this has now aged off left and the graph is much more regular - could that be it?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: New Servers Idle?

Post by UofM.MartinK »

There was definitely a spike, yes, and some more extreme ups/downs. Is it certain that this was all only stats catch up?

The database knows when each WU was returned - and it is important to calculate the speed bonus points. So even if there was a backlog in processing the data, in theory the graph could reflect the actual work returned in each interval. So which of the two things happened:
* all stats finally processed correctly and we are now more real-time (then a two-week or month view would be great - since it would also be much more even and representative)
* this plot does not account for the actual return date of the WUs, it is merely a gauge for what the stats system is crunching

I know this credit-log is kinda "work-in-progress", so there is no "good" or "bad", "right" or "wrong", but can we quickly figure out/get feedback on what this (very useful!) plot is based on?
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: New Servers Idle?

Post by Neil-B »

Think the graph shows when the credit was made to the stats pool not then the credit was earned - but may well be wrong.

I'll know in a bit cause I have 80+ WUs returned to one of the severs that has yet to post anything to stats - and I won't be alone on that ... when they hit there should be a smallish but noticeable spike if my guess is right - if they work off credit earned there won't be a spike
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: New Servers Idle?

Post by bruce »

Neil-B wrote:Think the graph shows when the credit was made to the stats pool not then the credit was earned - but may well be wrong.
Right.
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: New Servers Idle?

Post by UofM.MartinK »

bruce wrote:
Neil-B wrote:Think the graph shows when the credit was made to the stats pool not then the credit was earned - but may well be wrong.
Right.
Right that he is wrong, or right that "the graph shows when the credit was made to the stats pool" ? :)
Last edited by UofM.MartinK on Wed Apr 15, 2020 6:45 pm, edited 1 time in total.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: New Servers Idle?

Post by Neil-B »

Normally I'd bet on me being wrong … but I think bruce might have been indicating that for once I'd got it right :) … or maybe I've got that wrong as well :?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
UofM.MartinK
Posts: 59
Joined: Tue Apr 07, 2020 8:53 pm

Re: New Servers Idle?

Post by UofM.MartinK »

I now also think he meant you got it right. I just reviewed some of the historical data (unfortunately the one older than 8 days is lost to me), and once an hourly value was "published" it seems to never change. This is supporting the hypothesis of "the graph shows when the credit was made to the stats pool not then the credit was earned"
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: New Servers Idle?

Post by PantherX »

The credits are allocated to the completed WU when it is successfully uploaded to the WS or CS. The Stats system simply goes out to each of the WS/CS and collects the credits to process it. Thus, if a WS/CS is offline, the processing of the credits is delayed but not lost.

Unfortunately, I don't have a screenshot of what the graph was few days/week ago.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Post Reply