STAT updates have stalled.

Moderators: Site Moderators, FAHC Science Team

Post Reply
cine.chris
Posts: 78
Joined: Sun Apr 26, 2020 1:29 pm

STAT updates have stalled.

Post by cine.chris »

Stat feeds have stalled... output from my feed timestamp tracker log.
20210227-010001
Sat Feb 27 04:04:26 GMT 2021
TimeStamp FAIL: EXITING
Image Image
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: STAT updates have stalled.

Post by drougnor »

I can confirm that the flat files are no longer updating as of 11pm EST on 2/26 (04:04 GMT, 2/27).

d
smoking2000
Posts: 471
Joined: Mon Dec 03, 2007 6:20 am
Location: Amsterdam
Contact:

Re: STAT updates have stalled.

Post by smoking2000 »

It works in a browser, but not with scripts (Perl LWP):

Code: Select all

$ HEAD -USe https://apps.foldingathome.org/teamstats/team92.txt
HEAD https://apps.foldingathome.org/teamstats/team92.txt
User-Agent: lwp-request/6.36 libwww-perl/6.36

500 Server closed connection without sending any data back
Content-Type: text/plain
Client-Date: Sat, 27 Feb 2021 14:21:58 GMT
Client-Warning: Internal response

Code: Select all

$ GET -USe https://apps.foldingathome.org/teamstats/team92.txt 
GET https://apps.foldingathome.org/teamstats/team92.txt
User-Agent: lwp-request/6.36 libwww-perl/6.36

500 Can't connect to apps.foldingathome.org:443 (SSL connect attempt failed)
Content-Type: text/plain
Client-Date: Sat, 27 Feb 2021 14:22:32 GMT
Client-Warning: Internal response

Can't connect to apps.foldingathome.org:443 (SSL connect attempt failed)

SSL connect attempt failed at /usr/share/perl5/LWP/Protocol/http.pm line 50.
The GET request sometimes works, but most times it fails.
cine.chris
Posts: 78
Joined: Sun Apr 26, 2020 1:29 pm

Re: STAT updates have stalled.

Post by cine.chris »

I was working on the tracker to follow the shifting updates.
Inspired to utilize the hourly updates, I wrote a fast updater.
Fast & phone friendly.. no ads, no graphics, no trackers.
Hopefully, they don't revert to the old 3hr cycle.

If people are interested, I'll upload the reduced active data file for download.

Page for my TechPowerUp friends
Image
Image Image
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: STAT updates have stalled.

Post by drougnor »

Stats are updating again, just FYI
Xavier Zepherious
Posts: 140
Joined: Fri Jan 21, 2011 8:02 am

Re: STAT updates have stalled.

Post by Xavier Zepherious »

Is stat server down again
EOC is not reporting
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: STAT updates have stalled.

Post by Neil-B »

Xavier Zepherious wrote:Is stat server down again
EOC is not reporting
I'm seeing updates in EOC ... haven't checked they are correct yet.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Xavier Zepherious
Posts: 140
Joined: Fri Jan 21, 2011 8:02 am

Re: STAT updates have stalled.

Post by Xavier Zepherious »

it was very late like 5:38
i was checking at regular intervals every few mins

it might mean that EOC will update later now from last stat server freeze
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: STAT updates have stalled.

Post by Neil-B »

There does seem to be a bit of varience in length of time to process - and possibly some minor(ish) differences between official stats and EOC that aren't readilly understandable ... I'll be giving it a day or so to stabilise as I guess there is a bit of catchup to resolve/balance out
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Joe_H
Site Admin
Posts: 7926
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: STAT updates have stalled.

Post by Joe_H »

EOC downloads a copy of the team and donor lists every 3 hours. For a while pre-COVID those files were generated every hour, so at worst of the creation fo the file took a bit longer than normal, the file EOC would get would only be an hour old.

Due to the loads on the database and front end server during the peaks of WU submission, the list files were taking longer than an hour to create. The frequency was reduced to every 4 hours. Jason from EOC contacted the person handling the stats server setup, and he agreed that every 3 hours would also work. As of the last I checked, that was still the frequency for creating those files.

Also during the peak period last year much of the stats API based checks were disabled. I have not looked to see which have been reenabled if any. I am assuming cine.chris is using the donor and team lists that can be downloaded from links on this page - https://foldingathome.org/statistics/, any others are not guaranteed to be always available. The page itself was not updated to reflect the change from every hour to every 3 hours last year, if they have returned to generating those files more frequently, that will show up in the timestamps.

Timestamp of the Team List I just downloaded is Sun Feb 28 16:54:58 GMT 2021, just under an hour old.

Edit: and just downloaded the team list again, its time stamp is about an hour later than the previous, so it appears they have gone back to hourly updates. No idea when the update starts.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
smoking2000
Posts: 471
Joined: Mon Dec 03, 2007 6:20 am
Location: Amsterdam
Contact:

Re: STAT updates have stalled.

Post by smoking2000 »

The update frequency of the stats files not particularly important, clients should use the If-Modified-Since to only download the files if they were modified on the server. That can be executed every hour with the actual processing of the data kicking in after there was something to download.

The problem was 500 Internal Server Error responses for something that should be a flat file.
Joe_H
Site Admin
Posts: 7926
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: STAT updates have stalled.

Post by Joe_H »

smoking2000 wrote:The problem was 500 Internal Server Error responses for something that should be a flat file.
Yes, that was fixed yesterday by the time of the EOC 3 PM CST update. My response was to the bits about EOC and stats updating frequency.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Xavier Zepherious
Posts: 140
Joined: Fri Jan 21, 2011 8:02 am

Re: STAT updates have stalled.

Post by Xavier Zepherious »

EOC updates now at 32 mins past the hour every 3 hrs
was something like 15m after before

so other than it's an annoyance that the time changed... it's fine

just sort make people go ??? every time the time drifts on it

begs the question is it EOC fault or the server
Joe_H
Site Admin
Posts: 7926
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: STAT updates have stalled.

Post by Joe_H »

Just the number of entries EOC is needing to process from the flat files. The update still starts on the hour every 3 hours, it just is taking longer to process, especially since Jason increased the number of teams and users tracked.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
cine.chris
Posts: 78
Joined: Sun Apr 26, 2020 1:29 pm

Re: STAT updates have stalled.

Post by cine.chris »

Timestamps, since the restart, have drifted about 3 minutes, settling in at 52:25 initially, currently 55:09.
So, it appears, the drift of ~3.6min/day that I was seeing, is still occurring.
But, that is a different thread. The stall is fixed.
Image Image
Post Reply