STAT updates have stalled.
Moderators: Site Moderators, FAHC Science Team
-
- Posts: 78
- Joined: Sun Apr 26, 2020 1:29 pm
STAT updates have stalled.
Stat feeds have stalled... output from my feed timestamp tracker log.
20210227-010001
Sat Feb 27 04:04:26 GMT 2021
TimeStamp FAIL: EXITING
20210227-010001
Sat Feb 27 04:04:26 GMT 2021
TimeStamp FAIL: EXITING
Re: STAT updates have stalled.
I can confirm that the flat files are no longer updating as of 11pm EST on 2/26 (04:04 GMT, 2/27).
d
d
-
- Posts: 471
- Joined: Mon Dec 03, 2007 6:20 am
- Location: Amsterdam
- Contact:
Re: STAT updates have stalled.
It works in a browser, but not with scripts (Perl LWP):
The GET request sometimes works, but most times it fails.
Code: Select all
$ HEAD -USe https://apps.foldingathome.org/teamstats/team92.txt
HEAD https://apps.foldingathome.org/teamstats/team92.txt
User-Agent: lwp-request/6.36 libwww-perl/6.36
500 Server closed connection without sending any data back
Content-Type: text/plain
Client-Date: Sat, 27 Feb 2021 14:21:58 GMT
Client-Warning: Internal response
Code: Select all
$ GET -USe https://apps.foldingathome.org/teamstats/team92.txt
GET https://apps.foldingathome.org/teamstats/team92.txt
User-Agent: lwp-request/6.36 libwww-perl/6.36
500 Can't connect to apps.foldingathome.org:443 (SSL connect attempt failed)
Content-Type: text/plain
Client-Date: Sat, 27 Feb 2021 14:22:32 GMT
Client-Warning: Internal response
Can't connect to apps.foldingathome.org:443 (SSL connect attempt failed)
SSL connect attempt failed at /usr/share/perl5/LWP/Protocol/http.pm line 50.
-
- Posts: 78
- Joined: Sun Apr 26, 2020 1:29 pm
Re: STAT updates have stalled.
I was working on the tracker to follow the shifting updates.
Inspired to utilize the hourly updates, I wrote a fast updater.
Fast & phone friendly.. no ads, no graphics, no trackers.
Hopefully, they don't revert to the old 3hr cycle.
If people are interested, I'll upload the reduced active data file for download.
Page for my TechPowerUp friends
Inspired to utilize the hourly updates, I wrote a fast updater.
Fast & phone friendly.. no ads, no graphics, no trackers.
Hopefully, they don't revert to the old 3hr cycle.
If people are interested, I'll upload the reduced active data file for download.
Page for my TechPowerUp friends
Re: STAT updates have stalled.
Stats are updating again, just FYI
-
- Posts: 140
- Joined: Fri Jan 21, 2011 8:02 am
Re: STAT updates have stalled.
Is stat server down again
EOC is not reporting
EOC is not reporting
-
- Posts: 1996
- Joined: Sun Mar 22, 2020 5:52 pm
- Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21 - Location: UK
Re: STAT updates have stalled.
I'm seeing updates in EOC ... haven't checked they are correct yet.Xavier Zepherious wrote:Is stat server down again
EOC is not reporting
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
-
- Posts: 140
- Joined: Fri Jan 21, 2011 8:02 am
Re: STAT updates have stalled.
it was very late like 5:38
i was checking at regular intervals every few mins
it might mean that EOC will update later now from last stat server freeze
i was checking at regular intervals every few mins
it might mean that EOC will update later now from last stat server freeze
-
- Posts: 1996
- Joined: Sun Mar 22, 2020 5:52 pm
- Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21 - Location: UK
Re: STAT updates have stalled.
There does seem to be a bit of varience in length of time to process - and possibly some minor(ish) differences between official stats and EOC that aren't readilly understandable ... I'll be giving it a day or so to stabilise as I guess there is a bit of catchup to resolve/balance out
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
-
- Site Admin
- Posts: 7926
- Joined: Tue Apr 21, 2009 4:41 pm
- Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2 - Location: W. MA
Re: STAT updates have stalled.
EOC downloads a copy of the team and donor lists every 3 hours. For a while pre-COVID those files were generated every hour, so at worst of the creation fo the file took a bit longer than normal, the file EOC would get would only be an hour old.
Due to the loads on the database and front end server during the peaks of WU submission, the list files were taking longer than an hour to create. The frequency was reduced to every 4 hours. Jason from EOC contacted the person handling the stats server setup, and he agreed that every 3 hours would also work. As of the last I checked, that was still the frequency for creating those files.
Also during the peak period last year much of the stats API based checks were disabled. I have not looked to see which have been reenabled if any. I am assuming cine.chris is using the donor and team lists that can be downloaded from links on this page - https://foldingathome.org/statistics/, any others are not guaranteed to be always available. The page itself was not updated to reflect the change from every hour to every 3 hours last year, if they have returned to generating those files more frequently, that will show up in the timestamps.
Timestamp of the Team List I just downloaded is Sun Feb 28 16:54:58 GMT 2021, just under an hour old.
Edit: and just downloaded the team list again, its time stamp is about an hour later than the previous, so it appears they have gone back to hourly updates. No idea when the update starts.
Due to the loads on the database and front end server during the peaks of WU submission, the list files were taking longer than an hour to create. The frequency was reduced to every 4 hours. Jason from EOC contacted the person handling the stats server setup, and he agreed that every 3 hours would also work. As of the last I checked, that was still the frequency for creating those files.
Also during the peak period last year much of the stats API based checks were disabled. I have not looked to see which have been reenabled if any. I am assuming cine.chris is using the donor and team lists that can be downloaded from links on this page - https://foldingathome.org/statistics/, any others are not guaranteed to be always available. The page itself was not updated to reflect the change from every hour to every 3 hours last year, if they have returned to generating those files more frequently, that will show up in the timestamps.
Timestamp of the Team List I just downloaded is Sun Feb 28 16:54:58 GMT 2021, just under an hour old.
Edit: and just downloaded the team list again, its time stamp is about an hour later than the previous, so it appears they have gone back to hourly updates. No idea when the update starts.
iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
-
- Posts: 471
- Joined: Mon Dec 03, 2007 6:20 am
- Location: Amsterdam
- Contact:
Re: STAT updates have stalled.
The update frequency of the stats files not particularly important, clients should use the If-Modified-Since to only download the files if they were modified on the server. That can be executed every hour with the actual processing of the data kicking in after there was something to download.
The problem was 500 Internal Server Error responses for something that should be a flat file.
The problem was 500 Internal Server Error responses for something that should be a flat file.
-
- Site Admin
- Posts: 7926
- Joined: Tue Apr 21, 2009 4:41 pm
- Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2 - Location: W. MA
Re: STAT updates have stalled.
Yes, that was fixed yesterday by the time of the EOC 3 PM CST update. My response was to the bits about EOC and stats updating frequency.smoking2000 wrote:The problem was 500 Internal Server Error responses for something that should be a flat file.
iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
-
- Posts: 140
- Joined: Fri Jan 21, 2011 8:02 am
Re: STAT updates have stalled.
EOC updates now at 32 mins past the hour every 3 hrs
was something like 15m after before
so other than it's an annoyance that the time changed... it's fine
just sort make people go ??? every time the time drifts on it
begs the question is it EOC fault or the server
was something like 15m after before
so other than it's an annoyance that the time changed... it's fine
just sort make people go ??? every time the time drifts on it
begs the question is it EOC fault or the server
-
- Site Admin
- Posts: 7926
- Joined: Tue Apr 21, 2009 4:41 pm
- Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2 - Location: W. MA
Re: STAT updates have stalled.
Just the number of entries EOC is needing to process from the flat files. The update still starts on the hour every 3 hours, it just is taking longer to process, especially since Jason increased the number of teams and users tracked.
iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
-
- Posts: 78
- Joined: Sun Apr 26, 2020 1:29 pm
Re: STAT updates have stalled.
Timestamps, since the restart, have drifted about 3 minutes, settling in at 52:25 initially, currently 55:09.
So, it appears, the drift of ~3.6min/day that I was seeing, is still occurring.
But, that is a different thread. The stall is fixed.
So, it appears, the drift of ~3.6min/day that I was seeing, is still occurring.
But, that is a different thread. The stall is fixed.