Stats server is down, [and fixed] again

[Kent Irwin]

Thread title says it all.

[sortofageek]

What are you seeing that tells you stats are down? I just looked around at my account, a few team members and the team itself. It looked ok to me.

[Joe_H]

I have seen some slow connections to the fah-web.stanford.edu sites today, and looking at my team stats on EOC it appears an update may not have been done in time for the 3:00 load. But the 6:00 one is there on EOC. SO if there was a problem, it may have been just a temporary one.

[sco01]

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.

Apache/2.0.52 (CentOS) Server at fah-web.stanford.edu Port 80

[billford]

Same here.

As far as I can see it was OK until the update run at 01:00 (UTC), then something went pear-shaped.

[Kent Irwin]

Thread title says it all.

Stats server is down, again [yes]
It's down again. It was down for a short period when I originally posted. It is a chronic problem.

[Skufer]

I can confirm that I am also seeing an error 500 Internal Server Error on the stats page

[toTOW]

Yes it's down again, no data on 3 tast EOC updates

[drougnor]

My stats collector hasn't seen an update since the last one at 9pm EDT.

[Joe_H]

I have sent emails to the appropriate PG members. The problems appear to be only with fah-web.stanford.edu which accesses and reports the stats, as best as I can tell the database server itself is not affected.

[VijayPande]

Yes, the db is on a separate server and is isolated from these issues. This is related to the NaCl issue I posted about yesterday.

Short term, this should be fixed on Monday. Longer term, as mentioned in that post, we need to migrate out some services onto a new server to take care of this. We have those servers ready so it’s about migrating now.

[drougnor]

*lame joke warning* That's what you get for putting table salt in your servers!

Lame joking aside, thanks for the info, sir. I know there's nothing we can do but wait, but we at EVGA had our overtake of the Default team estimated to be in the next 12-18 hours or so and are a little disappointed not to be able to track the 'official' time of overtake. However, it is what it is and we gladly continue to Fold for the cause!

d

[beylie]

So this is why I've been seeing no stats for the last few days! Please tell me the work done isn't lost, but will show up when the stats servers are up again?

[cxh]

@beylie: No need to worry! Work is being returned and points are still being updated in our internal database. It's just the front-end that's not working at the moment.

[beylie]

Excellent! Thank you!