Not getting credit

[msultan]

We are currently running a manual stat update, it should take a few hours since A LOT of WUs are currently un-accredited. I am sorry this wasnt caught earlier on.

[msultan]

The stats should now start to get updated. i am still crediting the missing WUs but they should hopefully be done by the end of today. I am doing them in batches to figure out exactly why the system is breaking. Please let us know of further issues.

[drougnor]

Values did increment in the 2pm Eastern Update, so we'll see things getting back to normal on the 3rd party sites over the next few updates.

Thank you for the hard work you all do!

d

[ChristianVirtual]

I got a file and push was send as scheduled ... thanks for the work msultan

[bruce]

... over the next few updates.

Thank you for the hard work you all do!

Let me emphasize those words.

1) You'll be seeing incomplete updates until this is completely resolved. [I don't want a long list of folks saying "I completed WU xx (xx,xx,xx) but it wasn't credited."]

2) Many, many thanks to msultan and his team of FAH members who are supporting him.

[msultan]

Thanks a lot for the support everyone. This is the least we can do given that all of you are generously donating compute time for our calculations. I think I have the root cause figured out but it will likely require taking stats offline for more than a few hours. Please let me know if there are more issues. Again, I am really sorry about the problems that this has been causing recently.

[QuintLeo]

And the Standford stats server is borked AGAIN - seems like they "fix them" then they last perhaps a few hours and go down/fail AGAIN.

IMO Stanford should farm out their stats to an outside third party that actually CARES about keeping the stats infrastructure working, the last month has been totally UNACCEPTABLE on downtime to the point it's driving some folks OUT OF FOLDING (and has ME very close to that point).

[QuintLeo]

The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.

[drougnor]

I'm hoping that the current outage is you having taken them down for the fix you mentioned. If not, they've been down again since the 5pm Eastern update.

d

[SombraGuerrero]

I will be interested in a high level root cause analysis. As a QA analyst and coder, curiosity abounds.

[drougnor]

The problem is on Stanford's end - the server(s) that generate the "flat file" they REQUIRE third party stat sites to use has been borked more of the time this past week than not, and quite a bit of the time for the last month.
The issue has been going on so long it's starting to tick some folks off enough to QUIT FOLDING ENTIRELY - and it's got *ME* very close to that point, given that Standford has made it obvious through their INACTION that they don't care about the issue.

Except that you are responding directly after they HAVE taken action, potentially diagnosed the deeper issue and are potentially IN THE MIDDLE Of fixing it. Please don't spill your sour grapes all over the rest of us who are actually rooting for this to be fixed and who are doing our best to give useful information to assist IN that fixing.

Thanks.

d

[SteveWillis]

It appears to be down again.

[drougnor]

We got an update at 5pm EDT, missed one at 6pm EDT, picked up another at 7pm EDT and my system isn't downloading for the current hour, so it still see's the 7pm timestamp. They may still actively be working on things on the server, trying to flush the bug. We're likely to see more hinkyness before they get it sorted completely.

d

[bruce]

Yes, they are actively working on the problem.

The best assumption that seems to fit all the know facts, is that the server has been having hardware problems that appear as unexpected crashes. The hardware is old and does need to be replaced but it's essential that the stats for all WUs completed during these crashes are successfully placed in an updated copy of the stats db -- and that takes some extraordinary efforts.

There's already a plan to move the databasae and it's associated processes to new hardware -- and that will necessitate some planned downtime -- together with careful management of the data associated with the updates that should be occurring during the downtime so they can be re-added on the relocated database. Once all that has been accomplished, the automatic updates should resume. In the meantime, there's a lot of manual operations that need to be processed, checked, and rechecked.

I plead for patience on your part. I know the Pande Group takes this problem very seriously and they're doing everything they can to reach a permanent solution -- without a permanent loss of any data.

[drougnor]

Each error I've seen on my side for the last few hours have been DIFFERENT errors, so that's definitely a sign of progress being made instead of the server just sitting down and playing a few games of sol.exe on a non windows box. I'm definitely happy to see the progress.

However, it's getting late on the east coast so I'll likely have to call it a night soon. I'll happily check on things and provide a quick update on how things appear on my side of the system in the AM.

Thanks again, guys and gals!

d