I7 - FAH crashing, potentially due to Linux kernel upgrade?

Moderators: Site Moderators, FAHC Science Team

MeeLee
Posts: 1339
Joined: Tue Feb 19, 2019 10:16 pm

Re: FAH keeps crashing, potentially due to Linux kernel upgr

Post by MeeLee »

Are you by any chance overclocking, or running your system too hot?
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: FAH keeps crashing, potentially due to Linux kernel upgr

Post by fangfufu »

MeeLee wrote:Are you by any chance overclocking, or running your system too hot?
The syslog doesn't show CPU being overheat. I have the same set up for the last 3 years or so, the problem has started occurring only recently.
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by bruce »

(running on two thread to prevent thermal throttling...)
What temperatures are you seeing?
Have you considered improving the heat dissipation subsystem?
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

bruce wrote:
(running on two thread to prevent thermal throttling...)
What temperatures are you seeing?
Have you considered improving the heat dissipation subsystem?
76 degrees Celcius. Haswell CPUs tend to be this hot, according to the Internet. I can't improve the cooling, because it is a laptop. :( I have used it like this for a few years now, the problem has only occurred after upgrading to Debian Buster.
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by bruce »

Has the fan failed?
Have you blown the dues out of the heat-sink passages?
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

bruce wrote:Has the fan failed?
Have you blown the dues out of the heat-sink passages?
The fan has not failed, I have cleared the dust on the heatsink recently. No other programs crash on my laptop, other than FAH

In the last 24 hours, it has been folding smoothly. I have stuck with to 3 mins checkpoint interval. That seems to have done the trick. My gut feeling is that the new kernel might have introduced some constraints on memory allocation, so if you do 30 mins checkpoint interval, it crashes when FAH tries to save the state. I already have to put "vsyscall=emulate" in order for FAH to run properly. (That did took me a while to figure out.)
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

okay, it seems rebooting the computer also causes frame checksum error. This is really weird...

I guess everyone thinks that there is something wrong with my CPU then?
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by bruce »

fangfufu wrote:I guess everyone thinks that there is something wrong with my CPU then?
Or your disk... or even some other component.

Run a good diagnostic suite.
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

bruce wrote:
fangfufu wrote:I guess everyone thinks that there is something wrong with my CPU then?
Or your disk... or even some other component.

Run a good diagnostic suite.
Do you have any suggestions for Linux? Sorry, I know this is probably not the right forum for asking this question, but FAH is literally the program that's crashing on my computer right now...
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
MeeLee
Posts: 1339
Joined: Tue Feb 19, 2019 10:16 pm

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by MeeLee »

Check your log, does it say anything about "is your system overclocked?"

Sometimes overclocks may run stable for months. Even on my GPUs I see this. Then out of nowhere, a once stable overclock now gives errors.
Could be because more sensitive WUs that are more prone to overclocking errors.
Or sometimes it could be because of temperature changes.
third possibility is the errors start happening due to wearing of the silicon; or more commonly, because the thermal paste needs replacement (though usually you'd see a rise in temperature of a few degrees compared to before).
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

For the record, the problem seems to have been resolved after upgrading my kernel to the one in Buster backports. I did a diagnostics test, nothing showed up, I proceeded to upgrade the kernel.

Code: Select all

fangfufu@smithsonian:~$ uname -a
Linux smithsonian 5.2.0-0.bpo.2-amd64 #1 SMP Debian 5.2.9-2~bpo10+1 (2019-08-25) x86_64 GNU/Linux
The problem occured after I upgraded the kernel from linux-image-4.19.0-5-amd64 to linux-image-4.19.0-6-amd64. That particular upgrade included a fix for the spectre bug

Code: Select all

  * [x86] ptrace: fix up botched merge of spectrev1 fix (CVE-2019-15902)
Problem started after that particular kernel upgrade - that's when my machine stopped successfully folding WUs.

I will no longer update this thread, if FAH continues working correctly on my machine.
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

Right, if anyone can figure out why linux-image-4.19.0-6-amd64 caused FAH to crash, that would be great. I think FAH binary needs to be updated. I am not a massive fan of "vsyscall=emulate". I just hope this kind of thing doesn't happen again. It was a great mystery.
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by bruce »

fangfufu
Posts: 93
Joined: Thu Jan 01, 2009 3:26 am
Hardware configuration: 4 cores on a Intel(R) Core(TM) i7-8700K
Location: Cambridge, United Kingdom
Contact:

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by fangfufu »

Yes, I know that. I am just saying that without that kernel command, FAH wouldn't run on standard Debian. Not that it matters anyway - if a user is running Debian, that user probably should know this kind of thing anyway.

Anyways, I don't think that's what caused the problems in the other kernel. What annoys me is that everyone said I had hardware problem, it turned out to be some weird kernel issue all along. I don't know if I should place the blame on Debian or FAH. The latest Matlab wouldn't run on that kernel for some reason, it just kept crashing.
Folding with 4 cores on Intel(R) Core(TM) i7-8700K

I first started folding back in the Google Compute days!
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: I7 - FAH crashing, potentially due to Linux kernel upgra

Post by bruce »

The function gettimeofday and other syscalls like it are used by FAH (and presumably by Matlab). Nobody is going to eliminate them from their code just because your distro cannot support them.
Post Reply