Problem with the new WHQL R310.70 driver on GTX460s

[art_l_j_PlanetAMD64]

NVidia's update service notified me of the new WHQL R310.70 driver being available for download, so I downloaded and installed it on my 3 machines here.

I installed it on one system at a time, and verified that it worked OK on that system, before proceeding to the next system. I hoped that this would "catch" any problem before I got to updating my "top" system (#3).

BIG mistake, which I will describe below.

First of all, my 3 systems are:
1) WinXP Home SP3 32-bit, AMD Athlon 64 X2 4200+ (2.2GHz), GTX275/896MB/0x11 core, gets about 8000ppd
2) Win7 Pro SP1 32-bit, AMD Phenom II X4 965 (3.4 GHz), GTX285/1024MB/0x11 core, gets about 12000ppd
3) Win7 Home SP1 64-bit, AMD Phenom II X4 955 (3.2 GHz), dual GTX460/768MB/0x15 core, gets about 42000ppd

The OSs on all 3 systems are completely up-to-date, as of Tuesday last week (Dec 11 2012), "Microsoft Patch Tuesday".
All systems are using FAH v7.2.9.
Both the #2 and the #3 systems use the same motherboard, ASUS M4A79T Deluxe.
The #3 system is a new install last week (disk formatted, then Windows 7 installed).

Edit: The appropriate driver was downloaded/installed individually for each system:
1) WinXP, 32-bit
2) Win7, 32-bit
3) Win7, 64-bit

When updating each system, I clicked 'Quit' on FAHControl, so the FAH clients were stopped.
First I updated the #1 system, it worked OK and Folding continued from where it had left off.
Then I updated the #2 system, it worked OK and Folding continued from where it had left off.

When I updated the #3 system, the displays (LG W2343T at 1920x1080 on each GTX460) showed the background image as a small rectangle instead of full screen, all of the desktop icons had disappeared, and right-clicking on the desktop failed to show the menu (the one with 'Screen resolution', 'Gadgets', and 'Personalize' at the bottom).

Seeing this problem, I went back to the previous WHQL driver (306.97) on my #3 system, and everything appeared to be OK (the screen went back to how it should appear, the desktop icons were back, and right-clicking on the desktop produced the usual menu).

I started up FAHControl, and one GPU was Folding OK (TPF 11 minutes), but the other had drastically slowed down (TPF 60 minutes). Both GPUs had the same TPF (11 minutes) on their 762x WUs before the upgrade. Checking both GPUs using EVGA Precision X v3.0.3 showed both of them running at the same clock speeds (675/1800/3600).

Thinking that a reboot might fix the problem, I shutdown the system, unplugged the power cable for 20 seconds, then reconnected the power and started up the system. Same problem, no change.

I then used FAHControl to delete and add back both GPUs (losing the partially completed 762x WUs in the process). Same problem with the new (807x) WUs, no change (ie one GPU had almost 6 times the TPF as the other).

Then I uninstalled all of the following:
- NVIDIA 3D Vision Controller Driver
- NVIDIA 3D Vision Driver
- NVIDIA PhysX System Software
- FAHClient (losing the partially completed smp 0xa3 and both GPU 0x15 WUs)

Then I shutdown the system, unplugged the power cable for 20 seconds, then reconnected the power and started up the system.

Then I reinstalled FAHClient (FAHControl) v7.2.9, started it up, and everything is working OK now (ie both GPUs have the same TPF on 807x WUs).

The #3 system is now working on the following WUs:
1) smp:4 0xa3 PRCG 11060 (0, 2692, 34), TPF 5:28, Work Server 171.64.65.55
2) gpu0 0x15 PRCG 8074 (13, 12, 27), TPF 3:49, Work Server 171.67.108.36
3) gpu1 0x15 PRCG 8072 (0, 1262, 19), TPF 3:47, Work Server 171.67.108.36

I don't know if you can identify and re-issue the partially-completed WUs that got dumped during this process, rather than waiting for them to time out. My username/team is:
art_l_j_PlanetAMD64 45862

If it's not possible to find and re-issue these WUs prior to them timing out, then I sincerely apologize.

[art_l_j_PlanetAMD64]

NOTE:
I have posted all of the above information on the following NVidia forum, so they are aware of it:
"Official NVIDIA R310.70 WHQL Candidate Display Driver Feedback Thread (Released 12/4/12)"

My message is on page 22 of that forum, my username there is art_l_j:
link

[mmonnin]

I never update drivers/software unless there is an absolute need to or beneficial gain. It's working as is and if there's no benefit, only bad can happen. Instead of letting your other machines be guinea pigs, let others do that. Too late in this case, but something to consider in the future.

Why did you delete both WUs when only one was a problem? And did you give the WU several frames to recover its normal TPF? The PPD/TPF is horribly inaccurate in FAHControl. Something like HFM can average out TPFs when something else grabs cycles or a reboot in your case.

[art_l_j_PlanetAMD64]

I never update drivers/software unless there is an absolute need to or beneficial gain. It's working as is and if there's no benefit, only bad can happen. Instead of letting your other machines be guinea pigs, let others do that. Too late in this case, but something to consider in the future.

Why did you delete both WUs when only one was a problem? And did you give the WU several frames to recover its normal TPF? The PPD/TPF is horribly inaccurate in FAHControl. Something like HFM can average out TPFs when something else grabs cycles or a reboot in your case.

Thanks for your response, mmonnin.

I don't usually install upgrades like this, but this one claimed to be WHQL Certified, and NVidia also claimed that it contained several bug fixes and speed improvements.

I have installed NVidia's "WHQL Certified" drivers frequently in the past, all the way from 195.62 to 306.97, on dozens of different computer systems, and this is the first time there has ever been a failure. I do not install "beta" or other experimental drivers, only "WHQL Certified" drivers.

It (WHQL 310.70) also installed and ran perfectly on the first 2 systems, it was just the third system where I ran into problems.

Also:
WHQL Certified NVIDIA GeForce 310.70 Driver Is Ready for Download
NVIDIA 310.70 WHQL GeForce Drivers Improve Performance by up to 37%

NVidia's description of this release: link

NVIDIA released new GeForce R310 drivers today that are Microsoft Windows certified and optimized for this year’s biggest holiday games.

GeForce 310.70 drivers are now certified by the Microsoft Windows Hardware Quality Labs (WHQL) and are available for download from GeForce.com.

Deleting and adding back the GPU slots has been mentioned as a fix in several other threads, so that is why I tried it here.

When that failed, I tried another thing that has been mentioned in other threads, which is to uninstall and reinstall FAHClient. Of course, that has the unfortunate side effect of "dumping" the WUs that are currently being worked on.

In any case, I have now learned that NVidia can be just as bad as ATI usually is, on putting out screwed-up "WHQL Certified" drivers.

Bad for me (and FAH, if the dumped WUs have to wait for the timeout to be re-issued), but at least I can warn others here at the Folding Forum about the problems with this supposedly "WHQL Certified" driver.

The school of hard knocks is unpleasant, but effective, especially at my age (59).

[bruce]

In fact, testing the drivers on three separate systems sounds like a cautious approach to new drivers, but it turns out that it wasn't. You install three entirely different drivers: XP-32, Win7-32 and Win7-64. As you discovered, any one of them can be defective without the others being bad.

As has already been said, the reason for upgrading the drivers when FAH was running fine is always questionable. As NVidia said, the speed improvements were "optimized for this year’s biggest holiday games." The most common reason for driver upgrades are things that help specific games.

If it's not possible to find and re-issue these WUs prior to them timing out, then I sincerely apologize.

While it is possible, it's rarely done. It's a manual process and the project owner has to make a personal decision if it's worth allocating a large enough block of his time.

[art_l_j_PlanetAMD64]

In fact, testing the drivers on three separate systems sounds like a cautious approach to new drivers, but it turns out that it wasn't. You install three entirely different drivers: XP-32, Win7-32 and Win7-64. As you discovered, any one of them can be defective without the others being bad.

As has already been said, the reason for upgrading the drivers when FAH was running fine is always questionable. As NVidia said, the speed improvements were "optimized for this year’s biggest holiday games." The most common reason for driver upgrades are things that help specific games.

If it's not possible to find and re-issue these WUs prior to them timing out, then I sincerely apologize.

While it is possible, it's rarely done. It's a manual process and the project owner has to make a personal decision if it's worth allocating a large enough block of his time.

Fair enough, even though most of the code is likely to be "common", there obviously is (as you said) enough of a difference that NVidia was able to screw it up.

Equally obviously, I had become complacent (absolutely my fault, no question or argument), having never had a problem with several hundred total upgrades (number of systems x upgrades per system) like this in the past.

The solution is straightforward, on any future driver (or similar) upgrade I will:
1) Click the 'Finish' button on the FAHControl panel.
2) Wait for all smp and GPU clients to complete and send their results to the Work Server or Collection Server.
3) Click the 'Quit' button on the FAHControl panel to stop/exit all clients.
4) Do the upgrade, see it it works, and (if necessary) go back to the last "known good" driver.
5) Start up the FAHClient (FAHControl) so all smp and GPU clients will download and start new Work Units.

The only "silver lining" to this cloud is that by making and catching this mistake, I was able to document and warn other Folders here, not to make the same mistake that I did.

[art_l_j_PlanetAMD64]

.
Important Note:

By default, the "NVIDIA Control Panel" checks for driver updates once per day, and issues a popup when an updated NVidia "WHQL Certified" driver is available. As noted above, despite the supposed "Microsoft WHQL Certification", the new driver could be defective and would crash your system if it is installed.

To turn off notifications about updated NVidia "WHQL Certified" drivers, please do the following:
1) Right-click on the "NVIDIA Settings" icon in the taskbar, then left-click on the "NVIDIA Control Panel" menu item.
2) In the menu bar at the top, click on 'Help -> Updates'.
3) In the 'NVIDIA Update' window, click on the 'Preferences' tab, then click to clear the checkbox beside 'Automatically check for updates'.
4) At the bottom-right of this 'NVIDIA Update' window, click on 'Apply', then click on 'OK'.
5) Click on the top-right 'X' to exit the NVIDIA Control Panel.

[JimF]

I have been running 310.70 (initially the beta, but now the WHQL) on a GTX 660 under Win7 64-bit for a couple of days now with no problems (27,673 PPD on P7623). I first ran a single GTX 660 on a Core 2 Duo PC and now am running two of them on a Core i7-3770 machine.

But very importantly, I do a clean install, especially when two cards are involved; that always causes trouble in one form or another. First, remove the old drivers from the Control Panel/Add Remove Programs, rebooting as necessary, and then clean out the rest of the registry entries with Driver Fusion. And I never trust the Nvidia updater, and don't even install it when installing the new drivers. (In fact, I don't install the audio drivers, PhysX and whatever all that other stuff is either; I only install the graphics drivers, but you may need them and they probably don't cause problems by themselves, they just make the updates a little more complicated than necessary.)

[art_l_j_PlanetAMD64]

I have been running 310.70 (initially the beta, but now the WHQL) on a GTX 660 under Win7 64-bit for a couple of days now with no problems (27,673 PPD on P7623). I first ran a single GTX 660 on a Core 2 Duo PC and now am running two of them on a Core i7-3770 machine.

But very importantly, I do a clean install, especially when two cards are involved; that always causes trouble in one form or another. First, remove the old drivers from the Control Panel/Add Remove Programs, rebooting as necessary, and then clean out the rest of the registry entries with Driver Fusion. And I never trust the Nvidia updater, and don't even install it when installing the new drivers. (In fact, I don't install the audio drivers, PhysX and whatever all that other stuff is either; I only install the graphics drivers, but you may need them and they probably don't cause problems by themselves, they just make the updates a little more complicated than necessary.)

I guess it's a matter of interpretation. In my opinion, if running the 'default' install causes a problem, then the code is defective. Whether it's the fault of the driver or the fault of the installation code, it's a defective product. Nobody should have to jump through hoops just to update a simple video driver. None of the updates from 195.62 through 306.97 inclusive required any intervention whatsoever, on any system, with any number of GPUs. None.

As I said above, this is the first time out of hundreds of installations of video driver updates, that I have had a "WHQL" driver failure.

And it's not just me, on this NVidia forum, there are many users with horror stories to tell about the 310.70 "WHQL" driver:
Official NVIDIA R310.70 WHQL Candidate Display Driver Feedback Thread

24 pages in the forum (so far), and every page that I've looked at has users with a whole litany of failures of this driver (310.70). Many of them have only a single GPU.

Some people say it works OK in their system. It works OK in my WinXP-32 and Win7-32 systems.

But there have been enough people reporting serious problems, that I cannot give the 310.70 driver update a clean bill of health. From my experience and the experiences of others, it is completely unpredictable beforehand, whether an installation will work in a particular system. To me, that is completely unacceptable.

[JimF]

I have had problems with Nvidia WHQL drivers before, and reported them on this forum. I think you have been lucky. And whether it is WHQL or not is largely irrelevant to how it performs on Folding; Nvidia just submits them to Microsoft, which does a battery of tests to make sure they don't crash their system, or something like that. It does not guarantee that they will perform their intended function. So if you have found a driver that causes problems, avoid it, but I wouldn't stretch the point beyond that.

[art_l_j_PlanetAMD64]

I have had problems with Nvidia WHQL drivers before, and reported them on this forum. I think you have been lucky. And whether it is WHQL or not is largely irrelevant to how it performs on Folding; Nvidia just submits them to Microsoft, which does a battery of tests to make sure they don't crash their system, or something like that. It does not guarantee that they will perform their intended function. So if you have found a driver that causes problems, avoid it, but I wouldn't stretch the point beyond that.

Let's just agree to disagree.

Certainly, in the past (which I failed to do this time, my mistake) I would wait for some feedback that the "new" driver was stable and had a track record of reliability before doing the upgrade. So, I wasn't just downloading and installing every upgrade that came along. I chose only those that had been "proven" by others, so-to-speak.

As I said, many people, not just me, had serious problems with the 310.70 driver. And going back to 306.97 fixed the problem for the people (myself included) who wrote in to the NVidia forum.

[JimF]

I wasn't suggesting you didn't have a real problem, only that it did not apply to all cards, and an installation problem could be the culprit. But each case is different, as the Nvidia forums show. What fixes a problem for one person, causes another problem for someone else. I am glad that you found a good driver in any case.

[bruce]

Was this a certified driver or a candidate for certification?

[GreyWhiskers]

Here's what's on the Nvidia Driver page (lookup for my GTX560M laptop) right now - 6:16 pm PST Dec 18 2012. Lots of Beta versions, 310.70 released as WHQL yesterday.

Interesting to see that 310.70 was released as a BETA on Dec 4th, and the same version as WHQL on Dec 17.

[EDIT] The automatic lookup for my device came up with the 310.70- which they only do for the "certified" WHQL releases.

GeForce 310.70 Driver - BETA Version: 310.70 - Release Date: Tue Dec 04, 2012
GeForce 310.70 Driver - WHQL Version: 310.70 - Release Date: Mon Dec 17, 2012
GeForce 310.64 Driver - BETA Version: 310.64 - Release Date: Wed Nov 28, 2012
GeForce 310.61 Driver - BETA Version: 310.61 - Release Date: Tue Nov 20, 2012
GeForce 310.54 Driver - BETA Version: 310.54 - Release Date: Mon Nov 12, 2012
GeForce 310.33 Driver - BETA Version: 310.33 - Release Date: Tue Oct 23, 2012
GeForce 306.97 Driver - WHQL Version: 306.97 - Release Date: Wed Oct 10, 2012
GeForce 306.23 Driver - WHQL Version: 306.23 - Release Date: Thu Sep 13, 2012
GeForce 306.02 Driver - BETA Version: 306.02 - Release Date: Mon Aug 27, 2012
Verde 304.79 Driver - BETA Version: 304.79 - Release Date: Tue Jul 03, 2012

[art_l_j_PlanetAMD64]

Was this a certified driver or a candidate for certification?

NVidia says it is a certified driver: link

By Chris Daniel on Dec 17 2012

GeForce 310.70 drivers are now certified by the Microsoft Windows Hardware Quality Labs (WHQL) and are available for download from GeForce.com.

It was only the title of the NVidia forum that referred to a candidate for certification:
Official NVIDIA R310.70 WHQL Candidate Display Driver Feedback Thread (Released 12/4/12)

The forum was created on December 4, when the driver was only a candidate for certification. The actual certified driver was released yesterday, December 17.