I am having rampant failures with the newest NVIDIA drivers

[Turbo_T]

I did have another GTX 260 in the second slot previously and it folded fine which makes me suspect the slot is fine. I took it out to get the 480 in and be able to run SLI for gaming as well as fold more with the 480. I may try that test, but it is a pain as I have to drain the water cooling loop and re-plumb all the cards each time I change the configuration at all. I do have a third PCIe slot on the board but it will operate at 8X only by design. I will have to consider that as an option as well. do you have any suggestions on how to under clock the card? the EVGA software doesn't allow any lower clocks in the Precision X application for this card.

[folding_hoomer]

Try MSi Afterburner - it is more flexible/universal and works with most modern GPU´s (AMD and nVidia).

[Turbo_T]

Still having issues with the one card, this set is different than the others, not sure why, I have made no changes other than trying to run this solo.

Code: Select all

*********************** Log Started 2014-03-16T22:51:46Z ***********************
*********************** Log Started 2014-03-16T22:51:46Z ***********************
01:44:41:WARNING:Exception: 20:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
*********************** Log Started 2014-03-16T22:51:46Z ***********************
01:44:41:WARNING:Exception: 20:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
01:56:53:WU01:FS01:0x17:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-36)
01:56:56:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:33:26:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:34:32:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
03:38:52:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:39:58:WARNING:WU01:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
03:44:17:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:45:24:WARNING:WU01:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
03:49:43:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:50:49:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
03:55:10:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:55:10:WARNING:WU01:FS01:Too many errors, failing
03:55:14:WARNING:WU01:FS01:Server did not like results, dumping
04:00:42:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:01:48:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:06:12:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:07:18:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:11:40:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:12:46:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:17:06:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:18:13:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:22:33:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:22:33:WARNING:WU03:FS01:Too many errors, failing
04:22:37:WARNING:WU03:FS01:Server did not like results, dumping
04:28:03:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:29:10:WARNING:WU01:FS01:Detected clock skew (1 mins 07 secs), adjusting time estimates
04:33:29:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:34:35:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:38:54:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:40:00:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:44:21:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:45:27:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:49:47:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:49:47:WARNING:WU01:FS01:Too many errors, failing
04:49:48:WARNING:WU01:FS01:Server did not like results, dumping
******************************* Date: 2014-03-17 *******************************
04:55:15:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:56:21:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:00:42:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:01:48:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:06:07:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:07:14:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:11:33:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:12:39:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:17:02:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:17:02:WARNING:WU03:FS01:Too many errors, failing
05:17:02:WARNING:WU03:FS01:Server did not like results, dumping
01:44:41:WARNING:Exception: 20:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
01:56:53:WU01:FS01:0x17:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-36)
01:56:56:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:33:26:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:34:32:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
03:38:52:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:39:58:WARNING:WU01:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
03:44:17:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:45:24:WARNING:WU01:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
03:49:43:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:50:49:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
03:55:10:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
03:55:10:WARNING:WU01:FS01:Too many errors, failing
03:55:14:WARNING:WU01:FS01:Server did not like results, dumping
04:00:42:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:01:48:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:06:12:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:07:18:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:11:40:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:12:46:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:17:06:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:18:13:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
04:22:33:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:22:33:WARNING:WU03:FS01:Too many errors, failing
04:22:37:WARNING:WU03:FS01:Server did not like results, dumping
04:28:03:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:29:10:WARNING:WU01:FS01:Detected clock skew (1 mins 07 secs), adjusting time estimates
04:33:29:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:34:35:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:38:54:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:40:00:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:44:21:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:45:27:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:49:47:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:49:47:WARNING:WU01:FS01:Too many errors, failing
04:49:48:WARNING:WU01:FS01:Server did not like results, dumping
******************************* Date: 2014-03-17 *******************************
04:55:15:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:56:21:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:00:42:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:01:48:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:06:07:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:07:14:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:11:33:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:12:39:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:17:02:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:17:02:WARNING:WU03:FS01:Too many errors, failing
05:17:02:WARNING:WU03:FS01:Server did not like results, dumping
04:40:00:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:44:21:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:45:27:WARNING:WU01:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
04:49:47:WARNING:WU01:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:49:47:WARNING:WU01:FS01:Too many errors, failing
04:49:48:WARNING:WU01:FS01:Server did not like results, dumping
******************************* Date: 2014-03-17 *******************************
04:55:15:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
04:56:21:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:00:42:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:01:48:WARNING:WU03:FS01:Detected clock skew (1 mins 05 secs), adjusting time estimates
05:06:07:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:07:14:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:11:33:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:12:39:WARNING:WU03:FS01:Detected clock skew (1 mins 06 secs), adjusting time estimates
05:17:02:WARNING:WU03:FS01:FahCore returned: UNSTABLE_MACHINE (122 = 0x7a)
05:17:02:WARNING:WU03:FS01:Too many errors, failing
05:17:02:WARNING:WU03:FS01:Server did not like results, dumping
******************************* Date: 2014-03-17 *******************************
******************************* Date: 2014-03-17 *******************************
******************************* Date: 2014-03-17 *******************************

Mod edit: added code tags

[Turbo_T]

am not sure the cause or remedy for the clock skew? nothing on the computer is overclocked and the card is at stock settings. I am going to try to install the MSI afterburner software and see if that will help but theses are EVGA cards and I am not sure it will work.

[P5-133XL]

Clock skew is merely an adjustment for PPD calculations. Potential causes are a misalignment of your local computers clock to the server clocks or merely your machine was paused or went to sleep for some time. It is just a notification and will not affect processing in any way.

Unstable machine errors are much more serious because that typically means a dumped WU or at a minimum reverting to the last known good checkpoint

[bruce]

The "clock skew" message means that client was suspended and later resumed. That can happen if your computer enters sleep/hibernate (a process which is intentionally disabled by FAH) or because you rebooted. or you intentionally stopped the client. All of those events cause a certain amount of processing to be repeated so folding progress is impeded by whatever is happening.

If you wish to pause your GPU slot to play a game, you can leave the client running and you won't get clock skew messages, so that's not what you're doing.

Why is this happening so often? The UNSTABLE_MACHINE messages suggest that a GPU_RESET is being sent to your GPU frequently by the OS. This happens if your GPU enters Sleep or you make a connection using MS Remote Desktop our if your GPU hangs (perhaps due to overclocking or because you actually have a defective GPU (or probably several other possibilites). The GPU_RESET is the OS's attempt to get the GPU working again after it has stopped and it will crash the active GPU WU. After a WU crashes several times, it is dumped.

Once is probably a bad WU. Repeatedly, and there's SOMETHING wrong with the hardware or what you're doing with it.

This log does not show which WUs were dumped so I can't do any additional checking unless you post other portions of the log. NOTE: CPU-based WUs can survive a sleep/awaken. GPU WUs cannot.

[Turbo_T]

It's continuing to fail with the two GPU's running together, even after turning the clocks down to 730 Mhz gpu and 1840Mhz Mem , (752/1900 stock).
One runs fine but the GPU % loads have a sawtooth pattern,Varying from 99-95%instead of the smooth 99% load indication I get for other WU's. If I turn on the second it has a severe sawtooth load pattern that drops to mid 70% then back to 99% every couple seconds. Log excerpt below:

Code: Select all

*********************** Log Started 2014-03-24T04:40:02Z ***********************
04:40:02:************************* Folding@home Client *************************
04:40:02:      Website: http://folding.stanford.edu/
04:40:02:    Copyright: (c) 2009-2013 Stanford University
04:40:02:       Author: Joseph Coffland <[email protected]>
04:40:02:         Args: --open-web-control
04:40:02:       Config: E:/Stanford FAH/config.xml
04:40:02:******************************** Build ********************************
04:40:02:      Version: 7.3.6
04:40:02:         Date: Feb 18 2013
04:40:02:         Time: 15:25:17
04:40:02:      SVN Rev: 3923
04:40:02:       Branch: fah/trunk/client
04:40:02:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
04:40:02:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
04:40:02:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
04:40:02:     Platform: win32 XP
04:40:02:         Bits: 32
04:40:02:         Mode: Release
04:40:02:******************************* System ********************************
04:40:02:          CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
04:40:02:       CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
04:40:02:         CPUs: 12
04:40:02:       Memory: 11.99GiB
04:40:02:  Free Memory: 10.19GiB
04:40:02:      Threads: WINDOWS_THREADS
04:40:02:  Has Battery: false
04:40:02:   On Battery: false
04:40:02:   UTC offset: -4
04:40:02:          PID: 2312
04:40:02:          CWD: E:/Stanford FAH
04:40:02:           OS: Windows 7 Professional
04:40:02:      OS Arch: AMD64
04:40:02:         GPUs: 2
04:40:02:        GPU 0: NVIDIA:2 GF100 [GeForce GTX 480]
04:40:02:        GPU 1: NVIDIA:2 GF100 [GeForce GTX 480]
04:40:02:         CUDA: 2.0
04:40:02:  CUDA Driver: 6000
04:40:02:Win32 Service: false
04:40:02:***********************************************************************
04:40:02:<config>
04:40:02:  <!-- Folding Core -->
04:40:02:  <checkpoint v='30'/>
04:40:02:
04:40:02:  <!-- Folding Slot Configuration -->
04:40:02:  <power v='full'/>
04:40:02:
04:40:02:  <!-- HTTP Server -->
04:40:02:  <allow v='127.0.0.1 192.168.1.100-192.168.1.125'/>
04:40:02:
04:40:02:  <!-- Network -->
04:40:02:  <proxy v=':8080'/>
04:40:02:
04:40:02:  <!-- Remote Command Server -->
04:40:02:  <command-allow-no-pass v='127.0.0.1 192.168.1.100-192.168.1.125'/>
04:40:02:  <password v=''/>
04:40:02:
04:40:02:  <!-- Slot Control -->
04:40:02:  <pause-on-battery v='false'/>
04:40:02:
04:40:02:  <!-- User Information -->
04:40:02:  <passkey v='********************************'/>
04:40:02:  <team v='111065'/>
04:40:02:  <user v='Turbo_T'/>
04:40:02:
04:40:02:  <!-- Folding Slots -->
04:40:02:  <slot id='2' type='CPU'>
04:40:02:    <cpus v='10'/>
04:40:02:  </slot>
04:40:02:  <slot id='0' type='GPU'/>
04:40:02:  <slot id='1' type='GPU'/>
04:40:02:</config>
04:40:02:Trying to access database...
04:40:02:Successfully acquired database lock
04:40:02:Enabled folding slot 02: READY cpu:10
04:40:02:Enabled folding slot 00: READY gpu:0:GF100 [GeForce GTX 480]
04:40:02:Enabled folding slot 01: READY gpu:1:GF100 [GeForce GTX 480]
04:40:02:WU00:FS02:Starting
04:40:02:WU00:FS02:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a3.fah/FahCore_a3.exe" -dir 00 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -np 10
04:40:02:WU00:FS02:Started FahCore on PID 4804
04:40:02:WU00:FS02:Core PID:1556
04:40:02:WU00:FS02:FahCore 0xa3 started
04:40:02:WU01:FS01:Starting
04:40:02:WU01:FS01:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 01 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 1 -gpu-vendor nvidia
04:40:02:WU01:FS01:Started FahCore on PID 3516
04:40:03:WU00:FS02:0xa3:
04:40:03:WU00:FS02:0xa3:*------------------------------*
04:40:03:WU00:FS02:0xa3:Folding@Home Gromacs SMP Core
04:40:03:WU00:FS02:0xa3:Version 2.27 (Dec. 15, 2010)
04:40:03:WU00:FS02:0xa3:
04:40:03:WU00:FS02:0xa3:Preparing to commence simulation
04:40:03:WU00:FS02:0xa3:- Looking at optimizations...
04:40:03:WU00:FS02:0xa3:- Files status OK
04:40:03:WU01:FS01:Core PID:6408
04:40:03:WU01:FS01:FahCore 0x17 started
04:40:03:WU03:FS00:Starting
04:40:03:WU03:FS00:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 03 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
04:40:03:WU03:FS00:Started FahCore on PID 6552
04:40:03:WU03:FS00:Core PID:6580
04:40:03:WU03:FS00:FahCore 0x17 started
04:40:03:WU00:FS02:0xa3:- Expanded 3848289 -> 4384444 (decompressed 113.9 percent)
04:40:03:WU00:FS02:0xa3:Called DecompressByteArray: compressed_data_size=3848289 data_size=4384444, decompressed_data_size=4384444 diff=0
04:40:03:WU00:FS02:0xa3:- Digital signature verified
04:40:03:WU00:FS02:0xa3:
04:40:03:WU00:FS02:0xa3:Project: 8576 (Run 1, Clone 2, Gen 292)
04:40:03:WU00:FS02:0xa3:
04:40:03:WU00:FS02:0xa3:Assembly optimizations on if available.
04:40:03:WU00:FS02:0xa3:Entering M.D.
04:40:04:WU03:FS00:0x17:*********************** Log Started 2014-03-24T04:40:04Z ***********************
04:40:04:WU03:FS00:0x17:Project: 13001 (Run 168, Clone 4, Gen 1)
04:40:04:WU03:FS00:0x17:Unit: 0x00000004538b3db753287f5d0cf92450
04:40:04:WU03:FS00:0x17:CPU: 0x00000000000000000000000000000000
04:40:04:WU03:FS00:0x17:Machine: 0
04:40:04:WU03:FS00:0x17:Digital signatures verified
04:40:04:WU03:FS00:0x17:Folding@home GPU core17
04:40:04:WU03:FS00:0x17:Version 0.0.52
04:40:04:WU01:FS01:0x17:*********************** Log Started 2014-03-24T04:40:04Z ***********************
04:40:04:WU01:FS01:0x17:Project: 13001 (Run 346, Clone 2, Gen 1)
04:40:04:WU01:FS01:0x17:Unit: 0x00000004538b3db75328b1cc5024be11
04:40:04:WU01:FS01:0x17:CPU: 0x00000000000000000000000000000000
04:40:04:WU01:FS01:0x17:Machine: 1
04:40:04:WU01:FS01:0x17:Digital signatures verified
04:40:04:WU01:FS01:0x17:Folding@home GPU core17
04:40:04:WU01:FS01:0x17:Version 0.0.52
04:40:04:WU01:FS01:0x17:  Found a checkpoint file
04:40:05:3:127.0.0.1:New Web connection
04:40:09:WU00:FS02:0xa3:Using Gromacs checkpoints
04:40:10:WU00:FS02:0xa3:Mapping NT from 10 to 10 
04:40:11:WU00:FS02:0xa3:Resuming from checkpoint
04:40:11:WARNING:Exception: 8:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
04:40:11:WU00:FS02:0xa3:Verified 00/wudata_01.log
04:40:12:WU00:FS02:0xa3:Verified 00/wudata_01.trr
04:40:12:WU00:FS02:0xa3:Verified 00/wudata_01.edr
04:40:13:WU00:FS02:0xa3:Completed 379880 out of 500000 steps  (75%)
04:40:34:WU00:FS02:0xa3:Completed 380000 out of 500000 steps  (76%)
04:45:01:WU03:FS00:0x17:Completed 0 out of 5000000 steps (0%)
04:45:01:WU03:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:45:38:WU01:FS01:0x17:Completed 3500000 out of 5000000 steps (70%)
04:45:38:WU01:FS01:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:51:40:WU00:FS02:0xa3:Completed 385000 out of 500000 steps  (77%)
05:01:45:WU00:FS02:0xa3:Completed 390000 out of 500000 steps  (78%)
05:05:56:WU01:FS01:0x17:Completed 3550000 out of 5000000 steps (71%)
05:06:33:WU03:FS00:0x17:ERROR:exception: First periodic box vector must be parallel to x.
05:06:33:WU03:FS00:0x17:Saving result file logfile_01.txt
05:06:33:WU03:FS00:0x17:Saving result file log.txt
05:06:33:WU03:FS00:0x17:WARNING:While cleaning up: Failed to remove directory '01': boost::filesystem::remove: The directory is not empty: "01"
05:06:33:WU03:FS00:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
05:06:33:WARNING:WU03:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:06:33:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13001 run:168 clone:4 gen:1 core:0x17 unit:0x00000004538b3db753287f5d0cf92450
05:06:34:WU03:FS00:Uploading 3.06KiB to 140.163.4.231
05:06:34:WU03:FS00:Connecting to 140.163.4.231:8080
05:06:34:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
05:06:34:WU03:FS00:Upload complete
05:06:34:WU03:FS00:Server responded WORK_ACK (400)
05:06:34:WU03:FS00:Cleaning up
05:06:35:WU02:FS00:News: Welcome to Folding@Home
05:06:35:WU02:FS00:Assigned to work server 140.163.4.231
05:06:35:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:GF100 [GeForce GTX 480] from 140.163.4.231
05:06:35:WU02:FS00:Connecting to 140.163.4.231:8080
05:06:35:WU02:FS00:Downloading 4.84MiB
05:06:36:WU02:FS00:Download complete
05:06:37:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13001 run:162 clone:3 gen:2 core:0x17 unit:0x00000008538b3db753287da3c6bc410a
05:06:37:WU02:FS00:Starting
05:06:37:WU02:FS00:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 02 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
05:06:37:WU02:FS00:Started FahCore on PID 5240
05:06:37:WU02:FS00:Core PID:5828
05:06:37:WU02:FS00:FahCore 0x17 started
05:06:37:WU02:FS00:0x17:*********************** Log Started 2014-03-24T05:06:37Z ***********************
05:06:37:WU02:FS00:0x17:Project: 13001 (Run 162, Clone 3, Gen 2)
05:06:37:WU02:FS00:0x17:Unit: 0x00000008538b3db753287da3c6bc410a
05:06:37:WU02:FS00:0x17:CPU: 0x00000000000000000000000000000000
05:06:37:WU02:FS00:0x17:Machine: 0
05:06:37:WU02:FS00:0x17:Reading tar file state.xml
05:06:38:WU02:FS00:0x17:Reading tar file system.xml
05:06:39:WU02:FS00:0x17:Reading tar file integrator.xml
05:06:39:WU02:FS00:0x17:Reading tar file core.xml
05:06:39:WU02:FS00:0x17:Digital signatures verified
05:06:39:WU02:FS00:0x17:Folding@home GPU core17
05:06:39:WU02:FS00:0x17:Version 0.0.52
05:11:29:WU02:FS00:0x17:Completed 0 out of 5000000 steps (0%)
05:11:29:WU02:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:13:09:WU00:FS02:0xa3:Completed 395000 out of 500000 steps  (79%)
05:23:11:WU00:FS02:0xa3:Completed 400000 out of 500000 steps  (80%)
05:25:53:WU01:FS01:0x17:Completed 3600000 out of 5000000 steps (72%)
05:33:04:WU00:FS02:0xa3:Completed 405000 out of 500000 steps  (81%)
05:33:34:WU02:FS00:0x17:ERROR:exception: The periodic box size has decreased to less than twice the nonbonded cutoff.
05:33:34:WU02:FS00:0x17:Saving result file logfile_01.txt
05:33:34:WU02:FS00:0x17:Saving result file log.txt
05:33:34:WU02:FS00:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
05:33:34:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:33:35:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13001 run:162 clone:3 gen:2 core:0x17 unit:0x00000008538b3db753287da3c6bc410a
05:33:35:WU02:FS00:Uploading 2.47KiB to 140.163.4.231
05:33:35:WU02:FS00:Connecting to 140.163.4.231:8080
05:33:35:WU02:FS00:Upload complete
05:33:35:WU02:FS00:Server responded WORK_ACK (400)
05:33:35:WU02:FS00:Cleaning up
05:33:35:WU03:FS00:Connecting to assign-GPU.stanford.edu:80
05:33:36:WU03:FS00:News: Welcome to Folding@Home
05:33:36:WU03:FS00:Assigned to work server 140.163.4.231
05:33:36:WU03:FS00:Requesting new work unit for slot 00: READY gpu:0:GF100 [GeForce GTX 480] from 140.163.4.231
05:33:36:WU03:FS00:Connecting to 140.163.4.231:8080
05:33:36:WU03:FS00:Downloading 4.84MiB
05:33:37:WU03:FS00:Download complete
05:33:37:WU03:FS00:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13001 run:137 clone:4 gen:1 core:0x17 unit:0x00000001538b3db75328768e35161f6f
05:33:38:WU03:FS00:Starting
05:33:38:WU03:FS00:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 03 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
05:33:38:WU03:FS00:Started FahCore on PID 3764
05:33:38:WU03:FS00:Core PID:5516
05:33:38:WU03:FS00:FahCore 0x17 started
05:33:38:WU03:FS00:0x17:*********************** Log Started 2014-03-24T05:33:38Z ***********************
05:33:38:WU03:FS00:0x17:Project: 13001 (Run 137, Clone 4, Gen 1)
05:33:38:WU03:FS00:0x17:Unit: 0x00000001538b3db75328768e35161f6f
05:33:38:WU03:FS00:0x17:CPU: 0x00000000000000000000000000000000
05:33:38:WU03:FS00:0x17:Machine: 0
05:33:38:WU03:FS00:0x17:Reading tar file state.xml
05:33:39:WU03:FS00:0x17:Reading tar file system.xml
05:33:40:WU03:FS00:0x17:Reading tar file integrator.xml
05:33:40:WU03:FS00:0x17:Reading tar file core.xml
05:33:40:WU03:FS00:0x17:Digital signatures verified
05:33:40:WU03:FS00:0x17:Folding@home GPU core17
05:33:40:WU03:FS00:0x17:Version 0.0.52
05:38:42:WU03:FS00:0x17:Completed 0 out of 5000000 steps (0%)
05:38:42:WU03:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:42:44:WU00:FS02:0xa3:Completed 410000 out of 500000 steps  (82%)
05:46:19:WU01:FS01:0x17:Completed 3650000 out of 5000000 steps (73%)
05:52:33:WU00:FS02:0xa3:Completed 415000 out of 500000 steps  (83%)
05:59:15:WU03:FS00:0x17:ERROR:exception: First periodic box vector must be parallel to x.
05:59:15:WU03:FS00:0x17:Saving result file logfile_01.txt
05:59:15:WU03:FS00:0x17:Saving result file log.txt
05:59:15:WU03:FS00:0x17:WARNING:While cleaning up: Failed to remove directory '01': boost::filesystem::remove: The directory is not empty: "01"
05:59:15:WU03:FS00:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
05:59:16:WARNING:WU03:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:59:16:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13001 run:137 clone:4 gen:1 core:0x17 unit:0x00000001538b3db75328768e35161f6f
05:59:16:WU03:FS00:Uploading 2.45KiB to 140.163.4.231
05:59:16:WU03:FS00:Connecting to 140.163.4.231:8080
05:59:16:WU02:FS00:Connecting to assign-GPU.stanford.edu:80
05:59:16:WU03:FS00:Upload complete
05:59:16:WU02:FS00:News: Welcome to Folding@Home
05:59:16:WU02:FS00:Assigned to work server 140.163.4.231
05:59:16:WU02:FS00:Requesting new work unit for slot 00: READY gpu:0:GF100 [GeForce GTX 480] from 140.163.4.231
05:59:16:WU02:FS00:Connecting to 140.163.4.231:8080
05:59:16:WU03:FS00:Server responded WORK_ACK (400)
05:59:17:WU03:FS00:Cleaning up
05:59:17:WU02:FS00:Downloading 4.84MiB
05:59:18:WU02:FS00:Download complete
05:59:18:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13001 run:80 clone:5 gen:2 core:0x17 unit:0x00000005538b3db7532866799aed55c4
05:59:19:WU02:FS00:Starting
05:59:19:WU02:FS00:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 02 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
05:59:19:WU02:FS00:Started FahCore on PID 6904
05:59:19:WU02:FS00:Core PID:5560
05:59:19:WU02:FS00:FahCore 0x17 started
05:59:19:WU02:FS00:0x17:*********************** Log Started 2014-03-24T05:59:19Z ***********************
05:59:19:WU02:FS00:0x17:Project: 13001 (Run 80, Clone 5, Gen 2)
05:59:19:WU02:FS00:0x17:Unit: 0x00000005538b3db7532866799aed55c4
05:59:19:WU02:FS00:0x17:CPU: 0x00000000000000000000000000000000
05:59:19:WU02:FS00:0x17:Machine: 0
05:59:19:WU02:FS00:0x17:Reading tar file state.xml
05:59:20:WU02:FS00:0x17:Reading tar file system.xml
05:59:21:WU02:FS00:0x17:Reading tar file integrator.xml
05:59:21:WU02:FS00:0x17:Reading tar file core.xml
05:59:21:WU02:FS00:0x17:Digital signatures verified
05:59:21:WU02:FS00:0x17:Folding@home GPU core17
05:59:21:WU02:FS00:0x17:Version 0.0.52
06:02:34:WU00:FS02:0xa3:Completed 420000 out of 500000 steps  (84%)
06:03:25:WU02:FS00:0x17:ERROR:exception: Discrepancy: Forces are blowing up!20 0
06:03:25:WU02:FS00:0x17:Saving result file logfile_01.txt
06:03:25:WU02:FS00:0x17:Saving result file log.txt
06:03:25:WU02:FS00:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
06:03:26:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:03:26:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:13001 run:80 clone:5 gen:2 core:0x17 unit:0x00000005538b3db7532866799aed55c4
06:03:26:WU02:FS00:Uploading 2.27KiB to 140.163.4.231
06:03:26:WU02:FS00:Connecting to 140.163.4.231:8080
06:03:26:WU02:FS00:Upload complete
06:03:26:WU02:FS00:Server responded WORK_ACK (400)
06:03:26:WU02:FS00:Cleaning up
06:03:26:WU03:FS00:Connecting to assign-GPU.stanford.edu:80
06:03:27:WU03:FS00:News: Welcome to Folding@Home
06:03:27:WU03:FS00:Assigned to work server 140.163.4.231
06:03:27:WU03:FS00:Requesting new work unit for slot 00: READY gpu:0:GF100 [GeForce GTX 480] from 140.163.4.231
06:03:27:WU03:FS00:Connecting to 140.163.4.231:8080
06:03:27:WU03:FS00:Downloading 5.22MiB
06:03:28:WU03:FS00:Download complete
06:03:28:WU03:FS00:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13001 run:412 clone:7 gen:0 core:0x17 unit:0x00000004538b3db75328c4abfc0f487c
06:03:28:WU03:FS00:Starting
06:03:28:WU03:FS00:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_17.fah/FahCore_17.exe" -dir 03 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
06:03:28:WU03:FS00:Started FahCore on PID 1752
06:03:28:WU03:FS00:Core PID:6828
06:03:28:WU03:FS00:FahCore 0x17 started
06:03:29:WU03:FS00:0x17:*********************** Log Started 2014-03-24T06:03:29Z ***********************
06:03:29:WU03:FS00:0x17:Project: 13001 (Run 412, Clone 7, Gen 0)
06:03:29:WU03:FS00:0x17:Unit: 0x00000004538b3db75328c4abfc0f487c
06:03:29:WU03:FS00:0x17:CPU: 0x00000000000000000000000000000000
06:03:29:WU03:FS00:0x17:Machine: 0
06:03:29:WU03:FS00:0x17:Reading tar file system.xml
06:03:30:WU03:FS00:0x17:Reading tar file integrator.xml
06:03:30:WU03:FS00:0x17:Reading tar file state.xml
06:03:31:WU03:FS00:0x17:Reading tar file core.xml
06:03:31:WU03:FS00:0x17:Digital signatures verified
06:03:31:WU03:FS00:0x17:Folding@home GPU core17
06:03:31:WU03:FS00:0x17:Version 0.0.52
06:06:19:WU01:FS01:0x17:Completed 3700000 out of 5000000 steps (74%)
06:08:16:WU03:FS00:0x17:Completed 0 out of 5000000 steps (0%)
06:08:16:WU03:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:12:52:WU00:FS02:0xa3:Completed 425000 out of 500000 steps  (85%)
06:22:38:WU00:FS02:0xa3:Completed 430000 out of 500000 steps  (86%)
06:23:45:WU03:FS00:0x17:ERROR:exception: First periodic box vector must be parallel to x.
06:23:45:WU03:FS00:0x17:Saving result file logfile_01.txt
06:23:45:WU03:FS00:0x17:Saving result file log.txt
06:23:45:WU03:FS00:0x17:Folding@home Core Shutdown: BAD_WORK_UNIT
06:23:45:WARNING:WU03:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:23:45:WU03:FS00:Sending unit results: id:03 state:SEND error:FAULTY project:13001 run:412 clone:7 gen:0 core:0x17 unit:0x00000004538b3db75328c4abfc0f487c
06:23:45:WU03:FS00:Uploading 2.45KiB to 140.163.4.231
06:23:45:WU03:FS00:Connecting to 140.163.4.231:8080
06:23:45:WU03:FS00:Upload complete
06:23:45:WU03:FS00:Server responded WORK_ACK (400)
06:23:45:WU03:FS00:Cleaning up
06:26:17:WU01:FS01:0x17:Completed 3750000 out of 5000000 steps (75%)
06:31:38:WU00:FS02:0xa3:Completed 435000 out of 500000 steps  (87%)
06:40:31:WU00:FS02:0xa3:Completed 440000 out of 500000 steps  (88%)
06:46:41:WU01:FS01:0x17:Completed 3800000 out of 5000000 steps (76%)
06:49:22:WU00:FS02:0xa3:Completed 445000 out of 500000 steps  (89%)
06:58:14:WU00:FS02:0xa3:Completed 450000 out of 500000 steps  (90%)
07:06:41:WU01:FS01:0x17:Completed 3850000 out of 5000000 steps (77%)
07:07:05:WU00:FS02:0xa3:Completed 455000 out of 500000 steps  (91%)
07:15:56:WU00:FS02:0xa3:Completed 460000 out of 500000 steps  (92%)
07:24:48:WU00:FS02:0xa3:Completed 465000 out of 500000 steps  (93%)
07:27:04:WU01:FS01:0x17:Completed 3900000 out of 5000000 steps (78%)
07:33:38:WU00:FS02:0xa3:Completed 470000 out of 500000 steps  (94%)
07:42:30:WU00:FS02:0xa3:Completed 475000 out of 500000 steps  (95%)
07:47:01:WU01:FS01:0x17:Completed 3950000 out of 5000000 steps (79%)
07:51:21:WU00:FS02:0xa3:Completed 480000 out of 500000 steps  (96%)
08:00:10:WU00:FS02:0xa3:Completed 485000 out of 500000 steps  (97%)
08:06:57:WU01:FS01:0x17:Completed 4000000 out of 5000000 steps (80%)
08:09:02:WU00:FS02:0xa3:Completed 490000 out of 500000 steps  (98%)
08:09:03:WU02:FS02:Connecting to assign3.stanford.edu:8080
08:09:04:WU02:FS02:News: Welcome to Folding@Home
08:09:04:WU02:FS02:Assigned to work server 128.143.231.202
08:09:04:WU02:FS02:Requesting new work unit for slot 02: RUNNING cpu:10 from 128.143.231.202
08:09:04:WU02:FS02:Connecting to 128.143.231.202:8080
08:09:05:WU02:FS02:Downloading 3.67MiB
08:09:06:WU02:FS02:Download complete
08:09:06:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:8573 run:1 clone:7 gen:285 core:0xa3 unit:0x00000be30a3b1e59522885978de15a8c
08:17:53:WU00:FS02:0xa3:Completed 495000 out of 500000 steps  (99%)
08:26:44:WU00:FS02:0xa3:Completed 500000 out of 500000 steps  (100%)
08:26:47:WU00:FS02:0xa3:DynamicWrapper: Finished Work Unit: sleep=10000
08:26:57:WU00:FS02:0xa3:
08:26:57:WU00:FS02:0xa3:Finished Work Unit:
08:26:57:WU00:FS02:0xa3:- Reading up to 8054064 from "00/wudata_01.trr": Read 8054064
08:26:57:WU00:FS02:0xa3:trr file hash check passed.
08:26:57:WU00:FS02:0xa3:edr file hash check passed.
08:26:57:WU00:FS02:0xa3:logfile size: 58862
08:26:57:WU00:FS02:0xa3:Leaving Run
08:26:59:WU00:FS02:0xa3:- Writing 8149758 bytes of core data to disk...
08:27:00:WU00:FS02:0xa3:Done: 8149246 -> 7526801 (compressed to 92.3 percent)
08:27:00:WU00:FS02:0xa3:  ... Done.
08:27:01:WU00:FS02:0xa3:- Shutting down core
08:27:01:WU00:FS02:0xa3:
08:27:01:WU00:FS02:0xa3:Folding@home Core Shutdown: FINISHED_UNIT
08:27:02:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:27:03:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:8576 run:1 clone:2 gen:292 core:0xa3 unit:0x00000c440a3b1e595228885eeb1f7b6f
08:27:03:WU00:FS02:Uploading 7.18MiB to 128.143.231.202
08:27:03:WU00:FS02:Connecting to 128.143.231.202:8080
08:27:03:WU02:FS02:Starting
08:27:03:WU02:FS02:Running FahCore: "E:\Stanford FAH\FAHClient/FAHCoreWrapper.exe" "E:/Stanford FAH/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a3.fah/FahCore_a3.exe" -dir 02 -suffix 01 -version 703 -lifeline 2312 -checkpoint 30 -np 10
08:27:03:WU02:FS02:Started FahCore on PID 5552
08:27:03:WU02:FS02:Core PID:3380
08:27:03:WU02:FS02:FahCore 0xa3 started
08:27:03:WU02:FS02:0xa3:
08:27:03:WU02:FS02:0xa3:*------------------------------*
08:27:03:WU02:FS02:0xa3:Folding@Home Gromacs SMP Core
08:27:03:WU02:FS02:0xa3:Version 2.27 (Dec. 15, 2010)
08:27:03:WU02:FS02:0xa3:
08:27:03:WU02:FS02:0xa3:Preparing to commence simulation
08:27:03:WU02:FS02:0xa3:- Looking at optimizations...
08:27:03:WU02:FS02:0xa3:- Created dyn
08:27:03:WU02:FS02:0xa3:- Files status OK
08:27:03:WU02:FS02:0xa3:- Expanded 3850911 -> 4393460 (decompressed 114.0 percent)
08:27:03:WU02:FS02:0xa3:Called DecompressByteArray: compressed_data_size=3850911 data_size=4393460, decompressed_data_size=4393460 diff=0
08:27:03:WU02:FS02:0xa3:- Digital signature verified
08:27:03:WU02:FS02:0xa3:
08:27:03:WU02:FS02:0xa3:Project: 8573 (Run 1, Clone 7, Gen 285)
08:27:03:WU02:FS02:0xa3:
08:27:03:WU02:FS02:0xa3:Assembly optimizations on if available.
08:27:03:WU02:FS02:0xa3:Entering M.D.
08:27:09:WU00:FS02:Upload 40.05%
08:27:09:WU02:FS02:0xa3:Mapping NT from 10 to 10 
08:27:10:WU02:FS02:0xa3:Completed 0 out of 500000 steps  (0%)
08:27:15:WU00:FS02:Upload 82.71%
08:27:18:WU00:FS02:Upload complete
08:27:18:WU00:FS02:Server responded WORK_ACK (400)
08:27:18:WU00:FS02:Final credit estimate, 12591.00 points
08:27:18:WU00:FS02:Cleaning up
08:27:19:WU01:FS01:0x17:Completed 4050000 out of 5000000 steps (81%)
08:36:00:WU02:FS02:0xa3:Completed 5000 out of 500000 steps  (1%)
08:44:50:WU02:FS02:0xa3:Completed 10000 out of 500000 steps  (2%)
08:47:16:WU01:FS01:0x17:Completed 4100000 out of 5000000 steps (82%)
08:53:40:WU02:FS02:0xa3:Completed 15000 out of 500000 steps  (3%)
09:02:34:WU02:FS02:0xa3:Completed 20000 out of 500000 steps  (4%)
09:07:38:WU01:FS01:0x17:Completed 4150000 out of 5000000 steps (83%)
09:11:23:WU02:FS02:0xa3:Completed 25000 out of 500000 steps  (5%)
09:20:13:WU02:FS02:0xa3:Completed 30000 out of 500000 steps  (6%)
09:27:37:WU01:FS01:0x17:Completed 4200000 out of 5000000 steps (84%)
09:29:04:WU02:FS02:0xa3:Completed 35000 out of 500000 steps  (7%)
09:37:54:WU02:FS02:0xa3:Completed 40000 out of 500000 steps  (8%)
09:46:44:WU02:FS02:0xa3:Completed 45000 out of 500000 steps  (9%)
09:47:36:WU01:FS01:0x17:Completed 4250000 out of 5000000 steps (85%)
09:55:34:WU02:FS02:0xa3:Completed 50000 out of 500000 steps  (10%)
10:04:25:WU02:FS02:0xa3:Completed 55000 out of 500000 steps  (11%)
10:08:00:WU01:FS01:0x17:Completed 4300000 out of 5000000 steps (86%)
10:13:15:WU02:FS02:0xa3:Completed 60000 out of 500000 steps  (12%)
10:22:04:WU02:FS02:0xa3:Completed 65000 out of 500000 steps  (13%)
10:28:00:WU01:FS01:0x17:Completed 4350000 out of 5000000 steps (87%)
10:29:11:FS00:Paused
10:29:16:FS00:Unpaused
10:29:16:FS00:Finishing

[bruce]

The "WARNING:While cleaning up" error is fixed in V7.4.4.

The sawtooth performance variations are a result of how the drvers supply OpenCL data to the hardware. What does NVidia say about this problem?

I don't know what's going on with the ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-36)
Here's one option that says it's an overclocking problem: viewtopic.php?f=66&t=24337&p=243358#p243359