Any idea how many Core 16's are left?

[Dr.G]

Thanks for the interesting thread.

@ ChasingTheDream: Have you tried the latest 14.9 AMD drivers, which came out two days ago on 9/29/2014?

I have / had the exact same problem, with constant crashes or 'lost' runs when running FAHcore 0x16 (where did they appear from?) on my system with dual identical R9 290X GPU's (FAHclient V7.4.4, Win 7 64bit, stock speeds, no overclock).

I got to the point I actually though my GPU cards were faulty. I tried many hardware and software options to try and prevent the frequent crashes / hangs etc. There are so many variables, that it's almost impossible to systematically test each and every possibility. It was driving me insane! And the regular need for reboots was a real pain. I've been folding with a progression of 3 different dedicated rigs for over a year now and was considering giving up all together. I can't afford the time to babysit my computer.

However, I've updated to the latest 14.9 AMD drivers, which came out two days ago on 9/29/2014. http://support.amd.com/en-us/download/d ... ows+7+-+64

At least now the core 0x16 seems not to crash (fingers crossed), but core 0x16 still takes a looong time to complete and the PPD is depressingly low of course. And I still keep picking up more 0x16 cores (one core 0x16 and one 0x17 each time).

Not a completely satisfactory solution, but hopefully an improvement.
I'll see how things go and keep you updated. Let me know your thoughts and how you are getting on.

Example current log (without crashes) below:

Code: Select all

*********************** Log Started 2014-09-30T05:16:09Z ***********************
05:16:09:************************* Folding@home Client *************************
05:16:09:      Website: http://folding.stanford.edu/
05:16:09:    Copyright: (c) 2009-2014 Stanford University
05:16:09:       Author: Joseph Coffland <[email protected]>
05:16:09:         Args: --open-web-control
05:16:09:       Config: C:/ProgramData/FAHClient/config.xml
05:16:09:******************************** Build ********************************
05:16:09:      Version: 7.4.4
05:16:09:         Date: Mar 4 2014
05:16:09:         Time: 20:26:54
05:16:09:      SVN Rev: 4130
05:16:09:       Branch: fah/trunk/client
05:16:09:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
05:16:09:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
05:16:09:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
05:16:09:     Platform: win32 XP
05:16:09:         Bits: 32
05:16:09:         Mode: Release
05:16:09:******************************* System ********************************
05:16:09:          CPU: Intel(R) Pentium(R) CPU G840 @ 2.80GHz
05:16:09:       CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
05:16:09:         CPUs: 2
05:16:09:       Memory: 3.96GiB
05:16:09:  Free Memory: 2.98GiB
05:16:09:      Threads: WINDOWS_THREADS
05:16:09:   OS Version: 6.1
05:16:09:  Has Battery: false
05:16:09:   On Battery: false
05:16:09:   UTC Offset: 1
05:16:09:          PID: 860
05:16:09:          CWD: C:/ProgramData/FAHClient
05:16:09:           OS: Windows 7 Home Premium
05:16:09:      OS Arch: AMD64
05:16:09:         GPUs: 2
05:16:09:        GPU 0: ATI:5 Hawaii [Radeon R9 200X Series]
05:16:09:        GPU 1: ATI:5 Hawaii [Radeon R9 200X Series]
05:16:09:         CUDA: Not detected
05:16:09:Win32 Service: false
05:16:09:***********************************************************************
05:16:09:<config>
05:16:09:  <!-- HTTP Server -->
05:16:09:  <allow v='127.0.0.1 192.168.1.0/24'/>
05:16:09:
05:16:09:  <!-- Network -->
05:16:09:  <proxy v=':8080'/>
05:16:09:
05:16:09:  <!-- Remote Command Server -->
05:16:09:  <password v='***********'/>
05:16:09:
05:16:09:  <!-- Slot Control -->
05:16:09:  <power v='full'/>
05:16:09:
05:16:09:  <!-- User Information -->
05:16:09:  <passkey v='********************************'/>
05:16:09:  <team v='xxxxxx'/>
05:16:09:  <user v='xxxxxx'/>
05:16:09:
05:16:09:  <!-- Folding Slots -->
05:16:09:  <slot id='0' type='GPU'>
05:16:09:    <next-unit-percentage v='100'/>
05:16:09:    <pause-on-start v='true'/>
05:16:09:    <paused v='true'/>
05:16:09:  </slot>
05:16:09:  <slot id='1' type='GPU'>
05:16:09:    <next-unit-percentage v='100'/>
05:16:09:    <pause-on-start v='true'/>
05:16:09:    <paused v='true'/>
05:16:09:  </slot>
05:16:09:</config>
05:16:09:Trying to access database...
05:16:09:Successfully acquired database lock
05:16:09:Enabled folding slot 00: PAUSED gpu:0:Hawaii [Radeon R9 200X Series] (by user)
05:16:09:Enabled folding slot 01: PAUSED gpu:1:Hawaii [Radeon R9 200X Series] (by user)
05:16:19:8:127.0.0.1:New Web connection
05:16:59:FS00:Unpaused
05:16:59:WU01:FS00:Starting
05:16:59:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_17.fah/FahCore_17.exe -dir 01 -suffix 01 -version 704 -lifeline 860 -checkpoint 15 -gpu 0 -gpu-vendor ati
05:16:59:WU01:FS00:Started FahCore on PID 3924
05:16:59:WU01:FS00:Core PID:2868
05:16:59:WU01:FS00:FahCore 0x17 started
05:17:01:WU01:FS00:0x17:*********************** Log Started 2014-09-30T05:17:00Z ***********************
05:17:01:WU01:FS00:0x17:Project: 13000 (Run 191, Clone 0, Gen 73)
05:17:01:WU01:FS00:0x17:Unit: 0x00000074538b3db7530fd1e0f9aed399
05:17:01:WU01:FS00:0x17:CPU: 0x00000000000000000000000000000000
05:17:01:WU01:FS00:0x17:Machine: 0
05:17:01:WU01:FS00:0x17:Digital signatures verified
05:17:01:WU01:FS00:0x17:Folding@home GPU core17
05:17:01:WU01:FS00:0x17:Version 0.0.52
05:17:07:WU01:FS00:0x17:  Found a checkpoint file
05:17:10:Removing old file 'configs/config-20140913-065006.xml'
05:17:10:Saving configuration to config.xml
05:17:10:<config>
05:17:10:  <!-- HTTP Server -->
05:17:10:  <allow v='127.0.0.1 192.168.1.0/24'/>
05:17:10:
05:17:10:  <!-- Network -->
05:17:10:  <proxy v=':8080'/>
05:17:10:
05:17:10:  <!-- Remote Command Server -->
05:17:10:  <password v='***********'/>
05:17:10:
05:17:10:  <!-- Slot Control -->
05:17:10:  <power v='full'/>
05:17:10:
05:17:10:  <!-- User Information -->
05:17:10:  <passkey v='********************************'/>
05:17:10:  <team v='xxxxxx'/>
05:17:10:  <user v='xxxxxxxx'/>
05:17:10:
05:17:10:  <!-- Folding Slots -->
05:17:10:  <slot id='0' type='GPU'>
05:17:10:    <next-unit-percentage v='100'/>
05:17:10:    <pause-on-start v='true'/>
05:17:10:  </slot>
05:17:10:  <slot id='1' type='GPU'>
05:17:10:    <next-unit-percentage v='100'/>
05:17:10:    <pause-on-start v='true'/>
05:17:10:    <paused v='true'/>
05:17:10:  </slot>
05:17:10:</config>
05:20:53:WU01:FS00:0x17:Completed 3500000 out of 5000000 steps (70%)
05:20:53:WU01:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:23:35:FS01:Unpaused
05:23:35:WU00:FS01:Starting
05:23:35:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 00 -suffix 01 -version 704 -lifeline 860 -checkpoint 15 -gpu 1 -gpu-vendor ati
05:23:35:WU00:FS01:Started FahCore on PID 3904
05:23:35:WU00:FS01:Core PID:2208
05:23:35:WU00:FS01:FahCore 0x16 started
05:23:35:WU00:FS01:0x16:
05:23:35:WU00:FS01:0x16:*------------------------------*
05:23:35:WU00:FS01:0x16:Folding@Home GPU Core
05:23:35:WU00:FS01:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
05:23:35:WU00:FS01:0x16:
05:23:35:WU00:FS01:0x16:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86 
05:23:35:WU00:FS01:0x16:Build host: user-f6d030f24f
05:23:35:WU00:FS01:0x16:Board Type: AMD/OpenCL
05:23:35:WU00:FS01:0x16:Core      : x=16
05:23:35:WU00:FS01:0x16: Window's signal control handler registered.
05:23:35:WU00:FS01:0x16:Preparing to commence simulation
05:23:35:WU00:FS01:0x16:- Looking at optimizations...
05:23:35:WU00:FS01:0x16:- Files status OK
05:23:35:WU00:FS01:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
05:23:35:WU00:FS01:0x16:- Expanded 45049 -> 171163 (decompressed 379.9 percent)
05:23:35:WU00:FS01:0x16:Called DecompressByteArray: compressed_data_size=45049 data_size=171163, decompressed_data_size=171163 diff=0
05:23:35:WU00:FS01:0x16:- Digital signature verified
05:23:35:WU00:FS01:0x16:
05:23:35:WU00:FS01:0x16:Project: 11292 (Run 13, Clone 128, Gen 35)
05:23:35:WU00:FS01:0x16:
05:23:35:WU00:FS01:0x16:Assembly optimizations on if available.
05:23:35:WU00:FS01:0x16:Entering M.D.
05:23:37:WU00:FS01:0x16:Will resume from checkpoint file 00/wudata_01.ckp
05:23:37:WU00:FS01:0x16:Tpr hash 00/wudata_01.tpr:  3406269736 3245505055 1328482047 2128478410 3321667595
05:23:37:WU00:FS01:0x16:Working on ALZHEIMER DISEASE AMYLOID
05:23:37:WU00:FS01:0x16:Client config unavailable.
05:23:38:WU00:FS01:0x16:Starting GUI Server
05:23:40:WU00:FS01:0x16:Resuming from checkpoint
05:23:40:WU00:FS01:0x16:fcCheckPointResume: retreived and current tpr file hash:
05:23:40:WU00:FS01:0x16:   0   3406269736   3406269736
05:23:40:WU00:FS01:0x16:   1   3245505055   3245505055
05:23:40:WU00:FS01:0x16:   2   1328482047   1328482047
05:23:40:WU00:FS01:0x16:   3   2128478410   2128478410
05:23:40:WU00:FS01:0x16:   4   3321667595   3321667595
05:23:40:WU00:FS01:0x16:fcCheckPointResume: file hashes same.
05:23:40:WU00:FS01:0x16:fcCheckPointResume: state restored.
05:23:40:WU00:FS01:0x16:fcCheckPointResume: name 00/wudata_01.log Verified 00/wudata_01.log
05:23:40:WU00:FS01:0x16:fcCheckPointResume: name 00/wudata_01.trr Verified 00/wudata_01.trr
05:23:40:WU00:FS01:0x16:fcCheckPointResume: name 00/wudata_01.xtc Verified 00/wudata_01.xtc
05:23:40:WU00:FS01:0x16:fcCheckPointResume: name 00/wudata_01.edr Verified 00/wudata_01.edr
05:23:40:WU00:FS01:0x16:fcCheckPointResume: state restored 2
05:23:40:WU00:FS01:0x16:Resumed from checkpoint
05:23:40:WU00:FS01:0x16:Setting checkpoint frequency: 599998
05:23:40:WU00:FS01:0x16:Completed  23399923 out of 59999872 steps (38%).
05:23:40:WU00:FS01:0x16:Completed  23399951 out of 59999872 steps (39%).
05:23:43:FS01:Finishing
05:24:17:Removing old file 'configs/config-20140914-121805.xml'
05:24:17:Saving configuration to config.xml
05:24:17:<config>
05:24:17:  <!-- HTTP Server -->
05:24:17:  <allow v='127.0.0.1 192.168.1.0/24'/>
05:24:17:
05:24:17:  <!-- Network -->
05:24:17:  <proxy v=':8080'/>
05:24:17:
05:24:17:  <!-- Remote Command Server -->
05:24:17:  <password v='***********'/>
05:24:17:
05:24:17:  <!-- Slot Control -->
05:24:17:  <power v='full'/>
05:24:17:
05:24:17:  <!-- User Information -->
05:24:17:  <passkey v='********************************'/>
05:24:17:  <team v='xxxxxxx'/>
05:24:17:  <user v='xxxxxxxx'/>
05:24:17:
05:24:17:  <!-- Folding Slots -->
05:24:17:  <slot id='0' type='GPU'>
05:24:17:    <next-unit-percentage v='100'/>
05:24:17:    <pause-on-start v='true'/>
05:24:17:  </slot>
05:24:17:  <slot id='1' type='GPU'>
05:24:17:    <next-unit-percentage v='100'/>
05:24:17:    <pause-on-start v='true'/>
05:24:17:  </slot>
05:24:17:</config>
05:26:37:WU01:FS00:0x17:Completed 3550000 out of 5000000 steps (71%)
05:29:28:WU00:FS01:0x16:Completed  23999949 out of 59999872 steps (40%).
05:32:03:WU01:FS00:0x17:Completed 3600000 out of 5000000 steps (72%)
05:35:25:WU00:FS01:0x16:Completed  24599948 out of 59999872 steps (41%).
05:37:52:WU01:FS00:0x17:Completed 3650000 out of 5000000 steps (73%)
05:41:14:WU00:FS01:0x16:Completed  25199947 out of 59999872 steps (42%).
05:43:18:WU01:FS00:0x17:Completed 3700000 out of 5000000 steps (74%)
05:47:02:WU00:FS01:0x16:Completed  25799945 out of 59999872 steps (43%).
05:48:44:WU01:FS00:0x17:Completed 3750000 out of 5000000 steps (75%)
05:53:00:WU00:FS01:0x16:Completed  26399944 out of 59999872 steps (44%).
05:54:31:WU01:FS00:0x17:Completed 3800000 out of 5000000 steps (76%)
05:58:48:WU00:FS01:0x16:Completed  26999943 out of 59999872 steps (45%).
05:59:57:WU01:FS00:0x17:Completed 3850000 out of 5000000 steps (77%)
06:04:45:WU00:FS01:0x16:Completed  27599942 out of 59999872 steps (46%).
06:05:45:WU01:FS00:0x17:Completed 3900000 out of 5000000 steps (78%)
06:10:35:WU00:FS01:0x16:Completed  28199940 out of 59999872 steps (47%).
06:11:10:WU01:FS00:0x17:Completed 3950000 out of 5000000 steps (79%)
06:16:25:WU00:FS01:0x16:Completed  28799939 out of 59999872 steps (48%).
06:16:36:WU01:FS00:0x17:Completed 4000000 out of 5000000 steps (80%)
06:22:23:WU00:FS01:0x16:Completed  29399938 out of 59999872 steps (49%).
06:22:24:WU01:FS00:0x17:Completed 4050000 out of 5000000 steps (81%)
06:27:49:WU01:FS00:0x17:Completed 4100000 out of 5000000 steps (82%)
06:28:13:WU00:FS01:0x16:Completed  29999936 out of 59999872 steps (50%).
06:33:44:WU01:FS00:0x17:Completed 4150000 out of 5000000 steps (83%)
06:34:09:WU00:FS01:0x16:Completed  30599935 out of 59999872 steps (51%).
06:39:09:WU01:FS00:0x17:Completed 4200000 out of 5000000 steps (84%)
06:39:59:WU00:FS01:0x16:Completed  31199934 out of 59999872 steps (52%).
06:44:35:WU01:FS00:0x17:Completed 4250000 out of 5000000 steps (85%)
06:45:57:WU00:FS01:0x16:Completed  31799933 out of 59999872 steps (53%).
06:50:23:WU01:FS00:0x17:Completed 4300000 out of 5000000 steps (86%)
06:51:46:WU00:FS01:0x16:Completed  32399931 out of 59999872 steps (54%).
06:55:48:WU01:FS00:0x17:Completed 4350000 out of 5000000 steps (87%)
06:57:37:WU00:FS01:0x16:Completed  32999930 out of 59999872 steps (55%).
07:01:36:WU01:FS00:0x17:Completed 4400000 out of 5000000 steps (88%)
07:03:34:WU00:FS01:0x16:Completed  33599929 out of 59999872 steps (56%).
07:07:01:WU01:FS00:0x17:Completed 4450000 out of 5000000 steps (89%)
07:09:24:WU00:FS01:0x16:Completed  34199928 out of 59999872 steps (57%).
07:12:27:WU01:FS00:0x17:Completed 4500000 out of 5000000 steps (90%)
07:15:23:WU00:FS01:0x16:Completed  34799926 out of 59999872 steps (58%).
07:18:14:WU01:FS00:0x17:Completed 4550000 out of 5000000 steps (91%)
07:21:13:WU00:FS01:0x16:Completed  35399925 out of 59999872 steps (59%).
07:23:42:WU01:FS00:0x17:Completed 4600000 out of 5000000 steps (92%)
07:27:12:WU00:FS01:0x16:Completed  35999924 out of 59999872 steps (60%).
07:29:28:WU01:FS00:0x17:Completed 4650000 out of 5000000 steps (93%)
07:33:03:WU00:FS01:0x16:Completed  36599922 out of 59999872 steps (61%).
07:34:54:WU01:FS00:0x17:Completed 4700000 out of 5000000 steps (94%)
07:38:53:WU00:FS01:0x16:Completed  37199921 out of 59999872 steps (62%).
07:40:21:WU01:FS00:0x17:Completed 4750000 out of 5000000 steps (95%)
07:44:52:WU00:FS01:0x16:Completed  37799920 out of 59999872 steps (63%).
07:46:10:WU01:FS00:0x17:Completed 4800000 out of 5000000 steps (96%)
07:50:43:WU00:FS01:0x16:Completed  38399919 out of 59999872 steps (64%).
07:51:36:WU01:FS00:0x17:Completed 4850000 out of 5000000 steps (97%)
07:56:41:WU00:FS01:0x16:Completed  38999917 out of 59999872 steps (65%).
07:57:23:WU01:FS00:0x17:Completed 4900000 out of 5000000 steps (98%)
08:02:32:WU00:FS01:0x16:Completed  39599916 out of 59999872 steps (66%).
08:02:48:WU01:FS00:0x17:Completed 4950000 out of 5000000 steps (99%)
08:08:13:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
08:08:15:WU02:FS00:Connecting to 171.67.108.201:80
08:08:16:WU02:FS00:Assigned to work server 140.163.4.233
08:08:16:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Hawaii [Radeon R9 200X Series] from 140.163.4.233
08:08:16:WU02:FS00:Connecting to 140.163.4.233:8080
08:08:16:WU02:FS00:Downloading 4.26MiB
08:08:21:WU02:FS00:Download complete
08:08:21:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:10467 run:0 clone:155 gen:47 core:0x17 unit:0x0000005f538b3db9538bbdc10247d40f
08:08:28:WU00:FS01:0x16:Completed  40199915 out of 59999872 steps (67%).
08:08:36:WU01:FS00:0x17:Saving result file logfile_01.txt
08:08:36:WU01:FS00:0x17:Saving result file checkpointState.xml
08:08:38:WU01:FS00:0x17:Saving result file checkpt.crc
08:08:38:WU01:FS00:0x17:Saving result file log.txt
08:08:39:WU01:FS00:0x17:Saving result file positions.xtc
08:08:41:WU01:FS00:0x17:Folding@home Core Shutdown: FINISHED_UNIT
08:08:41:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:08:41:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:13000 run:191 clone:0 gen:73 core:0x17 unit:0x00000074538b3db7530fd1e0f9aed399
08:08:42:WU01:FS00:Uploading 12.83MiB to 140.163.4.231
08:08:42:WU01:FS00:Connecting to 140.163.4.231:8080
08:08:42:WU02:FS00:Starting
08:08:42:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_17.fah/FahCore_17.exe -dir 02 -suffix 01 -version 704 -lifeline 860 -checkpoint 15 -gpu 0 -gpu-vendor ati
08:08:42:WU02:FS00:Started FahCore on PID 3344
08:08:42:WU02:FS00:Core PID:544
08:08:42:WU02:FS00:FahCore 0x17 started
08:08:43:WU02:FS00:0x17:*********************** Log Started 2014-09-30T08:08:42Z ***********************
08:08:44:WU02:FS00:0x17:Project: 10467 (Run 0, Clone 155, Gen 47)
08:08:44:WU02:FS00:0x17:Unit: 0x0000005f538b3db9538bbdc10247d40f
08:08:44:WU02:FS00:0x17:CPU: 0x00000000000000000000000000000000
08:08:44:WU02:FS00:0x17:Machine: 0
08:08:44:WU02:FS00:0x17:Reading tar file state.xml
08:08:44:WU02:FS00:0x17:Reading tar file system.xml
08:08:44:WU02:FS00:0x17:Reading tar file integrator.xml
08:08:44:WU02:FS00:0x17:Reading tar file core.xml
08:08:44:WU02:FS00:0x17:Digital signatures verified
08:08:44:WU02:FS00:0x17:Folding@home GPU core17
08:08:44:WU02:FS00:0x17:Version 0.0.52
08:08:48:WU01:FS00:Upload 7.31%
08:08:54:WU01:FS00:Upload 12.18%
08:09:00:WU01:FS00:Upload 17.53%
08:09:06:WU01:FS00:Upload 22.40%
08:09:12:WU01:FS00:Upload 27.76%
08:09:18:WU01:FS00:Upload 32.63%
08:09:24:WU01:FS00:Upload 37.99%
08:09:30:WU01:FS00:Upload 42.86%
08:09:36:WU01:FS00:Upload 47.73%
08:09:42:WU01:FS00:Upload 52.60%
08:09:48:WU01:FS00:Upload 57.47%
08:09:54:WU01:FS00:Upload 62.34%
08:10:00:WU01:FS00:Upload 66.73%
08:10:06:WU01:FS00:Upload 72.08%
08:10:12:WU01:FS00:Upload 76.95%
08:10:18:WU01:FS00:Upload 81.82%
08:10:24:WU01:FS00:Upload 86.70%
08:10:30:WU01:FS00:Upload 91.57%
08:10:36:WU01:FS00:Upload 96.44%
08:10:50:WU01:FS00:Upload complete
08:10:51:WU01:FS00:Server responded WORK_ACK (400)
08:10:51:WU01:FS00:Final credit estimate, 75562.00 points
08:10:51:WU01:FS00:Cleaning up
08:11:44:WU02:FS00:0x17:Completed 0 out of 5000000 steps (0%)
08:11:44:WU02:FS00:0x17:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
08:15:28:WU00:FS01:0x16:Completed  40799913 out of 59999872 steps (68%).
08:16:28:WU02:FS00:0x17:Completed 50000 out of 5000000 steps (1%)
08:20:55:WU02:FS00:0x17:Completed 100000 out of 5000000 steps (2%)
08:21:11:WU00:FS01:0x16:Completed  41399912 out of 59999872 steps (69%).
08:25:40:WU02:FS00:0x17:Completed 150000 out of 5000000 steps (3%)
08:26:59:WU00:FS01:0x16:Completed  41999911 out of 59999872 steps (70%).
08:30:03:WU02:FS00:0x17:Completed 200000 out of 5000000 steps (4%)
08:32:43:WU00:FS01:0x16:Completed  42599910 out of 59999872 steps (71%).
08:34:27:WU02:FS00:0x17:Completed 250000 out of 5000000 steps (5%)
08:38:33:WU00:FS01:0x16:Completed  43199908 out of 59999872 steps (72%).
08:39:09:WU02:FS00:0x17:Completed 300000 out of 5000000 steps (6%)
08:43:33:WU02:FS00:0x17:Completed 350000 out of 5000000 steps (7%)
08:44:16:WU00:FS01:0x16:Completed  43799907 out of 59999872 steps (73%).
08:48:16:WU02:FS00:0x17:Completed 400000 out of 5000000 steps (8%)
08:50:07:WU00:FS01:0x16:Completed  44399906 out of 59999872 steps (74%).
08:52:46:WU02:FS00:0x17:Completed 450000 out of 5000000 steps (9%)
08:55:48:WU00:FS01:0x16:Completed  44999904 out of 59999872 steps (75%).
08:57:12:WU02:FS00:0x17:Completed 500000 out of 5000000 steps (10%)
09:01:38:WU00:FS01:0x16:Completed  45599903 out of 59999872 steps (76%).
09:01:55:WU02:FS00:0x17:Completed 550000 out of 5000000 steps (11%)
09:06:20:WU02:FS00:0x17:Completed 600000 out of 5000000 steps (12%)
09:07:21:WU00:FS01:0x16:Completed  46199902 out of 59999872 steps (77%).
09:11:04:WU02:FS00:0x17:Completed 650000 out of 5000000 steps (13%)
09:13:11:WU00:FS01:0x16:Completed  46799901 out of 59999872 steps (78%).
09:15:28:WU02:FS00:0x17:Completed 700000 out of 5000000 steps (14%)
09:18:54:WU00:FS01:0x16:Completed  47399899 out of 59999872 steps (79%).
09:19:58:WU02:FS00:0x17:Completed 750000 out of 5000000 steps (15%)
09:24:40:WU02:FS00:0x17:Completed 800000 out of 5000000 steps (16%)
09:24:43:WU00:FS01:0x16:Completed  47999898 out of 59999872 steps (80%).
09:29:01:WU02:FS00:0x17:Completed 850000 out of 5000000 steps (17%)
09:30:28:WU00:FS01:0x16:Completed  48599897 out of 59999872 steps (81%).
09:33:44:WU02:FS00:0x17:Completed 900000 out of 5000000 steps (18%)
09:36:17:WU00:FS01:0x16:Completed  49199896 out of 59999872 steps (82%).
09:38:09:WU02:FS00:0x17:Completed 950000 out of 5000000 steps (19%)
09:42:00:WU00:FS01:0x16:Completed  49799894 out of 59999872 steps (83%).
09:42:36:WU02:FS00:0x17:Completed 1000000 out of 5000000 steps (20%)
09:47:21:WU02:FS00:0x17:Completed 1050000 out of 5000000 steps (21%)
09:47:49:WU00:FS01:0x16:Completed  50399893 out of 59999872 steps (84%).
09:51:51:WU02:FS00:0x17:Completed 1100000 out of 5000000 steps (22%)
09:53:32:WU00:FS01:0x16:Completed  50999892 out of 59999872 steps (85%).
09:56:31:WU02:FS00:0x17:Completed 1150000 out of 5000000 steps (23%)
09:59:23:WU00:FS01:0x16:Completed  51599890 out of 59999872 steps (86%).
10:00:57:WU02:FS00:0x17:Completed 1200000 out of 5000000 steps (24%)
10:05:06:WU00:FS01:0x16:Completed  52199889 out of 59999872 steps (87%).
10:05:20:WU02:FS00:0x17:Completed 1250000 out of 5000000 steps (25%)
10:10:04:WU02:FS00:0x17:Completed 1300000 out of 5000000 steps (26%)
10:10:56:WU00:FS01:0x16:Completed  52799888 out of 59999872 steps (88%).
10:14:29:WU02:FS00:0x17:Completed 1350000 out of 5000000 steps (27%)
10:16:39:WU00:FS01:0x16:Completed  53399887 out of 59999872 steps (89%).
10:19:13:WU02:FS00:0x17:Completed 1400000 out of 5000000 steps (28%)
10:22:30:WU00:FS01:0x16:Completed  53999885 out of 59999872 steps (90%).
10:23:38:WU02:FS00:0x17:Completed 1450000 out of 5000000 steps (29%)
10:28:04:WU02:FS00:0x17:Completed 1500000 out of 5000000 steps (30%)
10:28:17:WU00:FS01:0x16:Completed  54599884 out of 59999872 steps (91%).
10:32:49:WU02:FS00:0x17:Completed 1550000 out of 5000000 steps (31%)
10:34:02:WU00:FS01:0x16:Completed  55199883 out of 59999872 steps (92%).
10:37:19:WU02:FS00:0x17:Completed 1600000 out of 5000000 steps (32%)
10:39:49:WU00:FS01:0x16:Completed  55799881 out of 59999872 steps (93%).
10:42:03:WU02:FS00:0x17:Completed 1650000 out of 5000000 steps (33%)
10:45:34:WU00:FS01:0x16:Completed  56399880 out of 59999872 steps (94%).
10:46:28:WU02:FS00:0x17:Completed 1700000 out of 5000000 steps (34%)
10:50:56:WU02:FS00:0x17:Completed 1750000 out of 5000000 steps (35%)
10:51:24:WU00:FS01:0x16:Completed  56999879 out of 59999872 steps (95%).
10:55:39:WU02:FS00:0x17:Completed 1800000 out of 5000000 steps (36%)
10:57:08:WU00:FS01:0x16:Completed  57599878 out of 59999872 steps (96%).
11:00:04:WU02:FS00:0x17:Completed 1850000 out of 5000000 steps (37%)
11:02:57:WU00:FS01:0x16:Completed  58199876 out of 59999872 steps (97%).
11:04:49:WU02:FS00:0x17:Completed 1900000 out of 5000000 steps (38%)
11:08:41:WU00:FS01:0x16:Completed  58799875 out of 59999872 steps (98%).
11:09:15:WU02:FS00:0x17:Completed 1950000 out of 5000000 steps (39%)
11:13:40:WU02:FS00:0x17:Completed 2000000 out of 5000000 steps (40%)
11:14:30:WU00:FS01:0x16:Completed  59399874 out of 59999872 steps (99%).
******************************* Date: 2014-09-30 *******************************
11:18:28:WU02:FS00:0x17:Completed 2050000 out of 5000000 steps (41%)
11:20:13:WU00:FS01:0x16:Completed  59999872 out of 59999872 steps (100%).
11:20:13:WU00:FS01:0x16:Finished fah_main
11:20:13:WU00:FS01:0x16:
11:20:13:WU00:FS01:0x16:Successful run
11:20:13:WU00:FS01:0x16:DynamicWrapper: Finished Work Unit: sleep=10000
11:20:23:WU00:FS01:0x16:Reserved 2922496 bytes for xtc file; Cosm status=0
11:20:23:WU00:FS01:0x16:Allocated 2922496 bytes for xtc file
11:20:23:WU00:FS01:0x16:- Reading up to 2922496 from "00/wudata_01.xtc": Read 2922496
11:20:23:WU00:FS01:0x16:Read 2922496 bytes from xtc file; available packet space=783507968
11:20:23:WU00:FS01:0x16:xtc file hash check passed.
11:20:23:WU00:FS01:0x16:Reserved 75840 75840 783507968 bytes for arc file=<00/wudata_01.trr> Cosm status=0
11:20:23:WU00:FS01:0x16:Allocated 75840 bytes for arc file
11:20:23:WU00:FS01:0x16:- Reading up to 75840 from "00/wudata_01.trr": Read 75840
11:20:23:WU00:FS01:0x16:Read 75840 bytes from arc file; available packet space=783432128
11:20:23:WU00:FS01:0x16:trr file hash check passed.
11:20:23:WU00:FS01:0x16:Allocated 544 bytes for edr file
11:20:23:WU00:FS01:0x16:Read bedfile
11:20:23:WU00:FS01:0x16:edr file hash check passed.
11:20:23:WU00:FS01:0x16:Allocated 140392 bytes for logfile
11:20:23:WU00:FS01:0x16:Read logfile
11:20:23:WU00:FS01:0x16:GuardedRun: success in DynamicWrapper
11:20:23:WU00:FS01:0x16:GuardedRun: done
11:20:23:WU00:FS01:0x16:Run: GuardedRun completed.
11:20:27:WU00:FS01:0x16:+ Opened results file
11:20:27:WU00:FS01:0x16:- Writing 3139784 bytes of core data to disk...
11:20:28:WU00:FS01:0x16:Done: 3139272 -> 2955982 (compressed to 94.1 percent)
11:20:28:WU00:FS01:0x16:  ... Done.
11:20:28:WU00:FS01:0x16:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
11:20:28:WU00:FS01:0x16:Shutting down core 
11:20:28:WU00:FS01:0x16:
11:20:28:WU00:FS01:0x16:Folding@home Core Shutdown: FINISHED_UNIT
11:20:29:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:20:29:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11292 run:13 clone:128 gen:35 core:0x16 unit:0x000000486652edbc4d62a43972483e3d
11:20:29:WU00:FS01:Uploading 2.82MiB to 171.67.108.44
11:20:29:WU00:FS01:Connecting to 171.67.108.44:8080
11:20:35:WU00:FS01:Upload 19.95%
11:20:41:WU00:FS01:Upload 44.33%
11:20:47:WU00:FS01:Upload 68.72%
11:20:53:WU00:FS01:Upload 93.10%
11:20:55:WU00:FS01:Upload complete
11:20:55:WU00:FS01:Server responded WORK_ACK (400)
11:20:55:WU00:FS01:Final credit estimate, 2224.00 points
11:20:55:WU00:FS01:Cleaning up
11:22:48:WU02:FS00:0x17:Completed 2100000 out of 5000000 steps (42%)
11:27:21:WU02:FS00:0x17:Completed 2150000 out of 5000000 steps (43%)
11:31:36:WU02:FS00:0x17:Completed 2200000 out of 5000000 steps (44%)
11:35:51:WU02:FS00:0x17:Completed 2250000 out of 5000000 steps (45%)
11:40:24:WU02:FS00:0x17:Completed 2300000 out of 5000000 steps (46%)
11:44:39:WU02:FS00:0x17:Completed 2350000 out of 5000000 steps (47%)
11:49:12:WU02:FS00:0x17:Completed 2400000 out of 5000000 steps (48%)
11:53:27:WU02:FS00:0x17:Completed 2450000 out of 5000000 steps (49%)
11:57:42:WU02:FS00:0x17:Completed 2500000 out of 5000000 steps (50%)
12:02:15:WU02:FS00:0x17:Completed 2550000 out of 5000000 steps (51%)
12:06:30:WU02:FS00:0x17:Completed 2600000 out of 5000000 steps (52%)
12:11:02:WU02:FS00:0x17:Completed 2650000 out of 5000000 steps (53%)
12:15:17:WU02:FS00:0x17:Completed 2700000 out of 5000000 steps (54%)
12:19:32:WU02:FS00:0x17:Completed 2750000 out of 5000000 steps (55%)
12:24:05:WU02:FS00:0x17:Completed 2800000 out of 5000000 steps (56%)
12:28:20:WU02:FS00:0x17:Completed 2850000 out of 5000000 steps (57%)
12:32:52:WU02:FS00:0x17:Completed 2900000 out of 5000000 steps (58%)
12:37:07:WU02:FS00:0x17:Completed 2950000 out of 5000000 steps (59%)
12:41:20:WU02:FS00:0x17:Completed 3000000 out of 5000000 steps (60%)
12:45:52:WU02:FS00:0x17:Completed 3050000 out of 5000000 steps (61%)
12:50:07:WU02:FS00:0x17:Completed 3100000 out of 5000000 steps (62%)
12:54:40:WU02:FS00:0x17:Completed 3150000 out of 5000000 steps (63%)
12:58:54:WU02:FS00:0x17:Completed 3200000 out of 5000000 steps (64%)
13:03:09:WU02:FS00:0x17:Completed 3250000 out of 5000000 steps (65%)
13:07:41:WU02:FS00:0x17:Completed 3300000 out of 5000000 steps (66%)
13:11:56:WU02:FS00:0x17:Completed 3350000 out of 5000000 steps (67%)
13:16:28:WU02:FS00:0x17:Completed 3400000 out of 5000000 steps (68%)
13:20:43:WU02:FS00:0x17:Completed 3450000 out of 5000000 steps (69%)
13:24:57:WU02:FS00:0x17:Completed 3500000 out of 5000000 steps (70%)
13:29:30:WU02:FS00:0x17:Completed 3550000 out of 5000000 steps (71%)
13:33:44:WU02:FS00:0x17:Completed 3600000 out of 5000000 steps (72%)
13:38:17:WU02:FS00:0x17:Completed 3650000 out of 5000000 steps (73%)
13:42:31:WU02:FS00:0x17:Completed 3700000 out of 5000000 steps (74%)
13:46:46:WU02:FS00:0x17:Completed 3750000 out of 5000000 steps (75%)
13:51:18:WU02:FS00:0x17:Completed 3800000 out of 5000000 steps (76%)
13:55:32:WU02:FS00:0x17:Completed 3850000 out of 5000000 steps (77%)
14:00:05:WU02:FS00:0x17:Completed 3900000 out of 5000000 steps (78%)
14:04:20:WU02:FS00:0x17:Completed 3950000 out of 5000000 steps (79%)
14:08:34:WU02:FS00:0x17:Completed 4000000 out of 5000000 steps (80%)
14:13:07:WU02:FS00:0x17:Completed 4050000 out of 5000000 steps (81%)
14:17:21:WU02:FS00:0x17:Completed 4100000 out of 5000000 steps (82%)
14:21:54:WU02:FS00:0x17:Completed 4150000 out of 5000000 steps (83%)
14:26:08:WU02:FS00:0x17:Completed 4200000 out of 5000000 steps (84%)
14:30:22:WU02:FS00:0x17:Completed 4250000 out of 5000000 steps (85%)
14:34:55:WU02:FS00:0x17:Completed 4300000 out of 5000000 steps (86%)
14:39:09:WU02:FS00:0x17:Completed 4350000 out of 5000000 steps (87%)
14:43:42:WU02:FS00:0x17:Completed 4400000 out of 5000000 steps (88%)
14:47:56:WU02:FS00:0x17:Completed 4450000 out of 5000000 steps (89%)
14:52:12:WU02:FS00:0x17:Completed 4500000 out of 5000000 steps (90%)
14:56:45:WU02:FS00:0x17:Completed 4550000 out of 5000000 steps (91%)
15:00:59:WU02:FS00:0x17:Completed 4600000 out of 5000000 steps (92%)
15:05:32:WU02:FS00:0x17:Completed 4650000 out of 5000000 steps (93%)
15:09:47:WU02:FS00:0x17:Completed 4700000 out of 5000000 steps (94%)
15:14:02:WU02:FS00:0x17:Completed 4750000 out of 5000000 steps (95%)
15:18:35:WU02:FS00:0x17:Completed 4800000 out of 5000000 steps (96%)
15:22:50:WU02:FS00:0x17:Completed 4850000 out of 5000000 steps (97%)
15:27:22:WU02:FS00:0x17:Completed 4900000 out of 5000000 steps (98%)
15:31:37:WU02:FS00:0x17:Completed 4950000 out of 5000000 steps (99%)
15:35:52:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
15:35:55:WU00:FS00:Connecting to 171.67.108.201:80
15:35:57:WU00:FS00:Assigned to work server 171.67.108.44
15:35:57:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:Hawaii [Radeon R9 200X Series] from 171.67.108.44
15:35:57:WU00:FS00:Connecting to 171.67.108.44:8080
15:35:58:WU00:FS00:Downloading 44.55KiB
15:35:58:WU00:FS00:Download complete
15:35:58:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11293 run:22 clone:85 gen:49 core:0x16 unit:0x0000005a6652edbc4d94b94d612d3176
15:36:11:WU02:FS00:0x17:Saving result file logfile_01.txt
15:36:11:WU02:FS00:0x17:Saving result file checkpointState.xml
15:36:13:WU02:FS00:0x17:Saving result file checkpt.crc
15:36:13:WU02:FS00:0x17:Saving result file log.txt
15:36:13:WU02:FS00:0x17:Saving result file positions.xtc
15:36:15:WU02:FS00:0x17:Folding@home Core Shutdown: FINISHED_UNIT
15:36:16:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
15:36:16:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:10467 run:0 clone:155 gen:47 core:0x17 unit:0x0000005f538b3db9538bbdc10247d40f
15:36:16:WU02:FS00:Uploading 12.69MiB to 140.163.4.233
15:36:16:WU02:FS00:Connecting to 140.163.4.233:8080
15:36:16:WU00:FS00:Starting
15:36:16:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 00 -suffix 01 -version 704 -lifeline 860 -checkpoint 15 -gpu 0 -gpu-vendor ati
15:36:16:WU00:FS00:Started FahCore on PID 2196
15:36:16:WU00:FS00:Core PID:3392
15:36:16:WU00:FS00:FahCore 0x16 started
15:36:17:WU00:FS00:0x16:
15:36:17:WU00:FS00:0x16:*------------------------------*
15:36:17:WU00:FS00:0x16:Folding@Home GPU Core
15:36:17:WU00:FS00:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
15:36:17:WU00:FS00:0x16:
15:36:17:WU00:FS00:0x16:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86 
15:36:17:WU00:FS00:0x16:Build host: user-f6d030f24f
15:36:17:WU00:FS00:0x16:Board Type: AMD/OpenCL
15:36:17:WU00:FS00:0x16:Core      : x=16
15:36:17:WU00:FS00:0x16: Window's signal control handler registered.
15:36:17:WU00:FS00:0x16:Preparing to commence simulation
15:36:17:WU00:FS00:0x16:- Looking at optimizations...
15:36:17:WU00:FS00:0x16:DeleteFrameFiles: successfully deleted file=00/wudata_01.ckp
15:36:17:WU00:FS00:0x16:- Created dyn
15:36:17:WU00:FS00:0x16:- Files status OK
15:36:17:WU00:FS00:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
15:36:17:WU00:FS00:0x16:- Expanded 45110 -> 171163 (decompressed 379.4 percent)
15:36:17:WU00:FS00:0x16:Called DecompressByteArray: compressed_data_size=45110 data_size=171163, decompressed_data_size=171163 diff=0
15:36:17:WU00:FS00:0x16:- Digital signature verified
15:36:17:WU00:FS00:0x16:
15:36:17:WU00:FS00:0x16:Project: 11293 (Run 22, Clone 85, Gen 49)
15:36:17:WU00:FS00:0x16:
15:36:17:WU00:FS00:0x16:Assembly optimizations on if available.
15:36:17:WU00:FS00:0x16:Entering M.D.
15:36:18:WU00:FS00:0x16:Tpr hash 00/wudata_01.tpr:  719209325 3453024595 2562531780 2062717776 4062818454
15:36:18:WU00:FS00:0x16:Working on ALZHEIMER DISEASE AMYLOID
15:36:18:WU00:FS00:0x16:Client config unavailable.
15:36:19:WU00:FS00:0x16:Starting GUI Server
15:36:21:WU00:FS00:0x16:Setting checkpoint frequency: 500000
15:36:21:WU00:FS00:0x16:Completed         3 out of 50000000 steps (0%).
15:36:22:WU02:FS00:Upload 4.92%
15:36:28:WU02:FS00:Upload 9.36%
15:36:34:WU02:FS00:Upload 14.28%
15:36:40:WU02:FS00:Upload 18.71%
15:36:46:WU02:FS00:Upload 23.64%
15:36:52:WU02:FS00:Upload 29.05%
15:36:58:WU02:FS00:Upload 33.49%
15:37:04:WU02:FS00:Upload 38.41%
15:37:10:WU02:FS00:Upload 43.83%
15:37:16:WU02:FS00:Upload 48.75%
15:37:22:WU02:FS00:Upload 53.67%
15:37:28:WU02:FS00:Upload 58.60%
15:37:34:WU02:FS00:Upload 63.52%
15:37:40:WU02:FS00:Upload 67.96%
15:37:46:WU02:FS00:Upload 72.88%
15:37:52:WU02:FS00:Upload 77.80%
15:37:58:WU02:FS00:Upload 82.73%
15:38:04:WU02:FS00:Upload 87.65%
15:38:10:WU02:FS00:Upload 92.58%
15:38:16:WU02:FS00:Upload 97.50%
15:38:28:WU02:FS00:Upload complete
15:38:28:WU02:FS00:Server responded WORK_ACK (400)
15:38:28:WU02:FS00:Final credit estimate, 77161.00 points
15:38:28:WU02:FS00:Cleaning up
15:40:59:WU00:FS00:0x16:Completed    500000 out of 50000000 steps (1%).
15:46:06:WU00:FS00:0x16:Completed   1000000 out of 50000000 steps (2%).
15:51:13:WU00:FS00:0x16:Completed   1500000 out of 50000000 steps (3%).
15:56:17:WU00:FS00:0x16:Completed   2000000 out of 50000000 steps (4%).
16:01:24:WU00:FS00:0x16:Completed   2500000 out of 50000000 steps (5%).
16:06:30:WU00:FS00:0x16:Completed   3000000 out of 50000000 steps (6%).
16:11:38:WU00:FS00:0x16:Completed   3500000 out of 50000000 steps (7%).
16:16:42:WU00:FS00:0x16:Completed   4000000 out of 50000000 steps (8%).
16:21:51:WU00:FS00:0x16:Completed   4500000 out of 50000000 steps (9%).
16:26:58:WU00:FS00:0x16:Completed   5000000 out of 50000000 steps (10%).
16:32:05:WU00:FS00:0x16:Completed   5500000 out of 50000000 steps (11%).
16:37:12:WU00:FS00:0x16:Completed   6000000 out of 50000000 steps (12%).
16:42:20:WU00:FS00:0x16:Completed   6500000 out of 50000000 steps (13%).
16:47:27:WU00:FS00:0x16:Completed   7000000 out of 50000000 steps (14%).
16:52:32:WU00:FS00:0x16:Completed   7500000 out of 50000000 steps (15%).
16:57:42:WU00:FS00:0x16:Completed   8000000 out of 50000000 steps (16%).
17:02:47:WU00:FS00:0x16:Completed   8500000 out of 50000000 steps (17%).
17:07:56:WU00:FS00:0x16:Completed   9000000 out of 50000000 steps (18%).
17:13:02:WU00:FS00:0x16:Completed   9500000 out of 50000000 steps (19%).
17:18:10:WU00:FS00:0x16:Completed  10000000 out of 50000000 steps (20%).
******************************* Date: 2014-09-30 *******************************
17:23:17:WU00:FS00:0x16:Completed  10500000 out of 50000000 steps (21%).
17:28:27:WU00:FS00:0x16:Completed  11000000 out of 50000000 steps (22%).
17:33:17:FS00:Finishing
17:33:33:WU00:FS00:0x16:Completed  11500000 out of 50000000 steps (23%).
17:38:43:WU00:FS00:0x16:Completed  12000000 out of 50000000 steps (24%).
17:43:50:WU00:FS00:0x16:Completed  12500000 out of 50000000 steps (25%).
17:48:57:WU00:FS00:0x16:Completed  13000000 out of 50000000 steps (26%).
17:54:04:WU00:FS00:0x16:Completed  13500000 out of 50000000 steps (27%).
17:59:11:WU00:FS00:0x16:Completed  14000000 out of 50000000 steps (28%).
18:04:18:WU00:FS00:0x16:Completed  14500000 out of 50000000 steps (29%).
18:09:25:WU00:FS00:0x16:Completed  15000000 out of 50000000 steps (30%).
18:14:32:WU00:FS00:0x16:Completed  15500000 out of 50000000 steps (31%).
18:19:40:WU00:FS00:0x16:Completed  16000000 out of 50000000 steps (32%).
18:24:47:WU00:FS00:0x16:Completed  16500000 out of 50000000 steps (33%).
18:29:53:WU00:FS00:0x16:Completed  17000000 out of 50000000 steps (34%).
18:35:02:WU00:FS00:0x16:Completed  17500000 out of 50000000 steps (35%).
18:40:08:WU00:FS00:0x16:Completed  18000000 out of 50000000 steps (36%).
18:45:17:WU00:FS00:0x16:Completed  18500000 out of 50000000 steps (37%).
18:50:23:WU00:FS00:0x16:Completed  19000000 out of 50000000 steps (38%).
18:55:33:WU00:FS00:0x16:Completed  19500000 out of 50000000 steps (39%).
19:00:40:WU00:FS00:0x16:Completed  20000000 out of 50000000 steps (40%).
19:05:47:WU00:FS00:0x16:Completed  20500000 out of 50000000 steps (41%).
19:10:53:WU00:FS00:0x16:Completed  21000000 out of 50000000 steps (42%).
19:16:02:WU00:FS00:0x16:Completed  21500000 out of 50000000 steps (43%).
19:21:11:WU00:FS00:0x16:Completed  22000000 out of 50000000 steps (44%).
19:26:19:WU00:FS00:0x16:Completed  22500000 out of 50000000 steps (45%).
19:31:27:WU00:FS00:0x16:Completed  23000000 out of 50000000 steps (46%).
19:36:34:WU00:FS00:0x16:Completed  23500000 out of 50000000 steps (47%).
19:41:41:WU00:FS00:0x16:Completed  24000000 out of 50000000 steps (48%).
19:46:48:WU00:FS00:0x16:Completed  24500000 out of 50000000 steps (49%).
19:51:56:WU00:FS00:0x16:Completed  25000000 out of 50000000 steps (50%).
19:57:03:WU00:FS00:0x16:Completed  25500000 out of 50000000 steps (51%).
20:02:10:WU00:FS00:0x16:Completed  26000000 out of 50000000 steps (52%).
20:07:16:WU00:FS00:0x16:Completed  26500000 out of 50000000 steps (53%).
20:12:24:WU00:FS00:0x16:Completed  27000000 out of 50000000 steps (54%).
20:17:34:WU00:FS00:0x16:Completed  27500000 out of 50000000 steps (55%).
20:22:40:WU00:FS00:0x16:Completed  28000000 out of 50000000 steps (56%).

[ChasingTheDream]

@ Dr.G

Great find! I checked AMD's drivers over the weekend to no avail. Looks like they issued them a day later! I'll get these installed everywhere and report back! Thanks again!

[ChasingTheDream]

@ DR.G Unfortunately, my results didn't turn out nearly as well as yours with the new drivers. I appear to have the same issues as before although it does look like the Core 17's may run a bit faster.

Core 16's however, still hard lock the multi-GPU computers, cause BSD Windows crashes, and now this which is actually on a single GPU system.

Code: Select all

*********************** Log Started 2014-10-01T20:22:50Z ***********************
20:22:50:WU01:FS01:Starting
20:22:50:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/FOLDER8/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 01 -suffix 01 -version 704 -lifeline 2988 -checkpoint 15 -gpu 0 -gpu-vendor ati
20:22:50:WU01:FS01:Started FahCore on PID 3516
20:22:50:WU01:FS01:Core PID:3536
20:22:50:WU01:FS01:FahCore 0x16 started
20:22:51:WU01:FS01:0x16:
20:22:51:WU01:FS01:0x16:*------------------------------*
20:22:51:WU01:FS01:0x16:Folding@Home GPU Core
20:22:51:WU01:FS01:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
20:22:51:WU01:FS01:0x16:
20:22:51:WU01:FS01:0x16:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86 
20:22:51:WU01:FS01:0x16:Build host: user-f6d030f24f
20:22:51:WU01:FS01:0x16:Board Type: AMD/OpenCL
20:22:51:WU01:FS01:0x16:Core      : x=16
20:22:51:WU01:FS01:0x16: Window's signal control handler registered.
20:22:51:WU01:FS01:0x16:Preparing to commence simulation
20:22:51:WU01:FS01:0x16:- Ensuring status. Please wait.
20:23:00:WU01:FS01:0x16:- Looking at optimizations...
20:23:00:WU01:FS01:0x16:- Working with standard loops on this execution.
20:23:00:WU01:FS01:0x16:- Previous termination of core was improper.
20:23:00:WU01:FS01:0x16:- Files status OK
20:23:00:WU01:FS01:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
20:23:00:WU01:FS01:0x16:- Expanded 44866 -> 171163 (decompressed 381.4 percent)
20:23:00:WU01:FS01:0x16:Called DecompressByteArray: compressed_data_size=44866 data_size=171163, decompressed_data_size=171163 diff=0
20:23:00:WU01:FS01:0x16:- Digital signature verified
20:23:00:WU01:FS01:0x16:
20:23:00:WU01:FS01:0x16:Project: 11292 (Run 3, Clone 128, Gen 25)
20:23:00:WU01:FS01:0x16:
20:23:00:WU01:FS01:0x16:Entering M.D.
20:23:02:WU01:FS01:0x16:Will resume from checkpoint file 01/wudata_01.ckp
20:23:02:WU01:FS01:0x16:Tpr hash 01/wudata_01.tpr:  2334513548 3578880146 446919189 3296134596 3488984289
20:23:03:WU01:FS01:0x16:Working on ALZHEIMER DISEASE AMYLOID
20:23:03:WU01:FS01:0x16:Client config unavailable.
20:23:03:WU01:FS01:0x16:Starting GUI Server
20:23:21:WU01:FS01:0x16:Resuming from checkpoint
20:23:21:WU01:FS01:0x16:fcCheckPointResume: retreived and current tpr file hash:
20:23:21:WU01:FS01:0x16:   0   2334513548   2334513548
20:23:21:WU01:FS01:0x16:   1   3578880146   3578880146
20:23:21:WU01:FS01:0x16:   2    446919189    446919189
20:23:21:WU01:FS01:0x16:   3   3296134596   3296134596
20:23:21:WU01:FS01:0x16:   4   3488984289   3488984289
20:23:21:WU01:FS01:0x16:fcCheckPointResume: file hashes same.
20:23:21:WU01:FS01:0x16:fcCheckPointResume: state restored.
20:23:21:WU01:FS01:0x16:fcCheckPointResume: name 01/wudata_01.log Verified 01/wudata_01.log
20:23:21:WU01:FS01:0x16:fcCheckPointResume: name 01/wudata_01.trr Verified 01/wudata_01.trr
20:23:21:WU01:FS01:0x16:fcCheckPointResume: name 01/wudata_01.xtc Verified 01/wudata_01.xtc
20:23:21:WU01:FS01:0x16:fcCheckPointResume: name 01/wudata_01.edr Verified 01/wudata_01.edr
20:23:21:WU01:FS01:0x16:fcCheckPointResume: state restored 2
20:23:21:WU01:FS01:0x16:Resumed from checkpoint
20:23:21:WU01:FS01:0x16:Setting checkpoint frequency: 500000
20:23:21:WU01:FS01:0x16:Completed         3 out of 50000000 steps (0%).
20:49:59:WU01:FS01:0x16:Completed    500000 out of 50000000 steps (1%).
21:19:27:WU01:FS01:0x16:Completed   1000000 out of 50000000 steps (2%).
21:48:54:WU01:FS01:0x16:Completed   1500000 out of 50000000 steps (3%).
22:18:25:WU01:FS01:0x16:Completed   2000000 out of 50000000 steps (4%).
22:47:34:WU01:FS01:0x16:Completed   2500000 out of 50000000 steps (5%).
23:16:03:WU01:FS01:0x16:Completed   3000000 out of 50000000 steps (6%).
23:44:53:WU01:FS01:0x16:Completed   3500000 out of 50000000 steps (7%).
00:14:04:WU01:FS01:0x16:Completed   4000000 out of 50000000 steps (8%).

This lovely Core 16 WU will take 4+ days to complete assuming it actually finishes. So it appears even single GPU systems aren't immune to the dreaded Core 16's.

Regarding you thinking about stopping folding all together, well you aren't alone. I've only been at this for about 5 months but I've thrown a lot of hardware at it (8 machines at the moment) so I'm in the top 200 overall donors, but it has been one non-stop battle. A battle that I'm now questioning and my wife has had it with. She called me one day while I was at work to tell me she was going to take a hammer to the computers. She could tell by the sounds they had crashed and knew what I would be doing when I home. In any event, I'm going to fight it a bit longer but if the Core 16's last much longer, I'll have to throw in the towel and move on as well.

[Dr.G]

@ ChasingTheDream

Sorry to hear your news. I know the feeling! I've spent a lot of money and time getting my systems up and running and maintaining them. I mean we are chucking some of the best (home) hardware at this...

I think FAH is a very worthy cause, but there also has to be a limit. My machines have been 'banished' to the garage out of the family house.

I've seen your figures and they are impressive. You should continue for now. I'm going to keep going for a another few weeks and see how things go...

At the moment FAHcore 0x17 is back and my system is 'happy'.

I don't understand why the FAHcore 0x16's have presented with such a vengeance recently.
If you look at other folders and even teams overall (the top 10), I think many have 'taken a hit', with WU's completed down by up to a quarter (!). That's my observation anyway, but the site mod's may be able to quote actual figures (I'd love to know!).

I think the new 14.9 drivers are a little more efficient and temps are ok. (see pic).





http://i.imgur.com/Wnh9Aqw.jpg?2
http://imgur.com/Wnh9Aqw

[EXT64]

I'm pretty sure Core 16 coming back with a vengeance has corresponded with the new Assignment Server (and glitches in the AS configuration or code). I certainly understand the pain of Core 16 - I don't mind poor PPD now and then, but when they just keep crashing that is completely worthless to everyone.

A couple notes:

Linux can only get Core 17. If you want you could switch some systems to Linux.

And this should be a temporary problem. If you are still getting a ton of Core 16 work, you could shutdown temporarily while the new AS bugs are worked out (say a week).

[7im]

When the Server status page shows 50,000+ core 16 WUs available and ~4000 core 17 WUs, one has to ask if it's really a configuration issue or not.

According to that same page both core types are going out to clients, just not in the ratio most people would prefer.

And if some people can fold on multiple GPUs and some cannot, we need to figure out why some fail, either in temps, power, cooling, drivers, etc. When some fold core 16 okay and others don't, we need to find that localized problem and fix it.

[LonePalm]

Any chance of getting the Core 17 bonuses applied to the Core 16 units?

[ChasingTheDream]

@ Dr.G

Yes indeed. The time and money is significant and of limited supply so it would be nice to use it efficiently. I do believe F@H is a worthy cause as well, but limits of what I can reasonably do are being reached. Fortunately there are alternatives.

I'm sure production is down. Many people seem to be upset about the points and have probably stopped folding. I don't understand why the points can't be weighted equally across the cores to avoid peoples concerns though.

Due to my background (IT development for 27+ years) I was warned by other folders in a large forum that much of what I was going to see regarding the decisions made with the project would drive me insane. I thought they were exaggerating, but unfortunately they are right.

This is actually my second attempt at typing a message because my machines have locked three times each since my return home and they were down when I got home so the bottom line is that they really are just wasting power at this point. They aren't completing much of anything, so I've begun shutting them down. It also appears that each GPU is running a Core 16 at this point. So I'm guessing this will be another 10+ Core 16 WU's that will never be completed. There is nothing I can do about it though.

Obviously there is competition to F@H so I'm going to start looking into it. I don't know if it would be any better or worse but I clearly need to get additional information since my usefulness here is coming to an end.

@ EXT64

Yeah I agree. I don't mind poor PPD but I've got to be able to run the WU's or stop them from being sent. Having non-stop crashes is non-productive. The WU will never complete anyway so I should have the option to exclude the core. But that isn't the case and it doesn't sound like it will be.

Regarding Linux, I haven't done anything on Linux so that would be another large time investment and who knows how long it would bypass the issue. Thanks for the information though!

@ 7im

As I've said before this is the same type of system behavior I ran into with Core 17 WU's when I first started folding. There is a long thread on this forum about all the things that were suggested to fix it. I spent hundreds of hours reinstalling Windows from scratch after a thousand crashes, swapping GPU's, trying a million different clock speeds etc. Nothing worked because it was a driver issue all along and a new driver is what ultimately fixed it.

So the question is should I put in hundreds more hours to solve the Core 16 mysteries when the core reached its end of life a year ago. Should I risk the Core 17 stability I finally had to try to get Core 16's to run? The obvious answer to me is of course not.

You've said it yourself earlier in the thread. There is no way the client can address every possible combination of hardware out there. So wouldn't it make sense to give the users the ability to exclude core's that clearly won't run on their hardware? What happens when new cores are released and they won't run on some hardware until a driver update clears the issues? Do we just keep repeating this cycle over and over again?

I don't understand why the users are being put in this situation.

[7im]

Any chance of getting the Core 17 bonuses applied to the Core 16 units?

No, sorry.

[7im]

If this is a driver issue, why do some multi-GPU systems fold okay, and others do not?

Excluding new cores is not the same as excluding old cores. New cores are beta tested before rolling out to all clients. They are excluded by default.

Consider the alternatives. If fah provided a way to exclude specific cores or projects, some projects would never get done because they score less points. Not because they are of less value, but simply because they are from an older points system that still need to be completed. And it's not as simple as just adding more points, or adding a QRB if that code doesn't support it.

Or if FAH excluded all ATI cards from core_16, that would hurt more people than it helped, because most donors have single cards, not multiple cards.

With fah, helping a few is not without bigger consequence.

I am not unsympathetic to your situation. I have an AMD card that BSODs once in a blue moon for no reason. So my next upgrade won't be AMD. But not worth it to drop a dime now.

[PS3EdOlkkola]

@ChasingTheDream: I didn't recall seeing if you were running SMP in addition to GPU folding. If you are, then the issue might not be your GPU or drivers, but CPU cooling. Core 17 work units put a very light load on the CPU, whereas Core 16 puts a fairly substantial load on the CPU. If you happen to have marginal CPU cooling, the extra stress caused by Core 16 might be pushing your system past it's thermal limits and causing a system reset. However, if you are running SMP work units alongside Core 17, you likely have adequate cooling; it's the non-SMP with Core 17 combination that might be masking a CPU cooling problem.

Just my $0.02

[ChasingTheDream]

@ 7im

All the multi-GPU systems that hard lock constantly have exactly the same hardware. Every single piece of them. I also have one older computer that only has two GPU's and it seems to handle things better. It is the only multi-GPU that even has a chance of running Core 16's. It still BSD's but not nearly as often.

Regarding the AMD cards... Yeah, I couldn't agree more! I really wish I wouldn't have done that but I saw the folding numbers for these cards at the time and thought they were going to be great at a lower price but I had no idea of the stability issues. These are the first (and last) AMD cards I've owned in years. Unfortunately, they are $10,000 worth of GPU's so it isn't like I can sell them and start over without taking a huge loss. I'll just have to do the best I can with what I got until they die.

I completely understand that things are never as easy as they appear.

I'm more interested in excluding cores than projects. Mainly because I'm fearful this process is going to repeat infinitely. A new core will be released at some point and I'm guessing my beloved AMD cards won't like it for a few months until a driver update addresses it.

Of course you are right about people exploiting it though.

[ChasingTheDream]

@ PS3EdOlkkola

Great thoughts!

Forgive me if I have this wrong but I'm assuming SMP folding means CPU folding through the F@H Control client. If that is the case then I do CPU fold via the client with a reduced number of threads. My CPU's are liquid cooled though and the digital readout is always visible to me on the motherboard. I've never seen a CPU temp higher than 60 C and most are under 50 C so I should be ok on temps don't you think?

[bruce]

If you've griped about getting Core_16 (or you plan to make such a report) and your post does not include the type of GPU you're running, your report is incomplete and can't be used. (Reports which include that information need not be edited.)

Some of the issues have been fixed; some have not. I'm having a terrible time trying to figure out how to categorize the reports.

[ChasingTheDream]

@ Bruce

Sorry but I haven't made any reports other than what was said in this thread. So if there are reports with missing GPU's I can assure you they aren't from me. The logs I've attached in this thread though I believe show the GPU's. They are all R9 290X TRI-X cards.