Can't get two (different) GPUs to work

[NathanH]

quick update for anyone paying attention: i ended up pulling the second video card, the 8600gts, and i am up and folding. using the latest 310.90 driver on a GTX 560 sc version. i would still like to find a dual card solution, none of the config.xml edits i tried worked. i was using the other card to output to my (analogue) tv. thanks for all the help.

[bruce]

Your problem isn't solved yet, so: Yes, we're still paying attention.

I'd like to see a fresh copy of the log showing the config the way you have it today plus the client trying to start up all three of the slots. Also, download FAHBench (See this topic) and run it in the text window. Post the results of FAHBench.exe --display-devices --disable-splash.

It's likely that you'll have to download a newer version of CUDA.

If you choose to benchmark each device on each platform, that's up to you.

[NathanH]

thank you. i realize that last post sounds ..snarky? sorry about that, just seemed like a dead thread. any way here is the log as of 15 mins of running.

Code: Select all

*********************** Log Started 2013-02-27T08:21:06Z ***********************
08:21:06:************************* Folding@home Client *************************
08:21:06:      Website: http://folding.stanford.edu/
08:21:06:    Copyright: (c) 2009-2012 Stanford University
08:21:06:       Author: Joseph Coffland <[email protected]>
08:21:06:         Args: --lifeline 2780 --command-port=36330
08:21:06:       Config: C:/Users/Nathan/FAHClient/config.xml
08:21:06:******************************** Build ********************************
08:21:06:      Version: 7.2.9
08:21:06:         Date: Oct 3 2012
08:21:06:         Time: 18:05:48
08:21:06:      SVN Rev: 3578
08:21:06:       Branch: fah/trunk/client
08:21:06:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
08:21:06:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
08:21:06:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
08:21:06:     Platform: win32 XP
08:21:06:         Bits: 32
08:21:06:         Mode: Release
08:21:06:******************************* System ********************************
08:21:06:          CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz
08:21:06:       CPU ID: GenuineIntel Family 6 Model 15 Stepping 11
08:21:06:         CPUs: 2
08:21:06:       Memory: 8.00GiB
08:21:06:  Free Memory: 6.45GiB
08:21:06:      Threads: WINDOWS_THREADS
08:21:06:   On Battery: false
08:21:06:   UTC offset: -6
08:21:06:          PID: 3824
08:21:06:          CWD: C:/Users/Nathan/FAHClient
08:21:06:           OS: Windows 7 Ultimate
08:21:06:      OS Arch: AMD64
08:21:06:         GPUs: 1
08:21:06:        GPU 0: NVIDIA:2 GF114 [GeForce GTX 560]
08:21:06:         CUDA: 2.1
08:21:06:  CUDA Driver: 5000
08:21:06:Win32 Service: false
08:21:06:***********************************************************************
08:21:06:<config>
08:21:06:  <!-- Network -->
08:21:06:  <proxy v=':8080'/>
08:21:06:
08:21:06:  <!-- User Information -->
08:21:06:  <passkey v='********************************'/>
08:21:06:  <team v='37726'/>
08:21:06:  <user v='Nathan'/>
08:21:06:
08:21:06:  <!-- Folding Slots -->
08:21:06:  <slot id='2' type='SMP'/>
08:21:06:  <slot id='0' type='GPU'/>
08:21:06:</config>
08:21:06:Trying to access database...
08:21:07:Successfully acquired database lock
08:21:07:Enabled folding slot 02: READY smp:2
08:21:07:Enabled folding slot 00: READY gpu:0:"GF114 [GeForce GTX 560]"
08:21:07:WU03:FS02:Starting
08:21:07:WU03:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 03 -suffix 01 -version 702 -lifeline 3824 -checkpoint 15 -np 2
08:21:07:WU03:FS02:Started FahCore on PID 1592
08:21:07:WU03:FS02:Core PID:2588
08:21:07:WU03:FS02:FahCore 0xa4 started
08:21:07:WU00:FS00:Starting
08:21:07:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 702 -lifeline 3824 -checkpoint 15 -gpu 0
08:21:07:WU00:FS00:Started FahCore on PID 4352
08:21:07:WU00:FS00:Core PID:4104
08:21:07:WU00:FS00:FahCore 0x15 started
08:21:08:WU03:FS02:0xa4:
08:21:08:WU03:FS02:0xa4:*------------------------------*
08:21:08:WU03:FS02:0xa4:Folding@Home Gromacs GB Core
08:21:08:WU03:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
08:21:08:WU03:FS02:0xa4:
08:21:08:WU03:FS02:0xa4:Preparing to commence simulation
08:21:08:WU03:FS02:0xa4:- Looking at optimizations...
08:21:08:WU03:FS02:0xa4:- Files status OK
08:21:08:WU03:FS02:0xa4:- Expanded 2079373 -> 5386224 (decompressed 259.0 percent)
08:21:08:WU03:FS02:0xa4:Called DecompressByteArray: compressed_data_size=2079373 data_size=5386224, decompressed_data_size=5386224 diff=0
08:21:08:WU03:FS02:0xa4:- Digital signature verified
08:21:08:WU03:FS02:0xa4:
08:21:08:WU03:FS02:0xa4:Project: 7809 (Run 0, Clone 169, Gen 26)
08:21:08:WU03:FS02:0xa4:
08:21:08:WU03:FS02:0xa4:Assembly optimizations on if available.
08:21:08:WU03:FS02:0xa4:Entering M.D.
08:21:08:WU00:FS00:0x15:
08:21:08:WU00:FS00:0x15:*------------------------------*
08:21:08:WU00:FS00:0x15:Folding@Home GPU Core
08:21:08:WU00:FS00:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
08:21:08:WU00:FS00:0x15:Build host             AmoebaRemote
08:21:08:WU00:FS00:0x15:Board Type             NVIDIA/CUDA
08:21:08:WU00:FS00:0x15:Core                   15
08:21:08:WU00:FS00:0x15:
08:21:08:WU00:FS00:0x15:Window's signal control handler registered.
08:21:08:WU00:FS00:0x15:Preparing to commence simulation
08:21:08:WU00:FS00:0x15:- Looking at optimizations...
08:21:08:WU00:FS00:0x15:- Files status OK
08:21:08:WU00:FS00:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
08:21:08:WU00:FS00:0x15:- Expanded 124329 -> 502918 (decompressed 404.5 percent)
08:21:08:WU00:FS00:0x15:Called DecompressByteArray: compressed_data_size=124329 data_size=502918, decompressed_data_size=502918 diff=0
08:21:08:WU00:FS00:0x15:- Digital signature verified
08:21:08:WU00:FS00:0x15:
08:21:08:WU00:FS00:0x15:Project: 7625 (Run 794, Clone 1, Gen 3)
08:21:08:WU00:FS00:0x15:
08:21:08:WU00:FS00:0x15:Assembly optimizations on if available.
08:21:08:WU00:FS00:0x15:Entering M.D.
08:21:09:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
08:21:10:WU00:FS00:0x15:Will resume from checkpoint file 00/wudata_01.ckp
08:21:10:WU00:FS00:0x15:Tpr hash 00/wudata_01.tpr:  2683096403 734510686 3981381824 3540871486 523860424
08:21:10:WU00:FS00:0x15:GPU device id=0
08:21:10:WU00:FS00:0x15:Working on Protein
08:21:10:WU00:FS00:0x15:Client config unavailable.
08:21:10:WU00:FS00:0x15:Starting GUI Server
08:21:14:WU03:FS02:0xa4:Using Gromacs checkpoints
08:21:14:WU03:FS02:0xa4:Mapping NT from 2 to 2 
08:21:15:WU03:FS02:0xa4:Resuming from checkpoint
08:21:15:WU03:FS02:0xa4:Verified 03/wudata_01.log
08:21:15:WU03:FS02:0xa4:Verified 03/wudata_01.trr
08:21:15:WU03:FS02:0xa4:Verified 03/wudata_01.xtc
08:21:15:WU03:FS02:0xa4:Verified 03/wudata_01.edr
08:21:16:WU03:FS02:0xa4:Completed 796700 out of 1500000 steps  (53%)
08:22:18:WU00:FS00:0x15:Resuming from checkpoint
08:22:18:WU00:FS00:0x15:fcCheckPointResume: retreived and current tpr file hash:
08:22:18:WU00:FS00:0x15:   0   2683096403   2683096403
08:22:18:WU00:FS00:0x15:   1    734510686    734510686
08:22:18:WU00:FS00:0x15:   2   3981381824   3981381824
08:22:18:WU00:FS00:0x15:   3   3540871486   3540871486
08:22:18:WU00:FS00:0x15:   4    523860424    523860424
08:22:18:WU00:FS00:0x15:fcCheckPointResume: file hashes same.
08:22:18:WU00:FS00:0x15:fcCheckPointResume: state restored.
08:22:18:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.log Verified 00/wudata_01.log
08:22:18:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.trr Verified 00/wudata_01.trr
08:22:18:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.xtc Verified 00/wudata_01.xtc
08:22:18:WU00:FS00:0x15:fcCheckPointResume: name 00/wudata_01.edr Verified 00/wudata_01.edr
08:22:18:WU00:FS00:0x15:fcCheckPointResume: state restored 2
08:22:18:WU00:FS00:0x15:Resumed from checkpoint
08:22:18:WU00:FS00:0x15:Setting checkpoint frequency: 400000
08:22:18:WU00:FS00:0x15:Completed   2800001 out of 40000000 steps (7%).
08:22:19:WARNING:WU00:FS00:Detected clock skew (1 mins 11 secs), adjusting time estimates
08:30:58:WU00:FS00:0x15:Completed   3200000 out of 40000000 steps (8%).

seems to be fine with just the one gpu. after pulling the second card i fired up FAHControl and manually inserted a gpu slot, it found the correct gpu right away and started folding. I will get back to you with the FAHBench results. Thanks again.

[bruce]

One GPU running FahCore_15 and one Dual-CPU running FahCore_a4 on two CPUs. Everything looks normal. Now what happens if you try re-adding the other GPU ... or has that become a non-problem?

[NathanH]

i have been busy, but up and folding on the one gpu. (and smp) which has more than 2x the ppd than the last time i was folding on 3 gpus.. so pretty happy with that. i was thinking you might want to change the name of the thread to two (different) gpu's? i think the sli guys and gals are not having the issues i have been having. i will have some time today to put the other gpu back in and see what we see. i will be trying the flip-flop that had been suggested earlier. a side thought: i am now running on the 306.97 driver. i don't think it matters though, i was folding on both this one and the 310.90. just fyi.

[NathanH]

so i just had a quick thought as i'm putting the other gpu in: do i still need the 'dummy plug?' dvi to vga adapter with some resistors tricking the driver to think there is a monitor.. now i'm thinking that is my whole problem. i have the second card outputing on the s-video (looks like s-video, whatever they call it) breakout dongle to composite.. maybe it can't do two monitors off one card. or i have to trick it into thinking there are 3 monitors, two DVI and the 's-video.' so i still have my analogue output, thoughts?

[Joe_H]

From what I understand, using a dummy plug is no longer needed. If you have one, it shouldn't hurt to try with that attached to see if it makes any difference.

[bruce]

No dummy plug is needed.

There are several important reasons why two different GPUs is harder.
1) V7 uses a different method of enumerating GPUs than Windows does. If the GPUs are the same and Windows thinks they're numbered 0,1 when V7 has really numbered them 1,0, it doesn't matter because they'll still both work.
2) Drivers have been causing a lot of problems, and getting two DIFFERENT sets of drivers to work correctly is several times more difficult than just getting one set to work.
3) You can have both a CUDA GPU 0 and an OpenCL GPU 0.
4) V7's "automatic" multi-GPU configuration process sometimes works and sometimes does not. If you try to help it by making changes, you may actually be making it more difficult.
5) Nobody actually understands all of the possible reasons why a configuration can fail.

[P5-133XL]

5) Nobody actually understands all of the possible reasons why a configuration can fail.

So very true!

[coredump4]

quick update for anyone paying attention: i ended up pulling the second video card, the 8600gts, and i am up and folding. using the latest 310.90 driver on a GTX 560 sc version. i would still like to find a dual card solution, none of the config.xml edits i tried worked. i was using the other card to output to my (analogue) tv. thanks for all the help.

I recently setup an old board that has a GeForce 9400 IGP and added a GTX 560 Ti to it for some heavy lifting. With the 310.90 driver, I could not get folding to work properly on both GPUs, and kept getting the GPU_MEMTEST_ERROR errors until I disabled the 9400 GPU. In the end, I decided the 9400's contribution would be too small to try to solve this conflict, and am going to stick with just the 560 Ti. This sounds similar to what you found in your system.

[bruce]

Did you reinstall drivers AFTER both GPUs were present? A driver install for Fermi does not provide the necessary drivers for a G80/G90 without the opportunity to add code that wasn't needed when you just had a Fermi.

Is the log you posted above when both were supposedly working? I'd guess that you might have a bios that does not support more than one GPU. (I have one of those and I've never figured out what to do about it.)

Where is the output that I asked for (above) while running FAHBench.exe --display-devices --disable-splash?.

[NathanH]

yes i re-installed 306.97 after putting the second card back in. i am getting the 'hang' again. my gpu_memtest_error has disappeared but it just sits there. no progress. i have had 3 cards on that same motherboard same cpu all folding. that was several years ago. 2x 8800gts and an 8600gts. in sli. now i have two completely different gen cards both working, but won't fold together. no sli. i guess i will end up pulling it. the extra ppd would be nice tho;)
last night's log after hanging for hours:

Code: Select all

********************** Log Started 2013-03-02T14:12:12Z ***********************
14:12:12:************************* Folding@home Client *************************
14:12:12:      Website: http://folding.stanford.edu/
14:12:12:    Copyright: (c) 2009-2012 Stanford University
14:12:12:       Author: Joseph Coffland <[email protected]>
14:12:12:         Args: --lifeline 5452 --command-port=36330
14:12:12:       Config: C:/Users/Nathan/FAHClient/config.xml
14:12:12:******************************** Build ********************************
14:12:12:      Version: 7.2.9
14:12:12:         Date: Oct 3 2012
14:12:12:         Time: 18:05:48
14:12:12:      SVN Rev: 3578
14:12:12:       Branch: fah/trunk/client
14:12:12:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
14:12:12:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
14:12:12:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
14:12:12:     Platform: win32 XP
14:12:12:         Bits: 32
14:12:12:         Mode: Release
14:12:12:******************************* System ********************************
14:12:12:          CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz
14:12:12:       CPU ID: GenuineIntel Family 6 Model 15 Stepping 11
14:12:12:         CPUs: 2
14:12:12:       Memory: 8.00GiB
14:12:12:  Free Memory: 5.96GiB
14:12:12:      Threads: WINDOWS_THREADS
14:12:12:   On Battery: false
14:12:12:   UTC offset: -6
14:12:12:          PID: 4656
14:12:12:          CWD: C:/Users/Nathan/FAHClient
14:12:12:           OS: Windows 7 Ultimate
14:12:12:      OS Arch: AMD64
14:12:12:         GPUs: 2
14:12:12:        GPU 0: NVIDIA:1 G80 [GeForce 8800 GTS]
14:12:12:        GPU 1: NVIDIA:2 GF114 [GeForce GTX 560]
14:12:12:         CUDA: 2.1
14:12:12:  CUDA Driver: 5000
14:12:12:Win32 Service: false
14:12:12:***********************************************************************
14:12:13:<config>
14:12:13:  <!-- Network -->
14:12:13:  <proxy v=':8080'/>
14:12:13:
14:12:13:  <!-- User Information -->
14:12:13:  <passkey v='********************************'/>
14:12:13:  <team v='37726'/>
14:12:13:  <user v='Nathan'/>
14:12:13:
14:12:13:  <!-- Folding Slots -->
14:12:13:  <slot id='2' type='SMP'/>
14:12:13:  <slot id='0' type='GPU'/>
14:12:13:  <slot id='1' type='GPU'/>
14:12:13:</config>
14:12:13:Trying to access database...
14:12:13:Successfully acquired database lock
14:12:13:Enabled folding slot 02: READY smp:2
14:12:13:Enabled folding slot 00: READY gpu:0:"G80 [GeForce 8800 GTS]"
14:12:13:Enabled folding slot 01: READY gpu:1:"GF114 [GeForce GTX 560]"
14:12:13:WU00:FS01:Starting
14:12:13:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 702 -lifeline 4656 -checkpoint 15 -gpu 1
14:12:13:WU00:FS01:Started FahCore on PID 5256
14:12:13:WU00:FS01:Core PID:6084
14:12:13:WU00:FS01:FahCore 0x15 started
14:12:13:WU03:FS02:Starting
14:12:13:WU03:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 03 -suffix 01 -version 702 -lifeline 4656 -checkpoint 15 -np 2
14:12:13:WU03:FS02:Started FahCore on PID 2920
14:12:14:WU03:FS02:Core PID:5904
14:12:14:WU03:FS02:FahCore 0xa4 started
14:12:14:WU00:FS01:0x15:
14:12:14:WU00:FS01:0x15:*------------------------------*
14:12:14:WU00:FS01:0x15:Folding@Home GPU Core
14:12:14:WU00:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
14:12:14:WU00:FS01:0x15:Build host             AmoebaRemote
14:12:14:WU00:FS01:0x15:Board Type             NVIDIA/CUDA
14:12:14:WU00:FS01:0x15:Core                   15
14:12:14:WU00:FS01:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=1
14:12:14:WU00:FS01:0x15:
14:12:14:WU00:FS01:0x15:Window's signal control handler registered.
14:12:14:WU00:FS01:0x15:Preparing to commence simulation
14:12:14:WU00:FS01:0x15:- Looking at optimizations...
14:12:14:WU00:FS01:0x15:- Files status OK
14:12:14:WU00:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
14:12:14:WU00:FS01:0x15:- Expanded 59067 -> 261034 (decompressed 441.9 percent)
14:12:14:WU00:FS01:0x15:Called DecompressByteArray: compressed_data_size=59067 data_size=261034, decompressed_data_size=261034 diff=0
14:12:14:WU00:FS01:0x15:- Digital signature verified
14:12:14:WU00:FS01:0x15:
14:12:14:WU00:FS01:0x15:Project: 8043 (Run 0, Clone 1104, Gen 44)
14:12:14:WU00:FS01:0x15:
14:12:14:WU00:FS01:0x15:Assembly optimizations on if available.
14:12:14:WU00:FS01:0x15:Entering M.D.
14:12:14:WU01:FS00:Starting
14:12:14:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 702 -lifeline 4656 -checkpoint 15 -gpu 0
14:12:14:WU01:FS00:Started FahCore on PID 5388
14:12:14:WU01:FS00:Core PID:6132
14:12:14:WU01:FS00:FahCore 0x11 started
14:12:14:WU03:FS02:0xa4:
14:12:14:WU03:FS02:0xa4:*------------------------------*
14:12:14:WU03:FS02:0xa4:Folding@Home Gromacs GB Core
14:12:14:WU03:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
14:12:14:WU03:FS02:0xa4:
14:12:14:WU03:FS02:0xa4:Preparing to commence simulation
14:12:14:WU03:FS02:0xa4:- Looking at optimizations...
14:12:14:WU03:FS02:0xa4:- Files status OK
14:12:14:WU03:FS02:0xa4:- Expanded 2079373 -> 5386224 (decompressed 259.0 percent)
14:12:14:WU03:FS02:0xa4:Called DecompressByteArray: compressed_data_size=2079373 data_size=5386224, decompressed_data_size=5386224 diff=0
14:12:14:WU03:FS02:0xa4:- Digital signature verified
14:12:14:WU03:FS02:0xa4:
14:12:14:WU03:FS02:0xa4:Project: 7809 (Run 0, Clone 169, Gen 26)
14:12:14:WU03:FS02:0xa4:
14:12:14:WU03:FS02:0xa4:Assembly optimizations on if available.
14:12:14:WU03:FS02:0xa4:Entering M.D.
14:12:15:WU01:FS00:0x11:
14:12:15:WU01:FS00:0x11:*------------------------------*
14:12:15:WU01:FS00:0x11:Folding@Home GPU Core
14:12:15:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
14:12:15:WU01:FS00:0x11:
14:12:15:WU01:FS00:0x11:Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
14:12:15:WU01:FS00:0x11:Build host: amoeba
14:12:15:WU01:FS00:0x11:Board Type: Nvidia
14:12:15:WU01:FS00:0x11:Core      : 
14:12:15:WU01:FS00:0x11:Preparing to commence simulation
14:12:15:WU01:FS00:0x11:- Ensuring status. Please wait.
14:12:15:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
14:12:16:WU00:FS01:0x15:Tpr hash 00/wudata_01.tpr:  3229035939 29513159 2575880245 2729909659 1478953511
14:12:16:WU00:FS01:0x15:GPU device id=1
14:12:16:WU00:FS01:0x15:Working on GROningen MAchine for Chemical Simulation
14:12:16:WU00:FS01:0x15:Client config unavailable.
14:12:16:WU00:FS01:0x15:Starting GUI Server
14:12:21:WU03:FS02:0xa4:Using Gromacs checkpoints
14:12:21:WU03:FS02:0xa4:Mapping NT from 2 to 2 
14:12:22:WU03:FS02:0xa4:Resuming from checkpoint
14:12:22:WU03:FS02:0xa4:Verified 03/wudata_01.log
14:12:22:WU03:FS02:0xa4:Verified 03/wudata_01.trr
14:12:22:WU03:FS02:0xa4:Verified 03/wudata_01.xtc
14:12:22:WU03:FS02:0xa4:Verified 03/wudata_01.edr
14:12:23:WU03:FS02:0xa4:Completed 1359010 out of 1500000 steps  (90%)
14:12:24:WU01:FS00:0x11:- Looking at optimizations...
14:12:24:WU01:FS00:0x11:- Working with standard loops on this execution.
14:12:24:WU01:FS00:0x11:- Previous termination of core was improper.
14:12:24:WU01:FS00:0x11:- Files status OK
14:12:24:WU01:FS00:0x11:- Expanded 46752 -> 252912 (decompressed 540.9 percent)
14:12:24:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=46752 data_size=252912, decompressed_data_size=252912 diff=0
14:12:24:WU01:FS00:0x11:- Digital signature verified
14:12:24:WU01:FS00:0x11:
14:12:24:WU01:FS00:0x11:Project: 5765 (Run 8, Clone 335, Gen 3872)
14:12:24:WU01:FS00:0x11:
14:12:24:WU01:FS00:0x11:Entering M.D.
14:12:30:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr:  2239637262 3460662241 3914730317 3544561212 560043258
14:12:30:WU01:FS00:0x11:
14:12:30:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
14:12:30:WU01:FS00:0x11:
14:35:41:WU03:FS02:0xa4:Completed 1365000 out of 1500000 steps  (91%)
15:32:08:WU03:FS02:0xa4:Completed 1380000 out of 1500000 steps  (92%)
16:28:40:WU03:FS02:0xa4:Completed 1395000 out of 1500000 steps  (93%)
17:25:11:WU03:FS02:0xa4:Completed 1410000 out of 1500000 steps  (94%)
19:29:40:WU03:FS02:0xa4:Completed 1425000 out of 1500000 steps  (95%)
******************************** Date: 02/03/13 ********************************
20:50:49:WU03:FS02:0xa4:Completed 1440000 out of 1500000 steps  (96%)
21:52:53:WU03:FS02:0xa4:Completed 1455000 out of 1500000 steps  (97%)
23:05:39:WU03:FS02:0xa4:Completed 1470000 out of 1500000 steps  (98%)
00:44:23:WU03:FS02:0xa4:Completed 1485000 out of 1500000 steps  (99%)
00:44:23:WU02:FS02:Connecting to assign3.stanford.edu:8080
00:44:24:WU02:FS02:News: Welcome to Folding@Home
00:44:24:WU02:FS02:Assigned to work server 171.64.65.99
00:44:24:WU02:FS02:Requesting new work unit for slot 02: RUNNING smp:2 from 171.64.65.99
00:44:24:WU02:FS02:Connecting to 171.64.65.99:8080
00:44:25:WU02:FS02:Downloading 1.96MiB
00:44:29:WU02:FS02:Download complete
00:44:29:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:7808 run:1 clone:453 gen:28 core:0xa4 unit:0x000000240a3b1e874e30f32686f4149f
00:44:30:WU02:FS02:Downloading project 7808 description
00:44:30:WU02:FS02:Connecting to fah-web.stanford.edu:80
00:44:30:WU02:FS02:Project 7808 description downloaded successfully
01:53:45:WU03:FS02:0xa4:Completed 1500000 out of 1500000 steps  (100%)
01:53:46:WU03:FS02:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
01:53:56:WU03:FS02:0xa4:
01:53:56:WU03:FS02:0xa4:Finished Work Unit:
01:53:56:WU03:FS02:0xa4:- Reading up to 2908800 from "03/wudata_01.trr": Read 2908800
01:53:56:WU03:FS02:0xa4:trr file hash check passed.
01:53:56:WU03:FS02:0xa4:- Reading up to 1554468 from "03/wudata_01.xtc": Read 1554468
01:53:56:WU03:FS02:0xa4:xtc file hash check passed.
01:53:56:WU03:FS02:0xa4:edr file hash check passed.
01:53:56:WU03:FS02:0xa4:logfile size: 46307
01:53:56:WU03:FS02:0xa4:Leaving Run
01:53:57:WU03:FS02:0xa4:- Writing 4514587 bytes of core data to disk...
01:53:59:WU03:FS02:0xa4:Done: 4514075 -> 4327241 (compressed to 95.8 percent)
01:53:59:WU03:FS02:0xa4:  ... Done.
01:54:01:WU03:FS02:0xa4:- Shutting down core
01:54:01:WU03:FS02:0xa4:
01:54:01:WU03:FS02:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
01:54:02:WU03:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:54:03:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:7809 run:0 clone:169 gen:26 core:0xa4 unit:0x000000210a3b1e874e3105370beccd42
01:54:03:WU03:FS02:Uploading 4.13MiB to 171.64.65.99
01:54:03:WU02:FS02:Starting
01:54:03:WU03:FS02:Connecting to 171.64.65.99:8080
01:54:03:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 702 -lifeline 4656 -checkpoint 15 -np 2
01:54:03:WU02:FS02:Started FahCore on PID 4464
01:54:03:WU02:FS02:Core PID:4884
01:54:03:WU02:FS02:FahCore 0xa4 started
01:54:03:WU02:FS02:0xa4:
01:54:03:WU02:FS02:0xa4:*------------------------------*
01:54:03:WU02:FS02:0xa4:Folding@Home Gromacs GB Core
01:54:03:WU02:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
01:54:03:WU02:FS02:0xa4:
01:54:03:WU02:FS02:0xa4:Preparing to commence simulation
01:54:03:WU02:FS02:0xa4:- Looking at optimizations...
01:54:03:WU02:FS02:0xa4:- Created dyn
01:54:03:WU02:FS02:0xa4:- Files status OK
01:54:04:WU02:FS02:0xa4:- Expanded 2053848 -> 5365960 (decompressed 261.2 percent)
01:54:04:WU02:FS02:0xa4:Called DecompressByteArray: compressed_data_size=2053848 data_size=5365960, decompressed_data_size=5365960 diff=0
01:54:04:WU02:FS02:0xa4:- Digital signature verified
01:54:04:WU02:FS02:0xa4:
01:54:04:WU02:FS02:0xa4:Project: 7808 (Run 1, Clone 453, Gen 28)
01:54:04:WU02:FS02:0xa4:
01:54:04:WU02:FS02:0xa4:Assembly optimizations on if available.
01:54:04:WU02:FS02:0xa4:Entering M.D.
01:54:09:WU03:FS02:Upload 19.69%
01:54:10:WU02:FS02:0xa4:Mapping NT from 2 to 2 
01:54:11:WU02:FS02:0xa4:Completed 0 out of 1500000 steps  (0%)
01:54:17:WU03:FS02:Upload 24.23%
01:54:23:WU03:FS02:Upload 31.80%
01:54:29:WU03:FS02:Upload 40.89%
01:54:35:WU03:FS02:Upload 48.46%
01:54:41:WU03:FS02:Upload 57.54%
01:54:47:WU03:FS02:Upload 65.12%
01:54:53:WU03:FS02:Upload 74.20%
01:54:59:WU03:FS02:Upload 81.77%
01:55:05:WU03:FS02:Upload 89.34%
01:55:12:WU03:FS02:Upload 96.92%
01:55:23:WU03:FS02:Upload complete
01:55:23:WU03:FS02:Server responded WORK_ACK (400)
01:55:23:WU03:FS02:Final credit estimate, 2730.00 points
01:55:23:WU03:FS02:Cleaning up
******************************** Date: 03/03/13 ********************************
03:37:40:WU02:FS02:0xa4:Completed 15000 out of 1500000 steps  (1%)
04:21:14:FS02:Paused
04:21:14:FS02:Shutting core down
04:21:24:WU02:FS02:0xa4:Client no longer detected. Shutting down core 
04:21:24:WU02:FS02:0xa4:
04:21:24:WU02:FS02:0xa4:Folding@home Core Shutdown: CLIENT_DIED
04:21:24:WU02:FS02:FahCore returned: INTERRUPTED (102 = 0x66)
04:21:32:FS00:Paused
04:21:32:FS00:Shutting core down
04:22:08:FS01:Paused
04:22:08:FS01:Shutting core down
04:22:09:WU00:FS01:0x15:Client no longer detected. Shutting down core 
04:22:09:WU00:FS01:0x15:
04:22:09:WU00:FS01:0x15:Folding@home Core Shutdown: CLIENT_DIED
04:22:09:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
04:22:33:WARNING:FS00:Killing WU01
04:22:33:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
04:23:02:WARNING:WU01:Slot ID 0 no longer exists and there are no other matching slots, dumping
04:23:02:WU01:Sending unit results: id:01 state:SEND error:DUMPED project:5765 run:8 clone:335 gen:3872 core:0x11 unit:0x17e19d755130d34c0f20014f00081685
04:23:02:WARNING:WU01:Work server too old for dump report
04:23:03:WU01:Cleaning up
04:23:03:Saving configuration to config.xml
04:23:03:<config>
04:23:03:  <!-- Network -->
04:23:03:  <proxy v=':8080'/>
04:23:03:
04:23:03:  <!-- User Information -->
04:23:03:  <passkey v='********************************'/>
04:23:03:  <team v='37726'/>
04:23:03:  <user v='Nathan'/>
04:23:03:
04:23:03:  <!-- Folding Slots -->
04:23:03:  <slot id='2' type='SMP'/>
04:23:03:  <slot id='1' type='GPU'/>
04:23:03:</config>
04:23:07:FS01:Unpaused
04:23:08:WU00:FS01:Starting
04:23:08:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Nathan/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_15.fah/FahCore_15.exe -dir 00 -suffix 01 -version 702 -lifeline 4656 -checkpoint 15 -gpu 1
04:23:08:WU00:FS01:Started FahCore on PID 5036
04:23:08:WU00:FS01:Core PID:1912
04:23:08:WU00:FS01:FahCore 0x15 started
04:23:08:WU00:FS01:0x15:
04:23:08:WU00:FS01:0x15:*------------------------------*
04:23:08:WU00:FS01:0x15:Folding@Home GPU Core
04:23:08:WU00:FS01:0x15:Version                2.25 (Wed May 9 17:03:01 EDT 2012)
04:23:08:WU00:FS01:0x15:Build host             AmoebaRemote
04:23:08:WU00:FS01:0x15:Board Type             NVIDIA/CUDA
04:23:08:WU00:FS01:0x15:Core                   15
04:23:08:WU00:FS01:0x15:GPU device info vendor=0 device=0 name=NA match=0 deviceId=1
04:23:08:WU00:FS01:0x15:
04:23:08:WU00:FS01:0x15:Window's signal control handler registered.
04:23:08:WU00:FS01:0x15:Preparing to commence simulation
04:23:08:WU00:FS01:0x15:- Looking at optimizations...
04:23:08:WU00:FS01:0x15:- Files status OK
04:23:08:WU00:FS01:0x15:sizeof(CORE_PACKET_HDR) = 512 file=<>
04:23:08:WU00:FS01:0x15:- Expanded 59067 -> 261034 (decompressed 441.9 percent)
04:23:08:WU00:FS01:0x15:Called DecompressByteArray: compressed_data_size=59067 data_size=261034, decompressed_data_size=261034 diff=0
04:23:08:WU00:FS01:0x15:- Digital signature verified
04:23:08:WU00:FS01:0x15:
04:23:08:WU00:FS01:0x15:Project: 8043 (Run 0, Clone 1104, Gen 44)
04:23:08:WU00:FS01:0x15:
04:23:08:WU00:FS01:0x15:Assembly optimizations on if available.
04:23:08:WU00:FS01:0x15:Entering M.D.
04:23:10:WU00:FS01:0x15:Tpr hash 00/wudata_01.tpr:  3229035939 29513159 2575880245 2729909659 1478953511
04:23:10:WU00:FS01:0x15:GPU device id=1
04:23:10:WU00:FS01:0x15:Working on GROningen MAchine for Chemical Simulation
04:23:10:WU00:FS01:0x15:Client config unavailable.
04:23:10:WU00:FS01:0x15:Starting GUI Server
04:28:13:FS01:Paused
04:28:13:FS01:Shutting core down
04:28:18:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
04:28:33:Lost lifeline PID 5452, exiting
04:28:34:Server connection id=1 ended
04:28:34:Clean exit

i tried removing one of the gpu slots but no love. i will just pull it when i get an hdmi to composite adaptater. then i can get back to folding 24000ppd. yea! *afterthought: if i could get the second one up and running ill go for 3 cards again! that would be more than 30k.

[JimF]

NathanH,

I have seen many similar problems, and you shouldn't think it is peculiar to your setup. In most cases, the dissimilar cards (whether different Nvidia types or mixed Nvidia/AMD) won't fold at all. I haven't even bothered to post on those, at least not recently, but in some cases they will fold with reduced capability.

This is my most recent go-around:
viewtopic.php?f=38&t=23179&p=237118#p236914

The original version of the problem:
viewtopic.php?f=38&t=20895&p=208915

The V6 solution works in some cases, often not:
viewtopic.php?f=16&t=23526&p=234636#p234636

To make a long story short, Folding does not handle dissimilar cards well.
(However, my GTX 560 and 650 TI are both working fine on GPUGrid.)

[art_l_j_PlanetAMD64]

yes i re-installed 306.97 after putting the second card back in. i am getting the 'hang' again. my gpu_memtest_error has disappeared but it just sits there. no progress. i have had 3 cards on that same motherboard same cpu all folding. that was several years ago. 2x 8800gts and an 8600gts. in sli. now i have two completely different gen cards both working, but won't fold together. no sli. i guess i will end up pulling it. the extra ppd would be nice tho;)

...

i tried removing one of the gpu slots but no love. i will just pull it when i get an hdmi to composite adapter. then i can get back to folding 24000ppd. yea! *afterthought: if i could get the second one up and running ill go for 3 cards again! that would be more than 30k.

Something I just noticed, is that all of these cases, where a "new" GPU is paired with an "older" GPU, will result in a FahCore_11.exe process and a FahCore_15.exe process both running in the same process tree. What if it turns out that this is an 'incompatible' situation, that cannot be resolved by changing 'settings'?

In order to test this out, I decided to try the following:

My #6 system currently has this configuration:

1. An smp:6 slot folding Core 0xa4, using CORE1+CORE2+CORE3+CORE4+CORE5+CORE6
2. Two GPU slots (both GTX 660 Ti OC's) folding Core 0x15, using CORE0+CORE7

I will try this out, in my #6 system:

1. An smp:6 slot folding Core 0xa4, using CORE1+CORE2+CORE3+CORE4+CORE5+CORE6
2. One GPU slot (GTX 660 Ti OC) folding Core 0x15, using CORE0
3. One GPU slot (GTX 285) folding Core 0x11, using CORE7

...

EDIT: No, it did not work, please see the following post for more information.

Hooray! It worked!

I have all 3 slots folding OK now:

1. Slot ID 00: GTX 285, PRCG 10504 (299, 1, 1209), 0x11, 678MHz, 49C, TPF 0:57, 8897.68 PPD (calculated from the TPF and Points/WU)
2. Slot ID 01: GTX 660 Ti, PRCG 8070 (100, 0, 45), 0x15, 1032MHz, 34C, TPF 2:10, 25747.20 PPD
3. Slot ID 02: smp:6, PRCG 7809 (10, 116, 32), 0xa4, 3900MHz, 36C, TPF 16:53, 7701.47 PPD

Total for this configuration is 42346 PPD.

Here are photos of the installation, first installing the GTX 285 in place of the GTX 660 Ti:
https://lh4.googleusercontent.com/-muFj ... C-108F.JPG

My #6 computer, ready to put back into its 'working' location, GTX 285 in the top slot (gpu0), GTX 660 Ti in the bottom slot (gpu1):
https://lh3.googleusercontent.com/-NsqG ... C-109F.JPG

My #6 computer, ready to start back up:
https://lh4.googleusercontent.com/-4BjB ... C-110F.JPG

-------------------------------------------------------

This works, by assigning specific CPU cores to each slot (or group of slots), that need to be kept separate (as the 0x11 and 0x15 cores do [need to be kept separate]). A batch file (*.bat) in each 'FAHClient' directory starts up an instance of FAHClient with the slot(s) that can be kept together. Relax, it's actually much easier to do, than to describe in words. Just please follow the instructions below.

So, Nathan, to 'cut to the chase', in order to get your "new" and "old" GPUs working together, you need to do this:

1. In FAHControl, you must be in either the 'Advanced' or 'Expert' mode, selected from the dropdown menu at the upper right
2. Click 'Pause' in FAHControl, and wait until all slots show Paused
3. Click on 'Configure', and select the 'Slots' tab
4. Click on the 'smp' slot to highlight it, then click on 'Edit'
5. In the 'SMP' part of the 'Configure folding slot' window, edit the 'CPUs' to be 1 less than your CPU's # of cores (eg 3 for a 4-core CPU)
6. Click on 'OK', then click on 'Save'
7. Click on 'Preferences', and deselect (uncheck the box for) 'Autostart' (FAHClient), then click on 'Save'
8. Click on 'Quit' to exit FAHControl and FAHClient

Now, you will need to 'Clone' your FAHClient directory, 'Edit' (only deleting lines) in the 2 'config.xml' files (1 in each directory), and create startup batch files with the correct instructions for starting each FAHClient instance with the correct CPU cores. Again, it's actually much easier to do, than it sounds.

To 'Clone' the FAHClient directory (assuming Windows 7), please do this: Using Windows Explorer, create a new directory (== 'folder') as C:\Users\<UserName>\AppData\Roaming\FAHClient2, and copy everything (all files, folders, etc.) from the original directory (C:\Users\<UserName>\AppData\Roaming\FAHClient) to this new directory.

Now, to 'Edit' the two 'config.xml' files (one in each directory), please do this: Decide which GPU type is to be made 'separate'. What this means, is that the SMP slot and one of the GPU types (either Core 0x11 or Core 0x15) can be kept together in one directory, the other GPU type (either Core 0x15 or Core 0x11) will be 'by itself' in the 'FAHClient2' directory. For example, let's say that you have Slot 00 = GTX 560 (Core 0x15), Slot 01 = 8600 GTS (Core 0x11), and Slot 02 = SMP. OK, so let's keep the SMP and "new" GPU together, and make the "old" GPU separate. To do this, please do the following:

• In 'FAHClient', edit 'config.xml' as follows:
  • Set the 'gpu' value to be 'false' (you must disable the GPU auto-detect)
  • Delete the lines with the GPU that will be separate (Slot 01 in this example)
  • Save the edited 'config.xml' file
• In 'FAHClient2', edit 'config.xml' as follows:
  • Set the 'gpu' value to be 'false' (you must disable the GPU auto-detect)
  • Delete the lines with the Slots that are used in 'FAHClient' (Slots 00 and 02 in this example)
  • Save the edited 'config.xml' file

Now, to create the startup batch files, please do this:

1. In 'FAHClient', please make a batch file "startfah1.bat", with this in it (for 64-bit Windows 7):

   Code: Select all

   start /AFFINITY 0E /b "" "C:\Program Files (x86)\FAHClient\FAHClient.exe" config.xml --command-port=36330

   For 32-bit Windows 7, it will be this:

   Code: Select all

   start /AFFINITY 0E /b "" "C:\Program Files\FAHClient\FAHClient.exe" config.xml --command-port=36330

   The part that you must change, is the '0E' just after the '/AFFINITY'. This is a 'bit mask' of the CPU cores to be used. '0E' hex (00001110 binary) is the value for a 4-core CPU (where you are using 3 of the 4 cores). This value would be '3E' for a 6-core CPU, or 'FE' for an 8-core CPU. Use all of the cores, except the least-significant bit (== Core 0, which is for the 'separate' GPU slot)
2. Save the batch file "startfah1.bat" in 'FAHClient'
3. Right-click on the batch file, and click on 'Create shortcut'
4. Rename the shortcut so it's just "startfah1"
5. Copy the shortcut 'startfah1' to the desktop, and to C:\Users\<UserName>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup
6. In 'FAHClient2', please make a batch file "startfah2.bat", with this in it (for 64-bit Windows 7):

   Code: Select all

   start /AFFINITY 01 /b "" "C:\Program Files (x86)\FAHClient\FAHClient.exe" config.xml --command-port=36331

   For 32-bit Windows 7, it will be this:

   Code: Select all

   start /AFFINITY 01 /b "" "C:\Program Files\FAHClient\FAHClient.exe" config.xml --command-port=36331

   This batch file will always be the same, as it uses Core 0 only. Also, please note that the '--command-port' is a different number than the one used in 'FAHClient'! This is important!
7. Save the batch file "startfah2.bat" in 'FAHClient2'
8. Right-click on the batch file, and click on 'Create shortcut'
9. Rename the shortcut so it's just "startfah2"
10. Copy the shortcut 'startfah2' to the desktop, and to C:\Users\<UserName>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Now, you can try out your new 'separate' GPU slot, by double-clicking on the 'startfah2' shortcut on the desktop. It should start up in a 'CMD' window, and display all of its log information (the same as you see in the log file). Do not close this window while folding, you can minimize it, though.

If that works OK, then double-click on the 'startfah1' shortcut on the desktop, to start up the SMP slot and the other GPU slot. They should start up in a 'CMD' window, and display all of their log information (the same as you see in the log file). Do not close this window while folding, you can minimize it, though.

Now start up 'FAHControl'. It should display the data for the SMP slot and the GPU slot that are together. To display the 'separate' GPU slot, you need to do the following:

1. Click on 'Configure', and select the 'Connection' tab
2. Change the 'Port' to be 36331, and click on 'Save'
3. You will now see the 'separate' GPU slot's data
4. To go back to looking at the SMP slot and the GPU slot that are together, just change the 'Port' back to 36330

FAHControl and both 'FAHClient' and 'FAHClient2' will automatically start when you log in in the future, as the shortcuts to the startup batch files were placed in the 'Startup' directory.

It sounds like a lot to do, but it's really not that complicated a procedure to follow. Just please be extremely careful when manually editing the 'config.xml' files, as any tiny mistake can have disastrous consequences.

I hope this helps,
Art

[art_l_j_PlanetAMD64]

Hold Everything!

Unfortunately, the two (different) GPUs did not stay folding, they eventually failed with a 'GPU_MEMTEST_ERROR', the same (or a similar) problem as others were having before.

So, there is some deeper issue here, that makes the FahCore_11.exe and FahCore_15.exe programs fail to work together.

Several possible causes and work-arounds come to mind, but it might be better overall just to declare that mixing 'old' and 'new' GPUs in one system is simply not supported. As I was told here:

They (PG) can do anything they want. By far the easiest would be just to issue new GPU2 projects. It is just a question of allocating manpower. From PG's perspective all they need to know is if it is worth it.

Unfortunately, over time the old technology becomes less valuable as people buy into newer, more capable, video cards and slowly over time there are fewer of the older cards still being used. Looking back and supporting the old is nice to those with older technologies, but I suspect that PG is more into looking forward and seeing what those future possibilities may contain.

Sorry, but that explanation does not help those that do not wish to upgrade. I assure you, I was not happy either when GPU1 (ATI 1xxx series) was abandoned. I just moved on.

Sound advice, in my opinion.