Project: 5751 (Run 6, Clone 154, Gen 8) instant stop

[ChasR]

Project: 5751 (Run 6, Clone 154, Gen 8) shut down another of my rigs today.

[Teddy]

Code: Select all

10:17:10] Trying to send all finished work units
[10:17:10] + No unsent completed units remaining.
[10:17:10] + Closed connections
[10:17:10] 
[10:17:10] + Processing work unit
[10:17:10] Core required: FahCore_11.exe
[10:17:10] Core found.
[10:17:10] Working on queue slot 00 [January 19 10:17:10 UTC]
[10:17:10] + Working ...
[10:17:10] - Calling '.\FahCore_11.exe -dir work/ -suffix 00 -priority 96 -checkpoint 3 -verbose -lifeline 2868 -version 620'

[10:17:11] 
[10:17:11] *------------------------------*
[10:17:11] Folding@Home GPU Core - Beta
[10:17:11] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[10:17:11] 
[10:17:11] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[10:17:11] Build host: amoeba
[10:17:11] Board Type: Nvidia
[10:17:11] Core      : 
[10:17:11] Preparing to commence simulation
[10:17:11] - Looking at optimizations...
[10:17:11] - Created dyn
[10:17:11] - Files status OK
[10:17:11] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[10:17:11] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[10:17:11] - Digital signature verified
[10:17:11] 
[10:17:11] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:17:11] 
[10:17:11] Assembly optimizations on if available.
[10:17:11] Entering M.D.
[10:17:17] Working on Protein
[10:17:20] Client config found, loading data.
[10:17:20] Starting GUI Server
[10:17:20] mdrun_gpu returned 
[10:17:20] NANs detected on GPU
[10:17:20] 
[10:17:20] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:17:23] CoreStatus = 7A (122)
[10:17:23] Sending work to server
[10:17:23] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:17:23] - Read packet limit of 540015616... Set to 524286976.
[10:17:23] - Error: Could not get length of results file work/wuresults_00.dat
[10:17:23] - Error: Could not read unit 00 file. Removing from queue.
[10:17:23] Trying to send all finished work units
[10:17:23] + No unsent completed units remaining.
[10:17:23] - Preparing to get new work unit...
[10:17:23] + Attempting to get work packet
[10:17:23] - Will indicate memory of 2046 MB
[10:17:23] - Connecting to assignment server
[10:17:23] Connecting to http://assign-GPU.stanford.edu:8080/
[10:17:24] Posted data.
[10:17:24] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[10:17:24] + News From Folding@Home: GPU folding beta
[10:17:24] Loaded queue successfully.
[10:17:24] Connecting to http://171.67.108.11:8080/
[10:17:25] Posted data.
[10:17:25] Initial: 0000; - Receiving payload (expected size: 99122)
[10:17:30] - Downloaded at ~19 kB/s
[10:17:30] - Averaged speed for that direction ~53 kB/s
[10:17:30] + Received work.
[10:17:30] Trying to send all finished work units
[10:17:30] + No unsent completed units remaining.
[10:17:30] + Closed connections
[10:17:35] 
[10:17:35] + Processing work unit
[10:17:35] Core required: FahCore_11.exe
[10:17:35] Core found.
[10:17:35] Working on queue slot 01 [January 19 10:17:35 UTC]
[10:17:35] + Working ...
[10:17:35] - Calling '.\FahCore_11.exe -dir work/ -suffix 01 -priority 96 -checkpoint 3 -verbose -lifeline 2868 -version 620'

[10:17:35] 
[10:17:35] *------------------------------*
[10:17:35] Folding@Home GPU Core - Beta
[10:17:35] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[10:17:35] 
[10:17:35] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[10:17:35] Build host: amoeba
[10:17:35] Board Type: Nvidia
[10:17:35] Core      : 
[10:17:35] Preparing to commence simulation
[10:17:35] - Looking at optimizations...
[10:17:35] - Created dyn
[10:17:35] - Files status OK
[10:17:35] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[10:17:35] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[10:17:35] - Digital signature verified
[10:17:35] 
[10:17:35] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:17:35] 
[10:17:35] Assembly optimizations on if available.
[10:17:35] Entering M.D.
[10:17:42] Working on Protein
[10:17:45] Client config found, loading data.
[10:17:45] Starting GUI Server
[10:17:45] mdrun_gpu returned 
[10:17:45] NANs detected on GPU
[10:17:45] 
[10:17:45] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:17:47] CoreStatus = 7A (122)
[10:17:47] Sending work to server
[10:17:47] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:17:47] - Read packet limit of 540015616... Set to 524286976.
[10:17:47] - Error: Could not get length of results file work/wuresults_01.dat
[10:17:47] - Error: Could not read unit 01 file. Removing from queue.
[10:17:47] Trying to send all finished work units
[10:17:47] + No unsent completed units remaining.
[10:17:47] - Preparing to get new work unit...
[10:17:47] + Attempting to get work packet
[10:17:47] - Will indicate memory of 2046 MB
[10:17:47] - Connecting to assignment server
[10:17:47] Connecting to http://assign-GPU.stanford.edu:8080/
[10:17:49] Posted data.
[10:17:49] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[10:17:49] + News From Folding@Home: GPU folding beta
[10:17:49] Loaded queue successfully.
[10:17:49] Connecting to http://171.67.108.11:8080/
[10:17:50] Posted data.
[10:17:50] Initial: 0000; - Receiving payload (expected size: 99122)
[10:17:53] - Downloaded at ~32 kB/s
[10:17:53] - Averaged speed for that direction ~49 kB/s
[10:17:53] + Received work.
[10:17:53] Trying to send all finished work units
[10:17:53] + No unsent completed units remaining.
[10:17:53] + Closed connections
[10:17:58] 
[10:17:58] + Processing work unit
[10:17:58] Core required: FahCore_11.exe
[10:17:58] Core found.
[10:17:58] Working on queue slot 02 [January 19 10:17:58 UTC]
[10:17:58] + Working ...
[10:17:58] - Calling '.\FahCore_11.exe -dir work/ -suffix 02 -priority 96 -checkpoint 3 -verbose -lifeline 2868 -version 620'

[10:17:58] 
[10:17:58] *------------------------------*
[10:17:58] Folding@Home GPU Core - Beta
[10:17:58] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[10:17:58] 
[10:17:58] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[10:17:58] Build host: amoeba
[10:17:58] Board Type: Nvidia
[10:17:58] Core      : 
[10:17:58] Preparing to commence simulation
[10:17:58] - Looking at optimizations...
[10:17:58] - Created dyn
[10:17:58] - Files status OK
[10:17:58] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[10:17:58] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[10:17:58] - Digital signature verified
[10:17:58] 
[10:17:58] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:17:58] 
[10:17:58] Assembly optimizations on if available.
[10:17:58] Entering M.D.
[10:18:04] Working on Protein
[10:18:07] Client config found, loading data.
[10:18:07] Starting GUI Server
[10:18:07] mdrun_gpu returned 
[10:18:07] NANs detected on GPU
[10:18:07] 
[10:18:07] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:18:10] CoreStatus = 7A (122)
[10:18:10] Sending work to server
[10:18:10] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:18:10] - Read packet limit of 540015616... Set to 524286976.
[10:18:10] - Error: Could not get length of results file work/wuresults_02.dat
[10:18:10] - Error: Could not read unit 02 file. Removing from queue.
[10:18:10] Trying to send all finished work units
[10:18:10] + No unsent completed units remaining.
[10:18:10] - Preparing to get new work unit...
[10:18:10] + Attempting to get work packet
[10:18:10] - Will indicate memory of 2046 MB
[10:18:10] - Connecting to assignment server
[10:18:10] Connecting to http://assign-GPU.stanford.edu:8080/
[10:18:14] Posted data.
[10:18:14] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[10:18:14] + News From Folding@Home: GPU folding beta
[10:18:14] Loaded queue successfully.
[10:18:14] Connecting to http://171.67.108.11:8080/
[10:18:16] Posted data.
[10:18:16] Initial: 0000; - Receiving payload (expected size: 99122)
[10:18:20] - Downloaded at ~24 kB/s
[10:18:20] - Averaged speed for that direction ~44 kB/s
[10:18:20] + Received work.
[10:18:20] Trying to send all finished work units
[10:18:20] + No unsent completed units remaining.
[10:18:20] + Closed connections
[10:18:25] 
[10:18:25] + Processing work unit
[10:18:25] Core required: FahCore_11.exe
[10:18:25] Core found.
[10:18:25] Working on queue slot 03 [January 19 10:18:25 UTC]
[10:18:25] + Working ...
[10:18:25] - Calling '.\FahCore_11.exe -dir work/ -suffix 03 -priority 96 -checkpoint 3 -verbose -lifeline 2868 -version 620'

[10:18:25] 
[10:18:25] *------------------------------*
[10:18:25] Folding@Home GPU Core - Beta
[10:18:25] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[10:18:25] 
[10:18:25] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[10:18:25] Build host: amoeba
[10:18:25] Board Type: Nvidia
[10:18:25] Core      : 
[10:18:25] Preparing to commence simulation
[10:18:25] - Looking at optimizations...
[10:18:25] - Created dyn
[10:18:25] - Files status OK
[10:18:25] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[10:18:25] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[10:18:25] - Digital signature verified
[10:18:25] 
[10:18:25] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:18:25] 
[10:18:25] Assembly optimizations on if available.
[10:18:25] Entering M.D.
[10:18:31] Working on Protein
[10:18:34] Client config found, loading data.
[10:18:34] Starting GUI Server
[10:18:34] mdrun_gpu returned 
[10:18:34] NANs detected on GPU
[10:18:34] 
[10:18:34] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:18:37] CoreStatus = 7A (122)
[10:18:37] Sending work to server
[10:18:37] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:18:37] - Read packet limit of 540015616... Set to 524286976.
[10:18:37] - Error: Could not get length of results file work/wuresults_03.dat
[10:18:37] - Error: Could not read unit 03 file. Removing from queue.
[10:18:37] Trying to send all finished work units
[10:18:37] + No unsent completed units remaining.
[10:18:37] - Preparing to get new work unit...
[10:18:37] + Attempting to get work packet
[10:18:37] - Will indicate memory of 2046 MB
[10:18:37] - Connecting to assignment server
[10:18:37] Connecting to http://assign-GPU.stanford.edu:8080/
[10:18:38] Posted data.
[10:18:38] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[10:18:38] + News From Folding@Home: GPU folding beta
[10:18:38] Loaded queue successfully.
[10:18:38] Connecting to http://171.67.108.11:8080/
[10:18:39] Posted data.
[10:18:39] Initial: 0000; - Receiving payload (expected size: 99122)
[10:18:41] - Downloaded at ~48 kB/s
[10:18:41] - Averaged speed for that direction ~45 kB/s
[10:18:41] + Received work.
[10:18:41] Trying to send all finished work units
[10:18:41] + No unsent completed units remaining.
[10:18:41] + Closed connections
[10:18:46] 
[10:18:46] + Processing work unit
[10:18:46] Core required: FahCore_11.exe
[10:18:46] Core found.
[10:18:46] Working on queue slot 04 [January 19 10:18:46 UTC]
[10:18:46] + Working ...
[10:18:46] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -priority 96 -checkpoint 3 -verbose -lifeline 2868 -version 620'

[10:18:46] 
[10:18:46] *------------------------------*
[10:18:46] Folding@Home GPU Core - Beta
[10:18:46] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[10:18:46] 
[10:18:46] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[10:18:46] Build host: amoeba
[10:18:46] Board Type: Nvidia
[10:18:46] Core      : 
[10:18:46] Preparing to commence simulation
[10:18:46] - Looking at optimizations...
[10:18:46] - Created dyn
[10:18:46] - Files status OK
[10:18:46] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[10:18:46] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[10:18:46] - Digital signature verified
[10:18:46] 
[10:18:46] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:18:46] 
[10:18:46] Assembly optimizations on if available.
[10:18:46] Entering M.D.
[10:18:52] Working on Protein
[10:18:55] Client config found, loading data.
[10:18:55] Starting GUI Server
[10:18:55] mdrun_gpu returned 
[10:18:55] NANs detected on GPU
[10:18:55] 
[10:18:55] Folding@home Core Shutdown: UNSTABLE_MACHINE
[10:18:58] CoreStatus = 7A (122)
[10:18:58] Sending work to server
[10:18:58] Project: 5751 (Run 6, Clone 154, Gen 8)
[10:18:58] - Read packet limit of 540015616... Set to 524286976.
[10:18:58] - Error: Could not get length of results file work/wuresults_04.dat
[10:18:58] - Error: Could not read unit 04 file. Removing from queue.
[10:18:58] EUE limit exceeded. Pausing 24 hours.

Same here, this unit has been around for 3 weeks doing bad things & seems to be doing the rounds...
What gives Stanford instant EUE, what a waste of my power sitting idle for 24hrs....

Edit well it would have been if I hadn't gone to bed at the normal time!!!!

[toTOW]

I got it again

[susato]

Thanks toTOW for reporting it to the Pande Group.

[werty316]

tsk tsk on Wu 5751 (Run 6, Clone 154, Gen 8). I got this exact WU 5 times in a row all of which failed

This WU has to be very unstable since I've never had an error for a while now.

Code: Select all

[14:24:30] - Preparing to get new work unit...
[14:24:30] + Attempting to get work packet
[14:24:30] - Connecting to assignment server
[14:24:31] - Successful: assigned to (171.67.108.11).
[14:24:31] + News From Folding@Home: GPU folding beta
[14:24:31] Loaded queue successfully.
[14:24:32] + Closed connections
[14:24:32] 
[14:24:32] + Processing work unit
[14:24:32] Core required: FahCore_11.exe
[14:24:32] Core found.
[14:24:32] Working on queue slot 01 [January 20 14:24:32 UTC]
[14:24:32] + Working ...
[14:24:32] 
[14:24:32] *------------------------------*
[14:24:32] Folding@Home GPU Core - Beta
[14:24:32] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[14:24:32] 
[14:24:32] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[14:24:32] Build host: amoeba
[14:24:32] Board Type: Nvidia
[14:24:32] Core      : 
[14:24:32] Preparing to commence simulation
[14:24:32] - Looking at optimizations...
[14:24:32] - Created dyn
[14:24:32] - Files status OK
[14:24:32] - Expanded 98610 -> 492276 (decompressed 499.2 percent)
[14:24:32] Called DecompressByteArray: compressed_data_size=98610 data_size=492276, decompressed_data_size=492276 diff=0
[14:24:32] - Digital signature verified
[14:24:32] 
[14:24:32] Project: 5751 (Run 6, Clone 154, Gen 8)
[14:24:32] 
[14:24:32] Assembly optimizations on if available.
[14:24:32] Entering M.D.
[14:24:39] Working on Protein
[14:24:41] Client config found, loading data.
[14:24:41] Starting GUI Server
[14:24:41] mdrun_gpu returned 
[14:24:41] NANs detected on GPU
[14:24:41] 
[14:24:41] Folding@home Core Shutdown: UNSTABLE_MACHINE
[14:24:44] CoreStatus = 7A (122)
[14:24:44] Sending work to server
[14:24:44] Project: 5751 (Run 6, Clone 154, Gen 8)
[14:24:44] - Error: Could not get length of results file work/wuresults_01.dat
[14:24:44] - Error: Could not read unit 01 file. Removing from queue.