171.64.65.56

[DocJonz]

I've been getting similar errors - I was wondering why my PPD dropped by 1million yesterday
Here's an example of the 'errors and warnings' from one log file;

Code: Select all

******************************* Date: 2016-01-06 *******************************
******************************* Date: 2016-01-06 *******************************
******************************* Date: 2016-01-06 *******************************
23:24:57:WARNING:WU01:FS02:Server did not like results, dumping
23:59:54:WARNING:WU02:FS01:Server did not like results, dumping
03:23:04:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
******************************* Date: 2016-01-07 *******************************
08:25:15:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
08:27:55:ERROR:WU00:FS01:Exception: Transfer failed
09:42:15:WARNING:WU01:FS02:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
******************************* Date: 2016-01-07 *******************************
15:48:04:WARNING:WU00:FS02:Server did not like results, dumping
******************************* Date: 2016-01-07 *******************************
18:02:43:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
18:05:23:ERROR:WU02:FS01:Exception: Transfer failed

Looks like a number of people are having similar issues (looking at the other recent threads in this group) - hopefully someone at the Stanford end will look into what is going on as a matter of urgency ....

[Duce H_K_]

GTX970 OC1496 @ Win 8.1. Other WUs pass OK with this GPU core clock.

Code: Select all

14:06:14:WU00:FS00:0x18:*********************** Log Started 2016-01-07T14:06:14Z ***********************
14:06:14:WU00:FS00:0x18:Project: 9430 (Run 109, Clone 2, Gen 104)
14:06:14:WU00:FS00:0x18:Unit: 0x0000008bab40413855474d5d919b6736
14:06:14:WU00:FS00:0x18:CPU: 0x00000000000000000000000000000000
14:06:14:WU00:FS00:0x18:Machine: 0
14:06:14:WU00:FS00:0x18:Digital signatures verified
14:06:14:WU00:FS00:0x18:Folding@home GPU core18
14:06:14:WU00:FS00:0x18:Version 0.0.4
14:06:15:WU00:FS00:0x18:  Found a checkpoint file
14:06:27:WU00:FS00:0x18:Completed 8750000 out of 16000000 steps (54%)
14:06:27:WU00:FS00:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
14:07:02:Removing old file 'configs/config-20160101-144802.xml'
14:07:02:Saving configuration to config.xml
14:07:02:<config>
14:07:02:  <!-- Folding Core -->
14:07:02:  <checkpoint v='6'/>
14:07:02:  <core-priority v='low'/>
14:07:02:
14:07:02:  <!-- Folding Slot Configuration -->
14:07:02:  <extra-core-args v='-forceasm -twait=80'/>
14:07:02:
14:07:02:  <!-- HTTP Server -->
14:07:02:  <allow v='0.0.0.0/0'/>
14:07:02:  <deny v='255.255.255.255/255.255.255.255'/>
14:07:02:
14:07:02:  <!-- Logging -->
14:07:02:  <verbosity v='5'/>
14:07:02:
14:07:02:  <!-- Network -->
14:07:02:  <proxy v=':8080'/>
14:07:02:
14:07:02:  <!-- Remote Command Server -->
14:07:02:  <command-deny-no-pass v=''/>
14:07:02:  <command-port v='7936'/>
14:07:02:  <password v='************'/>
14:07:02:
14:07:02:  <!-- Slot Control -->
14:07:02:  <pause-on-battery v='false'/>
14:07:02:  <pause-on-start v='true'/>
14:07:02:  <power v='full'/>
14:07:02:
14:07:02:  <!-- User Information -->
14:07:02:  <passkey v='********************************'/>
14:07:02:  <team v='47191'/>
14:07:02:  <user v='Duce-HK_FLDC_1CbwZo443yjerS5JY7QQ9AiavrSyT5NpXt'/>
14:07:02:
14:07:02:  <!-- Work Unit Control -->
14:07:02:  <max-queue v='2'/>
14:07:02:  <next-unit-percentage v='100'/>
14:07:02:
14:07:02:  <!-- Folding Slots -->
14:07:02:  <slot id='0' type='GPU'>
14:07:02:    <client-type v=''/>
14:07:02:    <gpu-index v='1'/>
14:07:02:    <opencl-index v='1'/>
14:07:02:  </slot>
14:07:02:  <slot id='1' type='CPU'>
14:07:02:    <client-type v='beta'/>
14:07:02:    <cpus v='4'/>
14:07:02:  </slot>
14:07:02:</config>
14:08:15:WU00:FS00:0x18:Completed 8800000 out of 16000000 steps (55%)
14:13:54:WU00:FS00:0x18:Completed 8960000 out of 16000000 steps (56%)
14:19:23:WU00:FS00:0x18:Completed 9120000 out of 16000000 steps (57%)
14:24:51:WU00:FS00:0x18:Completed 9280000 out of 16000000 steps (58%)
14:30:19:WU00:FS00:0x18:Completed 9440000 out of 16000000 steps (59%)
14:35:48:WU00:FS00:0x18:Completed 9600000 out of 16000000 steps (60%)
14:41:20:WU00:FS00:0x18:Completed 9760000 out of 16000000 steps (61%)
14:46:49:WU00:FS00:0x18:Completed 9920000 out of 16000000 steps (62%)
14:52:18:WU00:FS00:0x18:Completed 10080000 out of 16000000 steps (63%)
14:57:46:WU00:FS00:0x18:Completed 10240000 out of 16000000 steps (64%)
15:03:19:WU00:FS00:0x18:Completed 10400000 out of 16000000 steps (65%)
15:09:02:WU00:FS00:0x18:Completed 10560000 out of 16000000 steps (66%)
15:14:40:WU00:FS00:0x18:Completed 10720000 out of 16000000 steps (67%)
15:20:18:WU00:FS00:0x18:Completed 10880000 out of 16000000 steps (68%)
15:25:56:WU00:FS00:0x18:Completed 11040000 out of 16000000 steps (69%)
15:31:33:WU00:FS00:0x18:Completed 11200000 out of 16000000 steps (70%)
15:37:13:WU00:FS00:0x18:Completed 11360000 out of 16000000 steps (71%)
15:42:50:WU00:FS00:0x18:Completed 11520000 out of 16000000 steps (72%)
15:48:27:WU00:FS00:0x18:Completed 11680000 out of 16000000 steps (73%)
15:54:05:WU00:FS00:0x18:Completed 11840000 out of 16000000 steps (74%)
15:59:43:WU00:FS00:0x18:Completed 12000000 out of 16000000 steps (75%)
16:05:22:WU00:FS00:0x18:Completed 12160000 out of 16000000 steps (76%)
16:10:52:WU00:FS00:0x18:Completed 12320000 out of 16000000 steps (77%)
16:16:23:WU00:FS00:0x18:Completed 12480000 out of 16000000 steps (78%)
16:22:06:WU00:FS00:0x18:Completed 12640000 out of 16000000 steps (79%)
16:27:52:WU00:FS00:0x18:Completed 12800000 out of 16000000 steps (80%)
16:33:35:WU00:FS00:0x18:Completed 12960000 out of 16000000 steps (81%)
16:39:11:WU00:FS00:0x18:Completed 13120000 out of 16000000 steps (82%)
16:44:47:WU00:FS00:0x18:Completed 13280000 out of 16000000 steps (83%)
16:50:27:WU00:FS00:0x18:Completed 13440000 out of 16000000 steps (84%)
16:56:13:WU00:FS00:0x18:Completed 13600000 out of 16000000 steps (85%)
17:02:05:WU00:FS00:0x18:Completed 13760000 out of 16000000 steps (86%)
17:07:51:WU00:FS00:0x18:Completed 13920000 out of 16000000 steps (87%)
17:13:34:WU00:FS00:0x18:Completed 14080000 out of 16000000 steps (88%)
17:19:15:WU00:FS00:0x18:Completed 14240000 out of 16000000 steps (89%)
17:24:55:WU00:FS00:0x18:Completed 14400000 out of 16000000 steps (90%)
17:30:43:WU00:FS00:0x18:Completed 14560000 out of 16000000 steps (91%)
17:36:23:WU00:FS00:0x18:Completed 14720000 out of 16000000 steps (92%)
17:42:05:WU00:FS00:0x18:Completed 14880000 out of 16000000 steps (93%)
17:47:45:WU00:FS00:0x18:Completed 15040000 out of 16000000 steps (94%)
17:53:28:WU00:FS00:0x18:Completed 15200000 out of 16000000 steps (95%)
17:59:14:WU00:FS00:0x18:Completed 15360000 out of 16000000 steps (96%)
18:04:59:WU00:FS00:0x18:Completed 15520000 out of 16000000 steps (97%)
18:10:38:WU00:FS00:0x18:Completed 15680000 out of 16000000 steps (98%)
18:16:20:WU00:FS00:0x18:Completed 15840000 out of 16000000 steps (99%)
18:21:59:WU00:FS00:0x18:Completed 16000000 out of 16000000 steps (100%)
18:22:00:WU01:FS00:Connecting to 171.67.108.45:80
18:22:01:WU01:FS00:Assigned to work server 171.64.65.84
18:22:01:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:1:GM204 [GeForce GTX 970] from 171.64.65.84
18:22:01:WU01:FS00:Connecting to 171.64.65.84:8080
18:22:01:WU01:FS00:Downloading 2.69MiB
18:22:04:WU00:FS00:0x18:Saving result file logfile_01.txt
18:22:04:WU00:FS00:0x18:Saving result file checkpointState.xml
18:22:05:WU00:FS00:0x18:Saving result file checkpt.crc
18:22:05:WU00:FS00:0x18:Saving result file log.txt
18:22:05:WU00:FS00:0x18:Saving result file positions.xtc
18:22:07:WU01:FS00:Download 72.14%
18:22:09:WU01:FS00:Download complete
18:22:09:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9157 run:129 clone:0 gen:27 core:0x18 unit:0x0000001eab4041545673c800fff6f2cf
18:22:11:WU00:FS00:0x18:Folding@home Core Shutdown: FINISHED_UNIT
18:22:11:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:22:12:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9430 run:109 clone:2 gen:104 core:0x18 unit:0x0000008bab40413855474d5d919b6736
18:22:12:WU00:FS00:Uploading 24.03MiB to 171.64.65.56
18:22:12:WU01:FS00:Starting
18:22:12:WU00:FS00:Connecting to 171.64.65.56:8080
18:22:20:WU00:FS00:Upload 41.61%
18:22:29:WU00:FS00:Upload 92.85%
18:22:31:WU00:FS00:Upload complete
18:22:31:WU00:FS00:Server responded WORK_QUIT (404)
18:22:31:WARNING:WU00:FS00:Server did not like results, dumping
18:22:31:WU00:FS00:Cleaning up

Maybe the reason is that PC made sudden reboot on 15% of that 9430

Reports of additional failed WU's are not needed at this time.

Sorry I didn't see It for the first time. Just wanted to express my frustration about this

[Nert]

I haven't seen an answer to folding_hoomers question regarding whether dumped work units are lost ? What happens in this case ?

[mmonnin]

This WU was dumped for me as well when it tried to upload 1 hour ago.
14:54:16:WU03:FS01:0x18:Project: 9430 (Run 134, Clone 4, Gen 97)
23:33:36:WARNING:WU03:FS01:Server did not like results, dumping

[bruce]

There are several causes for "server did not like results" and it's really impossible to guess which one might apply in this case. Though it's a shame to have a completed WU discarded, it's still a pretty rare occurrence. It could be a situation with the server or it could be some kind of corruption before the WU was completed. I will see if we can eliminate the problem if it's on the server.

By the way, there's no advantage to increasing verbosity in the V7 client ... and in fact, it makes it harder for me to understand what settings you've used.

Dumped WUs are lost. The local copy of the WU being dumped is deleted from your system. WUs only get dumped if the server expects that it will never be able to accept the WU such as if it has been corrupted.

The primary purpose of dumping them is to prevent them from repeatedly retrying to upload, and then discarded by the server each time, wasting bandwidth.

[Joe_H]

GTX970 OC1496 @ Win 8.1. Other WUs pass OK with this GPU core clock.

Maybe the reason is that PC made sudden reboot on 15% of that 9430

Reports of additional failed WU's are not needed at this time.

Sorry I didn't see It for the first time. Just wanted to express my frustration about this

No problem, my message was back in November, So if this problem is reoccurring, new reports do need to be posted.

[Nert]

I had an occurrence of a work unit being dumped, which was why I asked. I don't know if this belongs in this thread or if it belongs in another location. I don't have enough familiarity with the details of the various projects and servers to know. Please move as needed. I will say that having a unit thrown away after completing it is, in my mind, the absolute worst possible scenario for someone donating resources to the project. Worse than not even donating in the first place.

Here's the beginning of the log:

Code: Select all

*********************** Log Started 2016-01-01T00:10:18Z ***********************
00:10:18:************************* Folding@home Client *************************
00:10:18:      Website: http://folding.stanford.edu/
00:10:18:    Copyright: (c) 2009-2014 Stanford University
00:10:18:       Author: Joseph Coffland <[email protected]>
00:10:18:         Args: 
00:10:18:       Config: C:/Users/roger/AppData/Roaming/FAHClient/config.xml
00:10:18:******************************** Build ********************************
00:10:18:      Version: 7.4.4
00:10:18:         Date: Mar 4 2014
00:10:18:         Time: 20:26:54
00:10:18:      SVN Rev: 4130
00:10:18:       Branch: fah/trunk/client
00:10:18:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
00:10:18:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
00:10:18:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
00:10:18:     Platform: win32 XP
00:10:18:         Bits: 32
00:10:18:         Mode: Release
00:10:18:******************************* System ********************************
00:10:18:          CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
00:10:18:       CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
00:10:18:         CPUs: 8
00:10:18:       Memory: 15.94GiB
00:10:18:  Free Memory: 14.02GiB
00:10:18:      Threads: WINDOWS_THREADS
00:10:18:   OS Version: 6.2
00:10:18:  Has Battery: false
00:10:18:   On Battery: false
00:10:18:   UTC Offset: -7
00:10:18:          PID: 7092
00:10:18:          CWD: C:/Users/roger/AppData/Roaming/FAHClient
00:10:18:           OS: Windows 10 Home
00:10:18:      OS Arch: AMD64
00:10:18:         GPUs: 2
00:10:18:        GPU 0: NVIDIA:4 GM107 [GeForce GTX 750 Ti]
00:10:18:        GPU 1: NVIDIA:5 GM204 [GeForce GTX 970]
00:10:18:         CUDA: 5.2
00:10:18:  CUDA Driver: 7050
00:10:18:Win32 Service: false
00:10:18:***********************************************************************
00:10:18:<config>
00:10:18:  <!-- Folding Slot Configuration -->
00:10:18:  <cause v='PARKINSONS'/>
00:10:18:
00:10:18:  <!-- Network -->
00:10:18:  <proxy v=':8080'/>
00:10:18:
00:10:18:  <!-- Slot Control -->
00:10:18:  <power v='FULL'/>
00:10:18:
00:10:18:  <!-- User Information -->
00:10:18:  <passkey v='********************************'/>
00:10:18:  <team v='165780'/>
00:10:18:  <user v='nert'/>
00:10:18:
00:10:18:  <!-- Folding Slots -->
00:10:18:  <slot id='1' type='GPU'>
00:10:18:    <max-packet-size v='big'/>
00:10:18:    <paused v='true'/>
00:10:18:  </slot>
00:10:18:  <slot id='2' type='GPU'>
00:10:18:    <paused v='true'/>
00:10:18:  </slot>
00:10:18:  <slot id='0' type='CPU'>
00:10:18:    <paused v='true'/>
00:10:18:  </slot>
00:10:18:</config>
00:10:18:Trying to access database...
00:10:18:Successfully acquired database lock

And here's the work unit being dumped:

Code: Select all

00:43:41:WU00:FS02:0x18:Completed 14400000 out of 16000000 steps (90%)
00:58:51:WU00:FS02:0x18:Completed 14560000 out of 16000000 steps (91%)
01:13:54:WU00:FS02:0x18:Completed 14720000 out of 16000000 steps (92%)
01:28:58:WU00:FS02:0x18:Completed 14880000 out of 16000000 steps (93%)
01:44:00:WU00:FS02:0x18:Completed 15040000 out of 16000000 steps (94%)
01:58:57:WU00:FS02:0x18:Completed 15200000 out of 16000000 steps (95%)
02:13:57:WU00:FS02:0x18:Completed 15360000 out of 16000000 steps (96%)
02:28:41:WU00:FS02:0x18:Completed 15520000 out of 16000000 steps (97%)
02:43:24:WU00:FS02:0x18:Completed 15680000 out of 16000000 steps (98%)
02:58:14:WU00:FS02:0x18:Completed 15840000 out of 16000000 steps (99%)
02:58:14:WU01:FS02:Connecting to 171.67.108.45:80
02:58:14:WU01:FS02:Assigned to work server 171.64.65.92
02:58:14:WU01:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:GM204 [GeForce GTX 970] from 171.64.65.92
02:58:14:WU01:FS02:Connecting to 171.64.65.92:8080
02:58:15:WU01:FS02:Downloading 3.09MiB
02:58:18:WU01:FS02:Download complete
02:58:18:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9161 run:199 clone:0 gen:9 core:0x18 unit:0x0000000aab40415c567466638273db32
03:12:57:WU00:FS02:0x18:Completed 16000000 out of 16000000 steps (100%)
03:13:01:WU00:FS02:0x18:Saving result file logfile_01.txt
03:13:01:WU00:FS02:0x18:Saving result file checkpointState.xml
03:13:02:WU00:FS02:0x18:Saving result file checkpt.crc
03:13:02:WU00:FS02:0x18:Saving result file log.txt
03:13:02:WU00:FS02:0x18:Saving result file positions.xtc
03:13:06:WU00:FS02:0x18:Folding@home Core Shutdown: FINISHED_UNIT
03:13:06:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:13:06:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:9430 run:58 clone:5 gen:185 core:0x18 unit:0x000000daab40413855474ccdd4eda35f
03:13:07:WU00:FS02:Uploading 24.06MiB to 171.64.65.56
03:13:07:WU00:FS02:Connecting to 171.64.65.56:8080
03:13:07:WU01:FS02:Starting
03:13:07:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/roger/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 01 -suffix 01 -version 704 -lifeline 7092 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
03:13:07:WU01:FS02:Started FahCore on PID 8732
03:13:07:WU01:FS02:Core PID:8564
03:13:07:WU01:FS02:FahCore 0x18 started
03:13:07:WU01:FS02:0x18:*********************** Log Started 2016-01-08T03:13:07Z ***********************
03:13:07:WU01:FS02:0x18:Project: 9161 (Run 199, Clone 0, Gen 9)
03:13:07:WU01:FS02:0x18:Unit: 0x0000000aab40415c567466638273db32
03:13:07:WU01:FS02:0x18:CPU: 0x00000000000000000000000000000000
03:13:07:WU01:FS02:0x18:Machine: 2
03:13:07:WU01:FS02:0x18:Reading tar file core.xml
03:13:07:WU01:FS02:0x18:Reading tar file system.xml
03:13:07:WU01:FS02:0x18:Reading tar file integrator.xml
03:13:07:WU01:FS02:0x18:Reading tar file state.xml
03:13:08:WU01:FS02:0x18:Digital signatures verified
03:13:08:WU01:FS02:0x18:Folding@home GPU core18
03:13:08:WU01:FS02:0x18:Version 0.0.4
03:13:13:WU00:FS02:Upload 3.38%
03:13:19:WU00:FS02:Upload 5.71%
03:13:25:WU00:FS02:Upload 7.79%
03:13:27:WU01:FS02:0x18:Completed 0 out of 2500000 steps (0%)
03:13:27:WU01:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:13:31:WU00:FS02:Upload 9.87%
03:13:37:WU00:FS02:Upload 12.21%
03:13:43:WU00:FS02:Upload 14.29%
03:13:49:WU00:FS02:Upload 16.37%
03:13:55:WU00:FS02:Upload 18.70%
03:14:01:WU00:FS02:Upload 20.78%
03:14:07:WU00:FS02:Upload 22.86%
03:14:13:WU00:FS02:Upload 24.94%
03:14:19:WU00:FS02:Upload 27.28%
03:14:25:WU00:FS02:Upload 29.35%
03:14:31:WU00:FS02:Upload 31.43%
03:14:37:WU00:FS02:Upload 33.51%
03:14:43:WU00:FS02:Upload 35.85%
03:14:49:WU00:FS02:Upload 37.93%
03:14:55:WU00:FS02:Upload 40.00%
03:15:01:WU00:FS02:Upload 42.34%
03:15:07:WU00:FS02:Upload 44.42%
03:15:13:WU00:FS02:Upload 46.50%
03:15:19:WU00:FS02:Upload 48.58%
03:15:25:WU00:FS02:Upload 50.91%
03:15:31:WU00:FS02:Upload 52.99%
03:15:37:WU00:FS02:Upload 55.07%
03:15:43:WU00:FS02:Upload 57.15%
03:15:49:WU00:FS02:Upload 59.49%
03:15:55:WU00:FS02:Upload 61.56%
03:16:01:WU00:FS02:Upload 63.64%
03:16:07:WU00:FS02:Upload 65.98%
03:16:13:WU00:FS02:Upload 68.06%
03:16:19:WU00:FS02:Upload 70.14%
03:16:25:WU00:FS02:Upload 72.47%
03:16:31:WU00:FS02:Upload 74.55%
03:16:37:WU00:FS02:Upload 76.63%
03:16:43:WU00:FS02:Upload 78.71%
03:16:49:WU00:FS02:Upload 81.05%
03:16:55:WU00:FS02:Upload 83.12%
03:17:01:WU00:FS02:Upload 85.20%
03:17:07:WU00:FS02:Upload 87.28%
03:17:13:WU00:FS02:Upload 89.62%
03:17:19:WU00:FS02:Upload 91.70%
03:17:25:WU00:FS02:Upload 93.77%
03:17:31:WU00:FS02:Upload 96.11%
03:17:37:WU00:FS02:Upload 98.19%
03:17:46:WU00:FS02:Upload complete
03:17:46:WU00:FS02:Server responded WORK_QUIT (404)
03:17:46:WARNING:WU00:FS02:Server did not like results, dumping
03:17:46:WU00:FS02:Cleaning up
03:17:56:WU01:FS02:0x18:Completed 25000 out of 2500000 steps (1%)
03:22:18:WU01:FS02:0x18:Completed 50000 out of 2500000 steps (2%)
03:26:41:WU01:FS02:0x18:Completed 75000 out of 2500000 steps (3%)
03:31:03:WU01:FS02:0x18:Completed 100000 out of 2500000 steps (4%)
03:35:33:WU01:FS02:0x18:Completed 125000 out of 2500000 steps (5%)
03:39:56:WU01:FS02:0x18:Completed 150000 out of 2500000 steps (6%)
03:44:18:WU01:FS02:0x18:Completed 175000 out of 2500000 steps (7%)
03:48:41:WU01:FS02:0x18:Completed 200000 out of 2500000 steps (8%)
03:53:11:WU01:FS02:0x18:Completed 225000 out of 2500000 steps (9%)
03:57:33:WU01:FS02:0x18:Completed 250000 out of 2500000 steps (10%)
04:01:56:WU01:FS02:0x18:Completed 275000 out of 2500000 steps (11%)
04:06:18:WU01:FS02:0x18:Completed 300000 out of 2500000 steps (12%)
04:10:48:WU01:FS02:0x18:Completed 325000 out of 2500000 steps (13%)
04:15:11:WU01:FS02:0x18:Completed 350000 out of 2500000 steps (14%)
04:19:33:WU01:FS02:0x18:Completed 375000 out of 2500000 steps (15%)
04:23:57:WU01:FS02:0x18:Completed 400000 out of 2500000 steps (16%)
04:28:29:WU01:FS02:0x18:Completed 425000 out of 2500000 steps (17%)

Let me know if there is anything else I can provide to help diagnose what is going on.

[DocJonz]

I have multiple machines dumping WUs and failing to transfer files - below is a snapshot of one of each from today.
Given the number of other people seeing similar issues, there is clearly a serverside issue that needs addressing ASAP.

Code: Select all

06:22:13:WU00:FS02:0x18:Completed 15520000 out of 16000000 steps (97%)
06:22:41:WU01:FS01:0x18:Completed 450000 out of 2500000 steps (18%)
06:23:50:WU01:FS01:0x18:Completed 475000 out of 2500000 steps (19%)
06:25:00:WU01:FS01:0x18:Completed 500000 out of 2500000 steps (20%)
06:25:52:WU00:FS02:0x18:Completed 15680000 out of 16000000 steps (98%)
06:26:12:WU01:FS01:0x18:Completed 525000 out of 2500000 steps (21%)
06:27:21:WU01:FS01:0x18:Completed 550000 out of 2500000 steps (22%)
06:28:31:WU01:FS01:0x18:Completed 575000 out of 2500000 steps (23%)
06:29:30:WU00:FS02:0x18:Completed 15840000 out of 16000000 steps (99%)
06:29:31:WU02:FS02:Connecting to 171.67.108.45:80
06:29:32:WU02:FS02:Assigned to work server 171.67.108.159
06:29:32:WU02:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:GM200 [GeForce GTX 980 Ti] from 171.67.108.159
06:29:32:WU02:FS02:Connecting to 171.67.108.159:8080
06:29:33:WU02:FS02:Downloading 18.92MiB
06:29:38:WU02:FS02:Download complete
06:29:38:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:9152 run:22 clone:6 gen:84 core:0x18 unit:0x0000005eab436c9f56623c603f31a3d1
06:29:40:WU01:FS01:0x18:Completed 600000 out of 2500000 steps (24%)
06:30:52:WU01:FS01:0x18:Completed 625000 out of 2500000 steps (25%)
06:32:02:WU01:FS01:0x18:Completed 650000 out of 2500000 steps (26%)
06:33:09:WU00:FS02:0x18:Completed 16000000 out of 16000000 steps (100%)
06:33:10:WU00:FS02:0x18:Saving result file logfile_01.txt
06:33:10:WU00:FS02:0x18:Saving result file checkpointState.xml
06:33:11:WU00:FS02:0x18:Saving result file checkpt.crc
06:33:11:WU00:FS02:0x18:Saving result file log.txt
06:33:11:WU00:FS02:0x18:Saving result file positions.xtc
06:33:11:WU01:FS01:0x18:Completed 675000 out of 2500000 steps (27%)
06:33:13:WU00:FS02:0x18:Folding@home Core Shutdown: FINISHED_UNIT
06:33:14:WU00:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:33:14:WU00:FS02:Sending unit results: id:00 state:SEND error:NO_ERROR project:9430 run:146 clone:4 gen:88 core:0x18 unit:0x00000070ab40413855474e5e0fc02530
06:33:14:WU00:FS02:Uploading 23.93MiB to 171.64.65.56
06:33:14:WU00:FS02:Connecting to 171.64.65.56:8080
06:33:14:WU02:FS02:Starting
06:33:14:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18 -dir 02 -suffix 01 -version 704 -lifeline 2395 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
06:33:14:WU02:FS02:Started FahCore on PID 16388
06:33:14:WU02:FS02:Core PID:16392
06:33:14:WU02:FS02:FahCore 0x18 started
06:33:14:WU02:FS02:0x18:*********************** Log Started 2016-01-08T06:33:14Z ***********************
06:33:14:WU02:FS02:0x18:Project: 9152 (Run 22, Clone 6, Gen 84)
06:33:14:WU02:FS02:0x18:Unit: 0x0000005eab436c9f56623c603f31a3d1
06:33:14:WU02:FS02:0x18:CPU: 0x00000000000000000000000000000000
06:33:14:WU02:FS02:0x18:Machine: 2
06:33:14:WU02:FS02:0x18:Reading tar file core.xml
06:33:14:WU02:FS02:0x18:Reading tar file integrator.xml
06:33:14:WU02:FS02:0x18:Reading tar file state.xml
06:33:14:WU02:FS02:0x18:Reading tar file system.xml
06:33:14:WU02:FS02:0x18:Digital signatures verified
06:33:14:WU02:FS02:0x18:Folding@home GPU core18
06:33:14:WU02:FS02:0x18:Version 0.0.4
06:33:19:WU02:FS02:0x18:Completed 0 out of 2500000 steps (0%)
06:33:19:WU02:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:33:20:WU00:FS02:Upload 18.28%
06:33:26:WU00:FS02:Upload 39.44%
06:33:32:WU00:FS02:Upload 76.52%
06:33:36:WU00:FS02:Upload complete
06:33:36:WU00:FS02:Server responded WORK_QUIT (404)
06:33:36:WARNING:WU00:FS02:Server did not like results, dumping
06:33:36:WU00:FS02:Cleaning up
06:34:19:WU02:FS02:0x18:Completed 25000 out of 2500000 steps (1%)
06:34:20:WU01:FS01:0x18:Completed 700000 out of 2500000 steps (28%)
06:35:17:WU02:FS02:0x18:Completed 50000 out of 2500000 steps (2%)
06:35:33:WU01:FS01:0x18:Completed 725000 out of 2500000 steps (29%)
06:36:14:WU02:FS02:0x18:Completed 75000 out of 2500000 steps (3%)
06:36:42:WU01:FS01:0x18:Completed 750000 out of 2500000 steps (30%)

Code: Select all

02:35:05:WU01:FS00:0x21:Completed 2050000 out of 5000000 steps (41%)
02:36:08:WU02:FS02:0x18:Completed 4950000 out of 5000000 steps (99%)
02:38:12:WU01:FS00:0x21:Completed 2100000 out of 5000000 steps (42%)
02:38:24:WU02:FS02:0x18:Completed 5000000 out of 5000000 steps (100%)
02:38:25:WU00:FS02:Connecting to 171.67.108.45:80
02:38:26:WU00:FS02:Assigned to work server 140.163.4.244
02:38:26:WU00:FS02:Requesting new work unit for slot 02: RUNNING gpu:0:GM204 [GeForce GTX 970] from 140.163.4.244
02:38:26:WU00:FS02:Connecting to 140.163.4.244:8080
02:38:26:WU00:FS02:Downloading 2.54MiB
02:38:27:WU00:FS02:Download complete
02:38:27:WU00:FS02:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10490 run:311 clone:0 gen:42 core:0x18 unit:0x000000308ca304f45537e90c68283a8b
02:38:28:WU02:FS02:0x18:Saving result file logfile_01.txt
02:38:28:WU02:FS02:0x18:Saving result file checkpointState.xml
02:38:29:WU02:FS02:0x18:Saving result file checkpt.crc
02:38:29:WU02:FS02:0x18:Saving result file log.txt
02:38:29:WU02:FS02:0x18:Saving result file positions.xtc
02:38:30:WU02:FS02:0x18:Folding@home Core Shutdown: FINISHED_UNIT
02:38:31:WU02:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:38:31:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:10476 run:0 clone:180 gen:511 core:0x18 unit:0x00000276538b3dba540f4b58524259e8
02:38:31:WU02:FS02:Uploading 6.12MiB to 140.163.4.234
02:38:31:WU02:FS02:Connecting to 140.163.4.234:8080
02:38:31:WU00:FS02:Starting
02:38:31:WU00:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18 -dir 00 -suffix 01 -version 704 -lifeline 924 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
02:38:31:WU00:FS02:Started FahCore on PID 26658
02:38:31:WU00:FS02:Core PID:26662
02:38:31:WU00:FS02:FahCore 0x18 started
02:38:31:WU00:FS02:0x18:*********************** Log Started 2016-01-08T02:38:31Z ***********************
02:38:31:WU00:FS02:0x18:Project: 10490 (Run 311, Clone 0, Gen 42)
02:38:31:WU00:FS02:0x18:Unit: 0x000000308ca304f45537e90c68283a8b
02:38:31:WU00:FS02:0x18:CPU: 0x00000000000000000000000000000000
02:38:31:WU00:FS02:0x18:Machine: 2
02:38:31:WU00:FS02:0x18:Reading tar file core.xml
02:38:31:WU00:FS02:0x18:Reading tar file system.xml
02:38:31:WU00:FS02:0x18:Reading tar file integrator.xml
02:38:31:WU00:FS02:0x18:Reading tar file state.xml
02:38:31:WU00:FS02:0x18:Digital signatures verified
02:38:31:WU00:FS02:0x18:Folding@home GPU core18
02:38:31:WU00:FS02:0x18:Version 0.0.4
02:38:41:WU00:FS02:0x18:Completed 0 out of 5000000 steps (0%)
02:38:41:WU00:FS02:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:41:16:WU00:FS02:0x18:Completed 50000 out of 5000000 steps (1%)
02:41:22:WU01:FS00:0x21:Completed 2150000 out of 5000000 steps (43%)
02:42:24:WU02:FS02:Upload 66.41%
02:43:17:WU02:FS02:Upload 76.62%
02:43:17:WARNING:WU02:FS02:Exception: Failed to send results to work server: Transfer failed
02:43:17:WU02:FS02:Trying to send results to collection server
02:43:17:WU02:FS02:Uploading 6.12MiB to 140.163.4.241
02:43:17:WU02:FS02:Connecting to 140.163.4.241:8080
02:43:47:WU00:FS02:0x18:Completed 100000 out of 5000000 steps (2%)
02:44:30:WU01:FS00:0x21:Completed 2200000 out of 5000000 steps (44%)
02:45:45:WU02:FS02:Upload 64.36%
02:46:03:WU02:FS02:Upload 76.62%
02:46:03:ERROR:WU02:FS02:Exception: Transfer failed
02:46:03:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:10476 run:0 clone:180 gen:511 core:0x18 unit:0x00000276538b3dba540f4b58524259e8
02:46:03:WU02:FS02:Uploading 6.12MiB to 140.163.4.234
02:46:03:WU02:FS02:Connecting to 140.163.4.234:8080
02:46:22:WU00:FS02:0x18:Completed 150000 out of 5000000 steps (3%)
02:47:38:WU01:FS00:0x21:Completed 2250000 out of 5000000 steps (45%)
02:48:40:WU02:FS02:Upload 73.56%
02:48:52:WU00:FS02:0x18:Completed 200000 out of 5000000 steps (4%)
02:48:54:WU02:FS02:Upload 82.75%
02:48:54:WARNING:WU02:FS02:Exception: Failed to send results to work server: Transfer failed
02:48:54:WU02:FS02:Trying to send results to collection server
02:48:54:WU02:FS02:Uploading 6.12MiB to 140.163.4.241
02:48:54:WU02:FS02:Connecting to 140.163.4.241:8080
02:50:47:WU01:FS00:0x21:Completed 2300000 out of 5000000 steps (46%)
02:51:22:WU02:FS02:Upload 62.32%
02:51:23:WU00:FS02:0x18:Completed 250000 out of 5000000 steps (5%)
02:51:46:WU02:FS02:Upload 77.64%
02:51:46:ERROR:WU02:FS02:Exception: Transfer failed
02:51:46:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:10476 run:0 clone:180 gen:511 core:0x18 unit:0x00000276538b3dba540f4b58524259e8
02:51:46:WU02:FS02:Uploading 6.12MiB to 140.163.4.234
02:51:46:WU02:FS02:Connecting to 140.163.4.234:8080
02:51:57:WU02:FS02:Upload complete
02:51:57:WU02:FS02:Server responded WORK_ACK (400)
02:51:57:WU02:FS02:Final credit estimate, 43375.00 points
02:51:57:WU02:FS02:Cleaning up
02:53:55:WU01:FS00:0x21:Completed 2350000 out of 5000000 steps (47%)
02:53:59:WU00:FS02:0x18:Completed 300000 out of 5000000 steps (6%)
02:56:29:WU00:FS02:0x18:Completed 350000 out of 5000000 steps (7%)
02:57:04:WU01:FS00:0x21:Completed 2400000 out of 5000000 steps (48%)
02:59:04:WU00:FS02:0x18:Completed 400000 out of 5000000 steps (8%)
03:00:12:WU01:FS00:0x21:Completed 2450000 out of 5000000 steps (49%)
03:01:35:WU00:FS02:0x18:Completed 450000 out of 5000000 steps (9%)
03:03:20:WU01:FS00:0x21:Completed 2500000 out of 5000000 steps (50%)
03:04:06:WU00:FS02:0x18:Completed 500000 out of 5000000 steps (10%)

[Mstenholm]

Something is not right with the 9340s/collecting server. I had another 404 on a different GPU (still a GTX 970). My first post was in this viewtopic.php?f=18&t=28358

7th of Jan 2016:

10:46:58:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9430 run:144 clone:8 gen:85 core:0x18 unit:0x0000006eab40413855474e544994f716
10:46:58:WU00:FS00:Uploading 24.04MiB to 171.64.65.56
.
.
10:52:27:WU00:FS00:Upload 97.76%
10:52:34:WU00:FS00:Upload 99.06%
10:52:38:WU00:FS00:Upload complete
10:52:38:WU00:FS00:Server responded WORK_QUIT (404)
10:52:38:WARNING:WU00:FS00:Server did not like results, dumping.

[mmonnin]

And another:
08:15:37:WARNING:WU00:FS01:Server did not like results, dumping

The next 9430 will be dumped by ME if I see it. This is not acceptable. Just stop assigning this crap.

[Nert]

And I had another one after my previous post. I've shut down all of my clients for now.

[bruce]

The server was taken off-line yesterday morning so you shouldn't have gotten any more projects from it since then. Unfortunately we can't do anything about WUs that have been distributed, but if you have one, feel free to dump it -- but only if it's from that particular server.