155.247.166.219 & 155.247.166.220 missing credit [resolved]

[bdo]

I have the same problem. Three Wus are not credited
Project 6395 Run 77, Clone 1, Gen 60 send 2015/03/08 on 20:02:35 GMT and accepted.

Code: Select all

20:02:03:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:02:03:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6395 run:77 clone:1 gen:60 core:0xa4 unit:0x000000470002894b5462c78a4ba8bc5b
20:02:03:WU00:FS00:Uploading 1.23MiB to 155.247.166.219
20:02:03:WU01:FS00:Starting
20:02:03:WU00:FS00:Connecting to 155.247.166.219:8080
...
20:02:15:WU00:FS00:Upload 35.46%
20:02:21:WU00:FS00:Upload 55.72%
20:02:27:WU00:FS00:Upload 75.98%
20:02:33:WU00:FS00:Upload 96.24%
20:02:35:WU00:FS00:Upload complete
20:02:35:WU00:FS00:Server responded WORK_ACK (400)
20:02:35:WU00:FS00:Final credit estimate, 1633.00 points

Project 6395 Run 8, Clone 3, Gen 20 send 2015/03/10 on 1:40:28 and accepted

Code: Select all

01:40:15:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6395 run:8 clone:3 gen:20 core:0xa4 unit:0x000000190002894b5462c6fea81d88f0
01:40:15:WU00:FS00:Uploading 1.23MiB to 155.247.166.219
01:40:15:WU01:FS00:Starting
01:40:15:WU00:FS00:Connecting to 155.247.166.219:8080
....
01:40:21:WU01:FS00:0xa4:Mapping NT from 4 to 4 
01:40:21:WU01:FS00:0xa4:Completed 0 out of 250000 steps  (0%)
01:40:27:WU00:FS00:Upload 96.31%
01:40:28:WU00:FS00:Upload complete
01:40:28:WU00:FS00:Server responded WORK_ACK (400)
01:40:28:WU00:FS00:Final credit estimate, 1740.00 points

Project 6395 Run 43, Clone 8, Gen 21 send 2015/03/10 on 10:54:22 and accepted

Code: Select all

10:54:10:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6395 run:43 clone:8 gen:21 core:0xa4 unit:0x0000001c0002894b5462c746d86379f4
10:54:10:WU00:FS00:Uploading 1.23MiB to 155.247.166.219
10:54:10:WU01:FS00:Starting
10:54:10:WU00:FS00:Connecting to 155.247.166.219:8080
....
10:54:16:WU00:FS00:Upload 45.61%
10:54:16:WU01:FS00:0xa4:Mapping NT from 4 to 4 
10:54:17:WU01:FS00:0xa4:Completed 0 out of 2000000 steps  (0%)
10:54:22:WU00:FS00:Upload 96.28%
10:54:22:WU00:FS00:Upload complete
10:54:22:WU00:FS00:Server responded WORK_ACK (400)
10:54:22:WU00:FS00:Final credit estimate, 1729.00 points

Code: Select all

*********************** Log Started 2015-03-08T19:18:12Z ***********************
19:18:12:************************* Folding@home Client *************************
19:18:12:      Website: http://folding.stanford.edu/
19:18:12:    Copyright: (c) 2009-2014 Stanford University
19:18:12:       Author: Joseph Coffland <[email protected]>
19:18:12:         Args: 
19:18:12:       Config: C:/Users/baudhuin/AppData/Roaming/FAHClient/config.xml
19:18:12:******************************** Build ********************************
19:18:12:      Version: 7.4.4
19:18:12:         Date: Mar 4 2014
19:18:12:         Time: 20:26:54
19:18:12:      SVN Rev: 4130
19:18:12:       Branch: fah/trunk/client
19:18:12:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
19:18:12:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
19:18:12:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
19:18:12:     Platform: win32 XP
19:18:12:         Bits: 32
19:18:12:         Mode: Release
19:18:12:******************************* System ********************************
19:18:12:          CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
19:18:12:       CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
19:18:12:         CPUs: 4
19:18:12:       Memory: 6.00GiB
19:18:12:  Free Memory: 4.56GiB
19:18:12:      Threads: WINDOWS_THREADS
19:18:12:   OS Version: 6.1
19:18:12:  Has Battery: false
19:18:12:   On Battery: false
19:18:12:   UTC Offset: 1
19:18:12:          PID: 6048
19:18:12:          CWD: C:/Users/baudhuin/AppData/Roaming/FAHClient
19:18:12:           OS: Windows 7 Home Premium
19:18:12:      OS Arch: AMD64
19:18:12:         GPUs: 1
19:18:12:        GPU 0: NVIDIA:1 G96 [GeForce 9400 GT]
19:18:12:         CUDA: 1.1
19:18:12:  CUDA Driver: 6050
19:18:12:Win32 Service: false
19:18:12:***********************************************************************
19:18:12:<config>
19:18:12:  <!-- Folding Slot Configuration -->
19:18:12:  <gpu v='false'/>
19:18:12:
19:18:12:  <!-- Network -->
19:18:12:  <proxy v=':8080'/>
19:18:12:
19:18:12:  <!-- Slot Control -->
19:18:12:  <pause-on-battery v='false'/>
19:18:12:  <power v='full'/>
19:18:12:
19:18:12:  <!-- User Information -->
19:18:12:  <passkey v='********************************'/>
19:18:12:  <team v='35819'/>
19:18:12:  <user v='Baudhuin'/>
19:18:12:
19:18:12:  <!-- Folding Slots -->
19:18:12:  <slot id='0' type='CPU'>
19:18:12:    <pause-on-start v='true'/>
19:18:12:  </slot>
19:18:12:</config>
19:18:12:Trying to access da

[billford]

I've no idea if it's connected but both 155.247.166.219 and 155.247.166.220 seem remarkably uncommunicative where serverstats is concerned.

[suprleg]

Thanks for investigating the issue and working to resolve it, uncle_fungus. Many thanks to you as well, "sorta"....

[uncle_fungus]

From the most recent posts this one is the only credited WU

Code: Select all

Hi Baudhuin (team 35819),
Your WU (P6395 R43 C8 G21) was added to the stats database on 2015-03-11 01:09:32 for 1728.99 points of credit.

I'm going to ping the stats staff again to see what's up.

[billford]

Some more that haven't (afaict) been credited:

Project: 6383 (Run 5, Clone 2, Gen 26) est credit: 1633
Project: 6385 (Run 81, Clone 2, Gen 16) est credit: 1911
Project: 6386 (Run 15, Clone 0, Gen 364) est credit: 1654
Project: 6390 (Run 18, Clone 0, Gen 222) est credit: 1653
Project: 6384 (Run 74, Clone 1, Gen 170) est credit: 1633
Project: 6381 (Run 6, Clone 46, Gen 242) est credit: 4761

I can't provide all the log entries- I've been doing a fair amount of upgrading/re-installing so a lot of the logs have been wiped. Those figures have been copied from HFM's history.

I don't think a single WU from 155.247.166.220 has been credited since I noticed the problem (haven't had any from .219)

[msultan]

Thanks for all your reports and help everyone! I just queried p6399, (Run 104,Clone 1, Gen 83) and p6393(Run 37, Clone 0, Gen 77) and both have been successfully credited at this time. Looking into what might be causing the WS to be lagging so much. Probably just needs a restart or something. If the issue doesn't resolve after that, I will investigate further.
Thanks,
Muneeb

[billford]

I've just had about half my missing WUs appear in the last stats run so you're getting there

Thanks.

[sortofageek]

I received lists of multiple WUs to check for a couple more folders. Some have credited. These are still missing.

For sashwa, Team 4

Project: 6380 (Run 0, Clone 142, Gen 108)
Uploading 3.10MiB to 155.247.166.220
can't execute check - this project may lack a table

Project: 6388 (Run 34, Clone 0, Gen 64)
Uploading 1.16MiB to 155.247.166.220
No data back from query

Project: 6386 (Run 72, Clone 2, Gen 10)
No data back from query

---
For parkut (He has far too many to check but sent me a long list from only one of his folders. Most have credited, but these are MIA still.

--
05:22:12:WU00:FS00:0xa4:Project: 6383 (Run 48, Clone 2, Gen 4)
10:29:41:WU00:FS00:Uploading 1.16MiB to 155.247.166.220

No data back from query

--
01:14:17:WU00:FS00:0xa4:Project: 6381 (Run 8, Clone 7, Gen 137)
16:59:45:WU00:FS00:Uploading 3.10MiB to 155.247.166.220

No data back from query

--
16:50:39:WU01:FS00:0xa4:Project: 6380 (Run 0, Clone 22, Gen 192)
08:25:57:WU01:FS00:Uploading 3.10MiB to 155.247.166.220

No data back from query

--
10:29:42:WU01:FS00:0xa4:Project: 6390 (Run 6, Clone 2, Gen 62)
15:39:41:WU01:FS00:Uploading 1.17MiB to 155.247.166.220

No data back from query

--
16:59:45:WU01:FS00:0xa4:Project: 6388 (Run 61, Clone 1, Gen 306)
22:08:14:WU01:FS00:Uploading 1.17MiB to 155.247.166.220

No data back from query
--

[msultan]

The WU that have not yet being credited are now being investigated and I can confirm that at least a few were returned to the WS. I think I have a handle on what is happening. Give me till tomorrow to sort out the slow WS issues and then resolve these uncredited WUs.

[sortofageek]

Thank you for getting on this so quickly after it was reported to you. Your most recent post sounds encouraging. I'm sure we have no trouble waiting until you have time to resolve the missing credits issue and get the server(s) working correctly.

[billford]

I'll second that, more so if the problem can be fixed for the future.

Bugs happen [shrug]

[msultan]

So a quick update. I have the script that will find the missing WUs written up. @cxh, @schwancr and I are gonna be testing it in the next few days in order to make sure that we don't accidentally misassign any WUs. After its run I will make another announcement.

[sortofageek]

Thank you for the update. A team mate this morning told me the new WUs from those servers seem to be getting credits just fine since you began to look into this.

I had noticed those reported to me previously were still not credited, but realize it can take awhile to get in place. I wasn't going to bug you on the weekend, but here you are with an update. Kudos from me.

[msultan]

Sorry for the late update everyone. I got busy with end of quarter craziness for a class that I was taking this quarter and only got to look into this problem again today.

While the script I wrote works and I did credit a few of the WUs, it seems like some WU logs might have gotten corrupted. We are trying to find the logs but it is possible that some of the WUs cannot be credited. If that is the case, we are very sorry about it. However, I need to be sure that is what is happening. Looking into it now and will make another post once I am done.

[vvoelz]

This is a quick message from the Voelz Lab -- we maintain fah servers vav3 (155.247.166.219) and vav4 (155.247.166.220). Just wanted to let you know we are aware of the state credit problems and are working hard to figure out what went wrong. The attached RAID storage on these machines had some problems in the last few weeks, but it's difficult to say if that affected the log files. In any case, we hope to figure it out soon.