Can't upload WUs

[iokeith]

Just checked the old logs and haven't been able to upload WUs since Nov 11th. Recently noticed lack of uploading a week ago with 9 slots waiting and thought it was servers down. Downloaded latest version 7.3.6 to replace previous 7.1.52, paused CPU to force upload and opened firewall to all but no change. Any clues about how to correct this? Thanks

Here's the original log start:

Code: Select all

*********************** Log Started 2013-11-05T15:26:36Z ***********************
15:26:36:************************* Folding@home Client *************************
15:26:36:      Website: http://folding.stanford.edu/
15:26:36:    Copyright: (c) 2009-2012 Stanford University
15:26:36:       Author: Joseph Coffland <[email protected]>
15:26:36:         Args: --lifeline 5004 --command-port=36330
15:26:36:       Config: C:/Users/kiordache/AppData/Roaming/FAHClient/config.xml
15:26:36:******************************** Build ********************************
15:26:36:      Version: 7.1.52
15:26:36:         Date: Mar 20 2012
15:26:36:         Time: 19:37:42
15:26:36:      SVN Rev: 3515
15:26:36:       Branch: fah/trunk/client
15:26:36:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
15:26:36:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
15:26:36:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT
15:26:36:     Platform: win32 XP
15:26:36:         Bits: 32
15:26:36:         Mode: Release
15:26:36:******************************* System ********************************
15:26:36:          CPU: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
15:26:36:       CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
15:26:36:         CPUs: 4
15:26:36:       Memory: 2.91GiB
15:26:36:  Free Memory: 1.69GiB
15:26:36:      Threads: WINDOWS_THREADS
15:26:36:   On Battery: false
15:26:36:   UTC offset: 0
15:26:36:          PID: 3592
15:26:36:          CWD: C:/Users/kiordache/AppData/Roaming/FAHClient
15:26:36:           OS: Windows 7 Professional
15:26:36:      OS Arch: AMD64
15:26:36:         GPUs: 0
15:26:36:         CUDA: Not detected
15:26:36:Win32 Service: false
15:26:36:***********************************************************************
15:26:36:<config>
15:26:36:  <!-- Folding Slot Configuration -->
15:26:36:  <gpu v='true'/>
15:26:36:
15:26:36:  <!-- User Information -->
15:26:36:  <user v='iokeith'/>
15:26:36:
15:26:36:  <!-- Folding Slots -->
15:26:36:</config>
15:26:36:Trying to access database...
15:26:36:Successfully acquired database lock
15:26:36:Enabled folding slot 00: READY smp:4
15:26:36:WU01:FS00:Starting
15:26:36:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/kiordache/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 701 -lifeline 3592 -checkpoint 15 -np 4
15:26:36:WU01:FS00:Started FahCore on PID 4608
15:26:36:WU01:FS00:Core PID:724
15:26:36:WU01:FS00:FahCore 0xa4 started
15:26:36:WU01:FS00:0xa4:
15:26:36:WU01:FS00:0xa4:*------------------------------*
15:26:36:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
15:26:36:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
15:26:36:WU01:FS00:0xa4:
15:26:36:WU01:FS00:0xa4:Preparing to commence simulation
15:26:36:WU01:FS00:0xa4:- Ensuring status. Please wait.
15:26:39:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
15:26:46:WU01:FS00:0xa4:- Looking at optimizations...
15:26:46:WU01:FS00:0xa4:- Working with standard loops on this execution.
15:26:46:WU01:FS00:0xa4:- Previous termination of core was improper.
15:26:46:WU01:FS00:0xa4:- Going to use standard loops.
15:26:46:WU01:FS00:0xa4:- Files status OK
15:26:46:WU01:FS00:0xa4:- Expanded 547409 -> 847748 (decompressed 154.8 percent)
15:26:46:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=547409 data_size=847748, decompressed_data_size=847748 diff=0
15:26:46:WU01:FS00:0xa4:- Digital signature verified
15:26:46:WU01:FS00:0xa4:
15:26:46:WU01:FS00:0xa4:Project: 7645 (Run 266, Clone 0, Gen 142)
15:26:46:WU01:FS00:0xa4:
15:26:46:WU01:FS00:0xa4:Entering M.D.
15:26:52:WU01:FS00:0xa4:Using Gromacs checkpoints
15:26:52:WU01:FS00:0xa4:Mapping NT from 4 to 4 
15:26:52:WU01:FS00:0xa4:Resuming from checkpoint
15:26:52:WU01:FS00:0xa4:Verified 01/wudata_01.log
15:26:53:WU01:FS00:0xa4:Verified 01/wudata_01.trr
15:26:53:WU01:FS00:0xa4:Verified 01/wudata_01.xtc
15:26:53:WU01:FS00:0xa4:Verified 01/wudata_01.edr
15:26:53:WU01:FS00:0xa4:Completed 2335840 out of 2500000 steps  (93%)

and here where it start to fail uploading 11/11/2013:

Code: Select all

23:25:06:WU00:FS00:0xa4:*------------------------------*
23:25:06:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
23:25:06:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
23:25:06:WU00:FS00:0xa4:
23:25:06:WU00:FS00:0xa4:Preparing to commence simulation
23:25:06:WU00:FS00:0xa4:- Looking at optimizations...
23:25:06:WU00:FS00:0xa4:- Created dyn
23:25:06:WU00:FS00:0xa4:- Files status OK
23:25:06:WU00:FS00:0xa4:- Expanded 546487 -> 843968 (decompressed 154.4 percent)
23:25:06:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=546487 data_size=843968, decompressed_data_size=843968 diff=0
23:25:06:WU00:FS00:0xa4:- Digital signature verified
23:25:06:WU00:FS00:0xa4:
23:25:06:WU00:FS00:0xa4:Project: 7646 (Run 48, Clone 0, Gen 171)
23:25:06:WU00:FS00:0xa4:
23:25:06:WU00:FS00:0xa4:Assembly optimizations on if available.
23:25:06:WU00:FS00:0xa4:Entering M.D.
23:25:12:WU00:FS00:0xa4:Mapping NT from 4 to 4 
23:25:12:WU00:FS00:0xa4:Completed 0 out of 2500000 steps  (0%)
23:25:26:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
23:25:26:WU01:FS00:Connecting to 171.64.65.99:80
23:25:26:WU01:FS00:Upload 1.54%
23:25:47:WU01:FS00:Upload 6.17%
23:25:47:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
23:25:47:WU01:FS00:Sending unit results: id:01 state:SEND error:OK project:7808 run:6 clone:42 gen:63 core:0xa4 unit:0x000000570a3b1e874e30fc12d0a5579a
23:25:47:WU01:FS00:Uploading 4.05MiB to 171.64.65.99
23:25:47:WU01:FS00:Connecting to 171.64.65.99:8080
23:26:08:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
23:26:08:WU01:FS00:Connecting to 171.64.65.99:80
23:26:08:WU01:FS00:Upload 1.54%
23:26:29:WU01:FS00:Upload 4.63%

Current configuration:

Code: Select all

*********************** Log Started 2013-11-25T10:10:17Z ***********************
10:10:17:************************* Folding@home Client *************************
10:10:17:      Website: http://folding.stanford.edu/
10:10:17:    Copyright: (c) 2009-2013 Stanford University
10:10:17:       Author: Joseph Coffland <[email protected]>
10:10:17:         Args: 
10:10:17:       Config: C:/Users/kiordache/AppData/Roaming/FAHClient/config.xml
10:10:17:******************************** Build ********************************
10:10:17:      Version: 7.3.6
10:10:17:         Date: Feb 18 2013
10:10:17:         Time: 15:25:17
10:10:17:      SVN Rev: 3923
10:10:17:       Branch: fah/trunk/client
10:10:17:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
10:10:17:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
10:10:17:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
10:10:17:     Platform: win32 XP
10:10:17:         Bits: 32
10:10:17:         Mode: Release
10:10:17:******************************* System ********************************
10:10:17:          CPU: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
10:10:17:       CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
10:10:17:         CPUs: 4
10:10:17:       Memory: 2.91GiB
10:10:17:  Free Memory: 1.84GiB
10:10:17:      Threads: WINDOWS_THREADS
10:10:17:  Has Battery: false
10:10:17:   On Battery: false
10:10:17:   UTC offset: 0
10:10:17:          PID: 3604
10:10:17:          CWD: C:/Users/kiordache/AppData/Roaming/FAHClient
10:10:17:           OS: Windows 7 Professional
10:10:17:      OS Arch: AMD64
10:10:17:         GPUs: 0
10:10:17:         CUDA: Not detected
10:10:17:Win32 Service: false
10:10:17:***********************************************************************
10:10:17:<config>
10:10:17:  <!-- Folding Slot Configuration -->
10:10:17:  <power v='full'/>
10:10:17:
10:10:17:  <!-- Network -->
10:10:17:  <proxy v=':8080'/>
10:10:17:
10:10:17:  <!-- User Information -->
10:10:17:  <user v='iokeith'/>
10:10:17:
10:10:17:  <!-- Folding Slots -->
10:10:17:  <slot id='0' type='CPU'/>
10:10:17:</config>
10:10:17:Trying to access database...
10:10:21:Successfully acquired database lock
10:10:21:Enabled folding slot 00: READY cpu:4
10:10:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:7808 run:7 clone:222 gen:66 core:0xa4 unit:0x000000550a3b1e874e30fefb59367ccc
10:10:23:WU02:FS00:Uploading 4.05MiB to 171.64.65.99
10:10:23:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:7808 run:6 clone:42 gen:63 core:0xa4 unit:0x000000570a3b1e874e30fc12d0a5579a
10:10:23:WU02:FS00:Connecting to 171.64.65.99:8080
10:10:24:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:7809 run:4 clone:106 gen:54 core:0xa4 unit:0x0000004e0a3b1e874e310da387b75cc6
10:10:24:WU01:FS00:Uploading 4.05MiB to 171.64.65.99
10:10:24:WU01:FS00:Connecting to 171.64.65.99:8080
10:10:25:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:7808 run:5 clone:80 gen:163 core:0xa4 unit:0x000000e10a3b1e874e30fa1ab34dba3b
10:10:25:WU03:FS00:Uploading 4.13MiB to 171.64.65.99
10:10:25:WU03:FS00:Connecting to 171.64.65.99:8080
10:10:25:WU04:FS00:Uploading 4.05MiB to 171.64.65.99
10:10:25:WU04:FS00:Connecting to 171.64.65.99:8080
10:10:25:WU05:FS00:Sending unit results: id:05 state:SEND error:NO_ERROR project:7646 run:48 clone:2 gen:167 core:0xa4 unit:0x00000334664f2dcd4fa7fdec3acd14af
10:10:26:WU05:FS00:Uploading 14.01MiB to 171.64.65.101
10:10:26:WU05:FS00:Connecting to 171.64.65.101:8080
10:10:44:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
10:10:44:WU02:FS00:Connecting to 171.64.65.99:80
10:10:44:WU02:FS00:Upload 1.54%

Here are current log entries:

Code: Select all

07:18:25:ERROR:WU05:FS00:Exception: Failed to connect to 171.67.108.49:80: No connection could be made because the target machine actively refused it.
07:18:30:WU08:FS00:Sending unit results: id:08 state:SEND error:NO_ERROR project:7647 run:176 clone:0 gen:148 core:0xa4 unit:0x0000015e664f2dcd4fa7fe945ee556f8
07:18:30:WU08:FS00:Uploading 14.05MiB to 171.64.65.101
07:18:30:WU08:FS00:Connecting to 171.64.65.101:8080
07:18:31:WARNING:WU06:FS00:WorkServer connection failed on port 8080 trying 80
07:18:31:WU06:FS00:Connecting to 128.143.199.97:80
07:18:31:WU06:FS00:Upload 0.87%
07:18:34:WARNING:WU07:FS00:WorkServer connection failed on port 8080 trying 80
07:18:34:WU07:FS00:Connecting to 171.64.65.101:80
07:18:34:WU07:FS00:Upload 0.45%
07:18:40:WU03:FS00:Upload 4.54%
07:18:40:WARNING:WU03:FS00:Exception: Failed to send results to work server: Transfer failed
07:18:41:WU09:FS00:Sending unit results: id:09 state:SEND error:NO_ERROR project:10450 run:54 clone:0 gen:199 core:0xa4 unit:0x000007620a3b1e7550a539e4a227a792
07:18:41:WU09:FS00:Uploading 3.74MiB to 171.64.65.81
07:18:41:WU09:FS00:Connecting to 171.64.65.81:8080
07:18:42:WU02:FS00:Upload 3.09%
07:18:42:WARNING:WU02:FS00:Exception: Failed to send results to work server: Transfer failed
07:18:51:WARNING:WU08:FS00:WorkServer connection failed on port 8080 trying 80
07:18:51:WU06:FS00:Upload 1.74%
07:18:51:WU08:FS00:Connecting to 171.64.65.101:80
07:18:51:ERROR:WU06:FS00:Exception: Transfer failed
07:18:51:WU08:FS00:Upload 0.44%
07:18:54:WU07:FS00:Upload 1.34%
07:18:55:WARNING:WU07:FS00:Exception: Failed to send results to work server: Transfer failed
07:18:55:WU07:FS00:Trying to send results to collection server
07:18:55:WU07:FS00:Uploading 14.02MiB to 171.67.108.49
07:18:55:WU07:FS00:Connecting to 171.67.108.49:8080
07:19:02:WARNING:WU09:FS00:WorkServer connection failed on port 8080 trying 80
07:19:02:WU09:FS00:Connecting to 171.64.65.81:80
07:19:02:WU09:FS00:Upload 1.67%
07:19:11:WU08:FS00:Upload 1.33%
07:19:11:WARNING:WU08:FS00:Exception: Failed to send results to work server: Transfer failed
07:19:16:WARNING:WU07:FS00:WorkServer connection failed on port 8080 trying 80
07:19:16:WU07:FS00:Connecting to 171.67.108.49:80
07:19:17:ERROR:WU07:FS00:Exception: Failed to connect to 171.67.108.49:80: No connection could be made because the target machine actively refused it.
07:19:22:WU09:FS00:Upload 3.34%
07:19:22:WARNING:WU09:FS00:Exception: Failed to send results to work server: Transfer failed
07:19:22:WU09:FS00:Trying to send results to collection server
07:19:22:WU09:FS00:Uploading 3.74MiB to 171.65.103.160
07:19:22:WU09:FS00:Connecting to 171.65.103.160:8080
07:19:43:WARNING:WU09:FS00:WorkServer connection failed on port 8080 trying 80
07:19:43:WU09:FS00:Connecting to 171.65.103.160:80
07:19:44:WU09:FS00:Upload 1.67%
07:20:04:WU09:FS00:Upload 5.01%
07:20:04:ERROR:WU09:FS00:Exception: Transfer failed

[bollix47]

Server 171.64.65.99 has had a few problems lately but does appear to be working at the moment. Try pausing the slot, exit the client, reboot the computer plus any communication devices and start the client if it hasn't already started automatically when windows started.

[PantherX]

171.64.65.99 -> Work Server is fully functional according to the Server Status (viewtopic.php?f=18&t=25318)

171.67.108.49 -> Server is a Collection Server so it can be ignored since it isn't expected to work. The Work Server is the issue here and not the Collection Server.

171.64.65.101 -> Work Server is fully functional according to the Server Status (viewtopic.php?f=18&t=25304).

171.64.65.81 -> Work Server is fully functional according to the Server Status.

171.65.103.160 -> Server is accepting WUs according to the Server Status.

It seems that there might be an internet connection issue since the transfer fails within a minute. Since you have stated that 9 WUs are waiting to upload, it seems that there might be running into a bandwidth contention issue. Unfortunately, there is nothing much that can be done. However, I do hope that it gets improved in a future release (https://fah.stanford.edu/projects/FAHClient/ticket/1038).

[7im]

Would someone please explain how to use the --send command from the CLI to send each work unit to avoid the bandwidth issue? Not near a PC for a while...

[bollix47]

One option you could try changing is the connection-timeout.

Configure > Expert

Add the option to the Extra Client Options (left pane):

Name: connection-timeout
Value: 600

Save and restart the client.

This will change your timeout from the default of 60 seconds to 600 seconds and may help with so many uploads occurring at once.

Disclaimer: I've never had a situation where I've had to try this and have no idea if it will actually help in this case.

[bollix47]

@ 7im

According to this page:

Code: Select all

-send all 	--send 	V7.0.7+, attempts to upload all completed work units to available servers
-send # 	  Not supported

If that info is old and it does work for individual work units (WU##) then the procedure should be:

Pause the slot(s). Stop the client. Open a command prompt and type the following:

FAHClient --send ##

Repeat for each work unit waiting to upload.

[7im]

--send ## does not work. Renaming all but one folder would send just that one WUs data. Then cycle through the rest.

[iokeith]

Thanks for your ideas! I'll check them out when i can.

[7im]

I haven't tried this yet, but should work. Of course, make a backup copy of these folders before starting in case it does not.

Stop the client.

Find your work folder. C:\Users\[your_user_name]\AppData\Roaming\FAHClient\work\ or go to Start/All Programs/FAHClient/Data Directory, Work folder.

You should have several folders like 00, 01, 02, 03, etc. Move all but one of those folders to a different location.

From a command line (DOS prompt) type in C:\Program Files (x86)\FAHClient\FAHClient --send

If you have the work folder of the currently active WU, nothing will be uploaded. Just leave it there for now. If you have the work folder for one of the completed WUs, it should try to upload. When done uploading, delete that work folder. When done, copy in the next work folder, rinse, repeat. When all done, you can delete the copies and start folding again.

[bruce]

Unfortunately this is a lot more difficult than it sounds. Suppose 01, 02, and 03 are renamed so 00 can be uploaded by itself. The client will detect the missing folders and will delete 01, 02, and 03 from the queue (database). I'm not sure if it will send a DUMPED report for them or not, but it might.

Once that's completed, moving 01 back into the work folder will not re-enter it in the queue (database), so, in fact, you'll have to restore the entire work folder plus the database and then delete 00 (which has already been sent). Repeating that process several times will eventually succeed PROVIDED it's actually a congested upload rather than a DOWNed server or something else like that.

If dumped reports were issued for 01, 02, and 03 when you started, then you won't be able to upload them for credit so it's a very tricky process. The backup also may not help. Once the server gets a report regarding a WU, it will reject additional attempts to upload the same WU, even if the first report was that you dumped it.

There's a good reason why the EULA says don't mess with FAH's files.

[iokeith]

I've tried out both ideas but its still refusing to upload and the log reports the same errors, so I've decided to put folding home on ice for now. Maybe try again next year sometime.
Thanks for your help!!