Rejection due to transmission failure? [WS 128.252.203.4]

[Foxbat]

I guess you can add me to the list. I thought it might be an issue with the server (128.252.203.4 which doesn't appear on the FAH Servers Status Page, BTW) and this all started when my iMac tried uploading a Project 13804 WU:

Code: Select all

*********************** Log Started 2017-05-21T05:48:03Z ***********************
05:48:03:************************* Folding@home Client *************************
05:48:03:    Website: http://folding.stanford.edu/
05:48:03:  Copyright: (c) 2009-2016 Stanford University
05:48:03:     Author: Joseph Coffland <[email protected]>
05:48:03:       Args: 
05:48:03:     Config: /Library/Application Support/FAHClient/config.xml
05:48:03:******************************** Build ********************************
05:48:03:    Version: 7.4.16
05:48:03:       Date: Jan 6 2017
05:48:03:       Time: 00:48:08
05:48:03: Repository: Git
05:48:03:   Revision: aa90508b6a2f89afb7f8ca8381b29459a8f64f73
05:48:03:     Branch: master
05:48:03:   Compiler: GNU 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)
05:48:03:    Options: -std=gnu++98 -O3 -funroll-loops -msse3 -mmacosx-version-min=10.6
05:48:03:             -Wno-unused-local-typedefs
05:48:03:   Platform: darwin 15.6.0
05:48:03:       Bits: 64
05:48:03:       Mode: Release
05:48:03:******************************* System ********************************
05:48:03:        CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
05:48:03:     CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
05:48:03:       CPUs: 8
05:48:03:     Memory: 8.00GiB
05:48:03:Free Memory: 7.13GiB
05:48:03:    Threads: POSIX_THREADS
05:48:03: OS Version: 10.12
05:48:03:Has Battery: false
05:48:03: On Battery: false
05:48:03: UTC Offset: -4
05:48:03:        PID: 93
05:48:03:        CWD: /Library/Application Support/FAHClient
05:48:03:         OS: Darwin 16.6.0 x86_64
05:48:03:    OS Arch: AMD64
05:48:03:       GPUs: 0
05:48:03:       CUDA: Not detected: Failed to open dynamic library 'libcuda.dylib':
05:48:03:             dlopen(libcuda.dylib, 1): image not found
05:48:03:     OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.dylib':
05:48:03:             dlopen(libOpenCL.dylib, 1): image not found
05:48:03:***********************************************************************
05:48:04:<config>
05:48:04:  <!-- Folding Core -->
05:48:04:  <core-priority v='low'/>
05:48:04:
05:48:04:  <!-- HTTP Server -->
05:48:04:  <allow v='127.0.0.1 ***.***.***.0/24'/>
05:48:04:
05:48:04:  <!-- Network -->
05:48:04:  <proxy v=':8080'/>
05:48:04:
05:48:04:  <!-- Remote Command Server -->
05:48:04:  <command-allow-no-pass v='127.0.0.1 ***.***.***.0/24'/>
05:48:04:  <password v='**********'/>
05:48:04:
05:48:04:  <!-- Slot Control -->
05:48:04:  <pause-on-battery v='false'/>
05:48:04:  <power v='light'/>
05:48:04:
05:48:04:  <!-- User Information -->
05:48:04:  <passkey v='********************************'/>
05:48:04:  <team v='55236'/>
05:48:04:  <user v='Foxbat'/>
05:48:04:
05:48:04:  <!-- Work Unit Control -->
05:48:04:  <next-unit-percentage v='100'/>
05:48:04:
05:48:04:  <!-- Folding Slots -->
05:48:04:  <slot id='0' type='CPU'/>
05:48:04:</config>
05:48:04:Trying to access database...
05:48:04:Successfully acquired database lock
05:48:04:Enabled folding slot 00: READY cpu:3
05:48:05:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:13804 run:0 clone:1312 gen:29 core:0xa7 unit:0x0000002480fccb0458dc00c821f0e637
05:48:05:WU01:FS00:Uploading 12.17MiB to 128.252.203.4
05:48:05:WU01:FS00:Connecting to 128.252.203.4:8080
05:48:44:Caught signal SIGPIPE(13) on PID 93
05:48:44:WU01:FS00:Upload 1.54%
05:48:44:Caught signal SIGPIPE(13) on PID 93
05:48:44:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
05:49:42:WU01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13804 run:0 clone:1312 gen:29 core:0xa7 unit:0x0000002480fccb0458dc00c821f0e637
05:49:42:WU01:Uploading 12.17MiB to 128.252.203.4
05:49:42:WU01:Connecting to 128.252.203.4:8080
05:50:19:Caught signal SIGPIPE(13) on PID 93
05:50:19:WU01:Upload 1.54%
05:50:19:Caught signal SIGPIPE(13) on PID 93
05:50:19:WARNING:WU01:Exception: Failed to send results to work server: Transfer failed
05:52:19:WU01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13804 run:0 clone:1312 gen:29 core:0xa7 unit:0x0000002480fccb0458dc00c821f0e637
05:52:19:WU01:Uploading 12.17MiB to 128.252.203.4
05:52:19:WU01:Connecting to 128.252.203.4:8080
05:52:56:Caught signal SIGPIPE(13) on PID 93
05:52:56:WU01:Upload 1.54%
05:52:56:Caught signal SIGPIPE(13) on PID 93
05:52:56:WARNING:WU01:Exception: Failed to send results to work server: Transfer failed
05:56:34:WU01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13804 run:0 clone:1312 gen:29 core:0xa7 unit:0x0000002480fccb0458dc00c821f0e637
05:56:34:WU01:Uploading 12.17MiB to 128.252.203.4
05:56:34:WU01:Connecting to 128.252.203.4:8080
05:56:46:WU01:Upload 1.54%
06:10:10:WU01:Upload 3.08%
06:10:16:WU01:Upload 13.86%
06:10:22:WU01:Upload 25.16%
06:10:28:WU01:Upload 33.89%
06:10:34:WU01:Upload 45.19%
06:10:40:WU01:Upload 54.94%
06:10:46:WU01:Upload 67.27%
06:10:52:WU01:Upload 77.02%
06:10:58:WU01:Upload 83.19%
06:11:04:WU01:Upload 92.94%
06:11:09:WU01:Upload complete
06:11:10:WU01:Server responded WORK_QUIT (404)
06:11:10:WARNING:WU01:Server did not like results, dumping
06:11:10:WU01:Cleaning up

(edited to remove other WU information)

[montemac]

Two of my computers have failed to upload work units this morning with logs similar to the following:

Code: Select all

06:09:49:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
06:09:49:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:09:49:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13804 run:0 clone:4302 gen:6 core:0xa7 unit:0x0000000b80fccb04591a03292f012c6f
06:09:49:WU00:FS00:Uploading 12.21MiB to 128.252.203.4
06:09:49:WU00:FS00:Connecting to 128.252.203.4:8080
06:09:49:WU01:FS00:Starting
06:09:49:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/montemac/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 9880 -checkpoint 10 -np 4
06:09:49:WU01:FS00:Started FahCore on PID 5540
06:09:49:WU01:FS00:Core PID:2952
06:09:49:WU01:FS00:FahCore 0xa4 started
06:09:50:WU01:FS00:0xa4:
06:09:50:WU01:FS00:0xa4:*------------------------------*
06:09:50:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
06:09:50:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
06:09:50:WU01:FS00:0xa4:
06:09:50:WU01:FS00:0xa4:Preparing to commence simulation
06:09:50:WU01:FS00:0xa4:- Looking at optimizations...
06:09:50:WU01:FS00:0xa4:- Created dyn
06:09:50:WU01:FS00:0xa4:- Files status OK
06:09:50:WU01:FS00:0xa4:- Expanded 832120 -> 2144424 (decompressed 257.7 percent)
06:09:50:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=832120 data_size=2144424, decompressed_data_size=2144424 diff=0
06:09:50:WU01:FS00:0xa4:- Digital signature verified
06:09:50:WU01:FS00:0xa4:
06:09:50:WU01:FS00:0xa4:Project: 8628 (Run 0, Clone 74, Gen 33)
06:09:50:WU01:FS00:0xa4:
06:09:50:WU01:FS00:0xa4:Assembly optimizations on if available.
06:09:50:WU01:FS00:0xa4:Entering M.D.
06:09:56:WU01:FS00:0xa4:Mapping NT from 4 to 4 
06:09:56:WU01:FS00:0xa4:Completed 0 out of 2500000 steps  (0%)
06:10:11:WU00:FS00:Upload 1.02%
06:10:11:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
06:10:11:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13804 run:0 clone:4302 gen:6 core:0xa7 unit:0x0000000b80fccb04591a03292f012c6f
06:10:11:WU00:FS00:Uploading 12.21MiB to 128.252.203.4
06:10:11:WU00:FS00:Connecting to 128.252.203.4:8080
06:10:31:WU00:FS00:Upload 1.02%
06:10:31:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
06:11:11:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13804 run:0 clone:4302 gen:6 core:0xa7 unit:0x0000000b80fccb04591a03292f012c6f
06:11:11:WU00:FS00:Uploading 12.21MiB to 128.252.203.4
06:11:11:WU00:FS00:Connecting to 128.252.203.4:8080
06:11:31:WU00:FS00:Upload 1.02%
06:11:31:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
06:12:48:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13804 run:0 clone:4302 gen:6 core:0xa7 unit:0x0000000b80fccb04591a03292f012c6f
06:12:48:WU00:FS00:Uploading 12.21MiB to 128.252.203.4
06:12:48:WU00:FS00:Connecting to 128.252.203.4:8080
06:12:54:WU00:FS00:Upload 1.02%
06:23:51:WU00:FS00:Upload 2.56%
06:23:57:WU00:FS00:Upload 34.79%
06:24:03:WU00:FS00:Upload 69.59%
06:24:08:WU00:FS00:Upload complete
06:24:08:WU00:FS00:Server responded WORK_QUIT (404)
06:24:08:WARNING:WU00:FS00:Server did not like results, dumping
06:24:08:WU00:FS00:Cleaning up
06:24:51:WU01:FS00:0xa4:Completed 25000 out of 2500000 steps  (1%)

Mod edit: added Code tags, merged with similar report

[Joe_H]

I also have a WU that is has tried to upload to this WS and not managed to send it in. I will send a message to the project manager using this WS.

[justinrporter]

Thanks for the heads up everybody. The server seems to have starting panicking for some reason. I've restarted it and I've been seeing successful submissions. Hopefully this solves the issue.

[montemac]

What happens to my two work units that failed to transfer? Will FAH automatically try again?

Thanks

[Joe_H]

Unfortunately, if the upload reached the point where it was examined and dumped, that will not be retried. At that point the WU is gone, and eventually will be reassigned to another.

But if the upload just failed and did not get dumped, eventually the FAH client should retry the transmission to the WS. Occasionally a bug will cause the client to not detect that an upload has failed and retry sending. In that case a restart of the FAHClient process will send any waiting WU's in. The easiest way to do the restart is a reboot.

[DrBB1]

Mod note: merged with existing report of problem with that WS during that timeframe, should be fixed now. - j

Not sure what happened, but apparently there was a problem with uploading this WU. This was a long WU (for my PC, anyway), so it would be a shame if the results are gone forever. Even if they are, I am hoping someone can figure out the problem and fix it for the future. I have included the log for the last few frames and the aftermath below.

Code: Select all

******************************* Date: 2017-05-21 *******************************
10:35:23:WU00:FS01:0xa7:Completed 230000 out of 250000 steps (92%)
10:44:16:WU00:FS01:0xa7:Completed 232500 out of 250000 steps (93%)
10:53:09:WU00:FS01:0xa7:Completed 235000 out of 250000 steps (94%)
11:02:03:WU00:FS01:0xa7:Completed 237500 out of 250000 steps (95%)
11:10:55:WU00:FS01:0xa7:Completed 240000 out of 250000 steps (96%)
11:19:47:WU00:FS01:0xa7:Completed 242500 out of 250000 steps (97%)
11:28:40:WU00:FS01:0xa7:Completed 245000 out of 250000 steps (98%)
11:37:32:WU00:FS01:0xa7:Completed 247500 out of 250000 steps (99%)
11:37:33:WU01:FS01:Connecting to 171.67.108.45:8080
11:37:33:WU01:FS01:Assigned to work server 171.67.108.101
11:37:33:WU01:FS01:Requesting new work unit for slot 01: RUNNING cpu:4 from 171.67.108.101
11:37:33:WU01:FS01:Connecting to 171.67.108.101:8080
11:37:34:ERROR:WU01:FS01:Exception: Server did not assign work unit
11:37:34:WU01:FS01:Connecting to 171.67.108.45:8080
11:37:34:WU01:FS01:Assigned to work server 171.67.108.158
11:37:34:WU01:FS01:Requesting new work unit for slot 01: RUNNING cpu:4 from 171.67.108.158
11:37:34:WU01:FS01:Connecting to 171.67.108.158:8080
11:37:35:WU01:FS01:Downloading 806.27KiB
11:37:35:WU01:FS01:Download complete
11:37:35:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9032 run:605 clone:0 gen:937 core:0xa4 unit:0x00000403ab436c9e569832b2112ed443
11:46:24:WU00:FS01:0xa7:Completed 250000 out of 250000 steps (100%)
11:46:30:WU00:FS01:0xa7:Saving result file ..\logfile_01.txt
11:46:30:WU00:FS01:0xa7:Saving result file frame115.trr
11:46:30:WU00:FS01:0xa7:Saving result file frame115.xtc
11:46:30:WU00:FS01:0xa7:Saving result file md.log
11:46:30:WU00:FS01:0xa7:Saving result file science.log
11:46:30:WU00:FS01:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
11:46:31:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:46:31:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:46:31:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:46:31:WU00:FS01:Connecting to 128.252.203.4:8080
11:46:31:WU01:FS01:Starting
11:46:31:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Main/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 2108 -checkpoint 30 -np 4
11:46:31:WU01:FS01:Started FahCore on PID 1072
11:46:31:WU01:FS01:Core PID:3644
11:46:31:WU01:FS01:FahCore 0xa4 started
11:46:32:WU01:FS01:0xa4:
11:46:32:WU01:FS01:0xa4:*------------------------------*
11:46:32:WU01:FS01:0xa4:Folding@Home Gromacs GB Core
11:46:32:WU01:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
11:46:32:WU01:FS01:0xa4:
11:46:32:WU01:FS01:0xa4:Preparing to commence simulation
11:46:32:WU01:FS01:0xa4:- Looking at optimizations...
11:46:32:WU01:FS01:0xa4:- Created dyn
11:46:32:WU01:FS01:0xa4:- Files status OK
11:46:32:WU01:FS01:0xa4:- Expanded 825109 -> 1398024 (decompressed 169.4 percent)
11:46:32:WU01:FS01:0xa4:Called DecompressByteArray: compressed_data_size=825109 data_size=1398024, decompressed_data_size=1398024 diff=0
11:46:32:WU01:FS01:0xa4:- Digital signature verified
11:46:32:WU01:FS01:0xa4:
11:46:32:WU01:FS01:0xa4:Project: 9032 (Run 605, Clone 0, Gen 937)
11:46:32:WU01:FS01:0xa4:
11:46:32:WU01:FS01:0xa4:Assembly optimizations on if available.
11:46:32:WU01:FS01:0xa4:Entering M.D.
11:46:38:WU01:FS01:0xa4:Mapping NT from 4 to 4 
11:46:38:WU01:FS01:0xa4:Completed 0 out of 250000 steps  (0%)
11:47:53:WU00:FS01:Upload 3.62%
11:47:53:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:47:53:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:47:53:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:47:53:WU00:FS01:Connecting to 128.252.203.4:8080
11:49:05:WU01:FS01:0xa4:Completed 2500 out of 250000 steps  (1%)
11:49:10:WU00:FS01:Upload 3.62%
11:49:10:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:49:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:49:10:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:49:10:WU00:FS01:Connecting to 128.252.203.4:8080
11:50:28:WU00:FS01:Upload 3.62%
11:50:28:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:50:48:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:50:48:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:50:48:WU00:FS01:Connecting to 128.252.203.4:8080
11:50:57:WU00:FS01:Upload 0.72%
11:51:34:WU01:FS01:0xa4:Completed 5000 out of 250000 steps  (2%)
11:52:06:WU00:FS01:Upload 3.62%
11:52:06:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:53:25:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:53:25:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:53:25:WU00:FS01:Connecting to 128.252.203.4:8080
11:54:03:WU01:FS01:0xa4:Completed 7500 out of 250000 steps  (3%)
11:54:47:WU00:FS01:Upload 3.62%
11:54:47:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:56:31:WU01:FS01:0xa4:Completed 10000 out of 250000 steps  (4%)
11:57:39:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
11:57:39:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
11:57:39:WU00:FS01:Connecting to 128.252.203.4:8080
11:58:59:WU01:FS01:0xa4:Completed 12500 out of 250000 steps  (5%)
11:59:01:WU00:FS01:Upload 3.62%
11:59:01:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:01:27:WU01:FS01:0xa4:Completed 15000 out of 250000 steps  (6%)
12:03:54:WU01:FS01:0xa4:Completed 17500 out of 250000 steps  (7%)
12:04:31:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
12:04:31:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
12:04:31:WU00:FS01:Connecting to 128.252.203.4:8080
12:05:43:WU00:FS01:Upload 3.62%
12:05:43:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:06:22:WU01:FS01:0xa4:Completed 20000 out of 250000 steps  (8%)
12:08:52:WU01:FS01:0xa4:Completed 22500 out of 250000 steps  (9%)
12:11:20:WU01:FS01:0xa4:Completed 25000 out of 250000 steps  (10%)
12:13:46:WU01:FS01:0xa4:Completed 27500 out of 250000 steps  (11%)
12:15:36:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
12:15:36:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
12:15:36:WU00:FS01:Connecting to 128.252.203.4:8080
12:16:14:WU01:FS01:0xa4:Completed 30000 out of 250000 steps  (12%)
12:16:58:WU00:FS01:Upload 3.62%
12:16:58:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:18:42:WU01:FS01:0xa4:Completed 32500 out of 250000 steps  (13%)
12:21:10:WU01:FS01:0xa4:Completed 35000 out of 250000 steps  (14%)
12:23:39:WU01:FS01:0xa4:Completed 37500 out of 250000 steps  (15%)
12:26:06:WU01:FS01:0xa4:Completed 40000 out of 250000 steps  (16%)
12:28:33:WU01:FS01:0xa4:Completed 42500 out of 250000 steps  (17%)
12:31:00:WU01:FS01:0xa4:Completed 45000 out of 250000 steps  (18%)
12:33:27:WU01:FS01:0xa4:Completed 47500 out of 250000 steps  (19%)
12:33:33:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
12:33:33:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
12:33:33:WU00:FS01:Connecting to 128.252.203.4:8080
12:34:54:WU00:FS01:Upload 3.62%
12:34:54:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:35:55:WU01:FS01:0xa4:Completed 50000 out of 250000 steps  (20%)
12:38:24:WU01:FS01:0xa4:Completed 52500 out of 250000 steps  (21%)
12:40:51:WU01:FS01:0xa4:Completed 55000 out of 250000 steps  (22%)
12:43:18:WU01:FS01:0xa4:Completed 57500 out of 250000 steps  (23%)
12:45:46:WU01:FS01:0xa4:Completed 60000 out of 250000 steps  (24%)
12:48:15:WU01:FS01:0xa4:Completed 62500 out of 250000 steps  (25%)
12:50:44:WU01:FS01:0xa4:Completed 65000 out of 250000 steps  (26%)
12:53:12:WU01:FS01:0xa4:Completed 67500 out of 250000 steps  (27%)
12:55:40:WU01:FS01:0xa4:Completed 70000 out of 250000 steps  (28%)
12:58:08:WU01:FS01:0xa4:Completed 72500 out of 250000 steps  (29%)
13:00:36:WU01:FS01:0xa4:Completed 75000 out of 250000 steps  (30%)
13:02:35:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
13:02:35:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
13:02:35:WU00:FS01:Connecting to 128.252.203.4:8080
13:03:05:WU01:FS01:0xa4:Completed 77500 out of 250000 steps  (31%)
13:03:57:WU00:FS01:Upload 3.62%
13:03:57:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:05:32:WU01:FS01:0xa4:Completed 80000 out of 250000 steps  (32%)
13:08:01:WU01:FS01:0xa4:Completed 82500 out of 250000 steps  (33%)
13:10:41:WU01:FS01:0xa4:Completed 85000 out of 250000 steps  (34%)
13:13:09:WU01:FS01:0xa4:Completed 87500 out of 250000 steps  (35%)
13:15:36:WU01:FS01:0xa4:Completed 90000 out of 250000 steps  (36%)
13:18:04:WU01:FS01:0xa4:Completed 92500 out of 250000 steps  (37%)
13:20:32:WU01:FS01:0xa4:Completed 95000 out of 250000 steps  (38%)
13:23:01:WU01:FS01:0xa4:Completed 97500 out of 250000 steps  (39%)
13:25:29:WU01:FS01:0xa4:Completed 100000 out of 250000 steps  (40%)
13:27:58:WU01:FS01:0xa4:Completed 102500 out of 250000 steps  (41%)
13:30:25:WU01:FS01:0xa4:Completed 105000 out of 250000 steps  (42%)
13:32:53:WU01:FS01:0xa4:Completed 107500 out of 250000 steps  (43%)
13:35:20:WU01:FS01:0xa4:Completed 110000 out of 250000 steps  (44%)
13:37:48:WU01:FS01:0xa4:Completed 112500 out of 250000 steps  (45%)
13:40:17:WU01:FS01:0xa4:Completed 115000 out of 250000 steps  (46%)
13:42:44:WU01:FS01:0xa4:Completed 117500 out of 250000 steps  (47%)
13:45:11:WU01:FS01:0xa4:Completed 120000 out of 250000 steps  (48%)
13:47:40:WU01:FS01:0xa4:Completed 122500 out of 250000 steps  (49%)
13:49:34:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:13800 run:0 clone:349 gen:115 core:0xa7 unit:0x0000008b80fccb045873be835d5f5063
13:49:34:WU00:FS01:Uploading 8.62MiB to 128.252.203.4
13:49:34:WU00:FS01:Connecting to 128.252.203.4:8080
13:50:07:WU01:FS01:0xa4:Completed 125000 out of 250000 steps  (50%)
13:52:34:WU01:FS01:0xa4:Completed 127500 out of 250000 steps  (51%)
13:55:02:WU01:FS01:0xa4:Completed 130000 out of 250000 steps  (52%)
13:57:31:WU01:FS01:0xa4:Completed 132500 out of 250000 steps  (53%)
13:59:59:WU01:FS01:0xa4:Completed 135000 out of 250000 steps  (54%)
14:02:27:WU01:FS01:0xa4:Completed 137500 out of 250000 steps  (55%)
14:04:55:WU01:FS01:0xa4:Completed 140000 out of 250000 steps  (56%)
14:07:23:WU01:FS01:0xa4:Completed 142500 out of 250000 steps  (57%)
14:09:52:WU01:FS01:0xa4:Completed 145000 out of 250000 steps  (58%)
14:12:19:WU01:FS01:0xa4:Completed 147500 out of 250000 steps  (59%)
14:14:47:WU01:FS01:0xa4:Completed 150000 out of 250000 steps  (60%)
14:17:17:WU01:FS01:0xa4:Completed 152500 out of 250000 steps  (61%)
14:19:45:WU01:FS01:0xa4:Completed 155000 out of 250000 steps  (62%)
14:22:12:WU01:FS01:0xa4:Completed 157500 out of 250000 steps  (63%)
14:24:41:WU01:FS01:0xa4:Completed 160000 out of 250000 steps  (64%)
14:27:10:WU01:FS01:0xa4:Completed 162500 out of 250000 steps  (65%)
14:29:38:WU01:FS01:0xa4:Completed 165000 out of 250000 steps  (66%)
14:32:05:WU01:FS01:0xa4:Completed 167500 out of 250000 steps  (67%)
14:34:32:WU01:FS01:0xa4:Completed 170000 out of 250000 steps  (68%)
14:37:01:WU01:FS01:0xa4:Completed 172500 out of 250000 steps  (69%)
14:38:54:WU00:FS01:Upload 5.80%
14:38:58:WU00:FS01:Upload complete
14:38:58:WU00:FS01:Server responded WORK_QUIT (404)
14:38:58:WARNING:WU00:FS01:Server did not like results, dumping
14:38:58:WU00:FS01:Cleaning up
14:39:29:WU01:FS01:0xa4:Completed 175000 out of 250000 steps  (70%)
14:41:56:WU01:FS01:0xa4:Completed 177500 out of 250000 steps  (71%)
14:44:24:WU01:FS01:0xa4:Completed 180000 out of 250000 steps  (72%)
14:46:52:WU01:FS01:0xa4:Completed 182500 out of 250000 steps  (73%)
14:49:22:WU01:FS01:0xa4:Completed 185000 out of 250000 steps  (74%)
14:51:51:WU01:FS01:0xa4:Completed 187500 out of 250000 steps  (75%)
14:54:20:WU01:FS01:0xa4:Completed 190000 out of 250000 steps  (76%)
14:56:48:WU01:FS01:0xa4:Completed 192500 out of 250000 steps  (77%)
14:59:17:WU01:FS01:0xa4:Completed 195000 out of 250000 steps  (78%)
15:01:45:WU01:FS01:0xa4:Completed 197500 out of 250000 steps  (79%)
15:04:13:WU01:FS01:0xa4:Completed 200000 out of 250000 steps  (80%)
15:06:40:WU01:FS01:0xa4:Completed 202500 out of 250000 steps  (81%)
15:09:10:WU01:FS01:0xa4:Completed 205000 out of 250000 steps  (82%)
15:11:38:WU01:FS01:0xa4:Completed 207500 out of 250000 steps  (83%)
15:14:06:WU01:FS01:0xa4:Completed 210000 out of 250000 steps  (84%)
15:16:34:WU01:FS01:0xa4:Completed 212500 out of 250000 steps  (85%)
15:19:04:WU01:FS01:0xa4:Completed 215000 out of 250000 steps  (86%)
15:21:31:WU01:FS01:0xa4:Completed 217500 out of 250000 steps  (87%)
15:24:00:WU01:FS01:0xa4:Completed 220000 out of 250000 steps  (88%)
15:26:27:WU01:FS01:0xa4:Completed 222500 out of 250000 steps  (89%)
15:28:57:WU01:FS01:0xa4:Completed 225000 out of 250000 steps  (90%)
15:31:25:WU01:FS01:0xa4:Completed 227500 out of 250000 steps  (91%)
15:33:53:WU01:FS01:0xa4:Completed 230000 out of 250000 steps  (92%)
15:36:21:WU01:FS01:0xa4:Completed 232500 out of 250000 steps  (93%)
15:38:52:WU01:FS01:0xa4:Completed 235000 out of 250000 steps  (94%)
15:41:19:WU01:FS01:0xa4:Completed 237500 out of 250000 steps  (95%)
15:43:47:WU01:FS01:0xa4:Completed 240000 out of 250000 steps  (96%)
15:46:15:WU01:FS01:0xa4:Completed 242500 out of 250000 steps  (97%)
15:48:45:WU01:FS01:0xa4:Completed 245000 out of 250000 steps  (98%)
15:51:13:WU01:FS01:0xa4:Completed 247500 out of 250000 steps  (99%)
15:51:13:WU00:FS01:Connecting to 171.67.108.45:8080
15:51:13:WU00:FS01:Assigned to work server 128.252.203.2
15:51:13:WU00:FS01:Requesting new work unit for slot 01: RUNNING cpu:4 from 128.252.203.2
15:51:13:WU00:FS01:Connecting to 128.252.203.2:8080
15:51:15:WU00:FS01:Downloading 1.85MiB
15:51:15:WU00:FS01:Download complete
15:51:15:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11659 run:19 clone:3 gen:154 core:0xa4 unit:0x000000ac4c71bbb0581f25d550ddf240
15:53:42:WU01:FS01:0xa4:Completed 250000 out of 250000 steps  (100%)
15:53:43:WU01:FS01:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
15:53:53:WU01:FS01:0xa4:
15:53:53:WU01:FS01:0xa4:Finished Work Unit:
15:53:53:WU01:FS01:0xa4:- Reading up to 811536 from "01/wudata_01.trr": Read 811536
15:53:53:WU01:FS01:0xa4:trr file hash check passed.
15:53:53:WU01:FS01:0xa4:- Reading up to 745956 from "01/wudata_01.xtc": Read 745956
15:53:53:WU01:FS01:0xa4:xtc file hash check passed.
15:53:53:WU01:FS01:0xa4:edr file hash check passed.
15:53:53:WU01:FS01:0xa4:logfile size: 22932
15:53:53:WU01:FS01:0xa4:Leaving Run
15:53:58:WU01:FS01:0xa4:- Writing 1582912 bytes of core data to disk...
15:53:58:WU01:FS01:0xa4:Done: 1582400 -> 1537784 (compressed to 97.1 percent)
15:53:58:WU01:FS01:0xa4:  ... Done.
15:53:58:WU01:FS01:0xa4:- Shutting down core
15:53:58:WU01:FS01:0xa4:
15:53:58:WU01:FS01:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
15:53:59:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
15:53:59:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:9032 run:605 clone:0 gen:937 core:0xa4 unit:0x00000403ab436c9e569832b2112ed443
15:53:59:WU01:FS01:Uploading 1.47MiB to 171.67.108.158
15:53:59:WU00:FS01:Starting
15:53:59:WU01:FS01:Connecting to 171.67.108.158:8080
15:53:59:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/Main/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 2108 -checkpoint 30 -np 4
15:53:59:WU00:FS01:Started FahCore on PID 4988
15:53:59:WU00:FS01:Core PID:5620
15:53:59:WU00:FS01:FahCore 0xa4 started
15:54:00:WU00:FS01:0xa4:
15:54:00:WU00:FS01:0xa4:*------------------------------*
15:54:00:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
15:54:00:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
15:54:00:WU00:FS01:0xa4:
15:54:00:WU00:FS01:0xa4:Preparing to commence simulation
15:54:00:WU00:FS01:0xa4:- Looking at optimizations...
15:54:00:WU00:FS01:0xa4:- Created dyn
15:54:00:WU00:FS01:0xa4:- Files status OK
15:54:00:WU00:FS01:0xa4:- Expanded 1942779 -> 6219524 (decompressed 320.1 percent)
15:54:00:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=1942779 data_size=6219524, decompressed_data_size=6219524 diff=0
15:54:00:WU00:FS01:0xa4:- Digital signature verified
15:54:00:WU00:FS01:0xa4:
15:54:00:WU00:FS01:0xa4:Project: 11659 (Run 19, Clone 3, Gen 154)
15:54:00:WU00:FS01:0xa4:
15:54:00:WU00:FS01:0xa4:Assembly optimizations on if available.
15:54:00:WU00:FS01:0xa4:Entering M.D.
15:54:01:WU01:FS01:Upload complete
15:54:01:WU01:FS01:Server responded WORK_ACK (400)
15:54:01:WU01:FS01:Final credit estimate, 1225.00 points
15:54:01:WU01:FS01:Cleaning up
15:54:06:WU00:FS01:0xa4:Mapping NT from 4 to 4 
15:54:06:WU00:FS01:0xa4:Completed 0 out of 1250000 steps  (0%)
16:08:40:WU00:FS01:0xa4:Completed 12500 out of 1250000 steps  (1%)
16:23:07:WU00:FS01:0xa4:Completed 25000 out of 1250000 steps  (2%)
******************************* Date: 2017-05-21 *******************************
16:37:39:WU00:FS01:0xa4:Completed 37500 out of 1250000 steps  (3%)
16:52:10:WU00:FS01:0xa4:Completed 50000 out of 1250000 steps  (4%)
17:06:41:WU00:FS01:0xa4:Completed 62500 out of 1250000 steps  (5%)
17:21:05:WU00:FS01:0xa4:Completed 75000 out of 1250000 steps  (6%)
17:35:30:WU00:FS01:0xa4:Completed 87500 out of 1250000 steps  (7%)
17:50:01:WU00:FS01:0xa4:Completed 100000 out of 1250000 steps  (8%)