Slow internet speeds to 171.64.65.124 [was A4 core]

[parkut]

Got several dumped in last 24 hours. Most recent, three (3) about 5 hours ago

Project: 9018 (Run 64, Clone 6, Gen 15)

Code: Select all

12:37:07:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
12:37:07:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:37:07:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9018 run:64 clone:6 gen:15 core:0xa4 unit:0x0000001cab40417c55b2cec0c449b621
12:37:07:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
12:37:07:WU01:FS00:Connecting to 171.64.65.124:8080
12:37:17:WU01:FS00:Upload 4.26%
12:38:22:WU01:FS00:Upload 8.52%
12:39:04:WU01:FS00:Upload 12.78%
12:39:50:WU01:FS00:Upload 17.03%
12:40:11:WU01:FS00:Upload 21.29%
12:41:10:WU01:FS00:Upload 25.55%
12:41:27:WU01:FS00:Upload 29.81%
12:41:57:WU01:FS00:Upload 34.07%
12:42:45:WU01:FS00:Upload 38.33%
12:43:07:WU01:FS00:Upload 42.59%
12:43:53:WU01:FS00:Upload 46.85%
12:44:24:WU01:FS00:Upload 51.10%
12:44:44:WU01:FS00:Upload 55.36%
12:45:43:WU01:FS00:Upload 59.62%
12:46:10:WU01:FS00:Upload 63.88%
12:46:36:WU01:FS00:Upload 68.14%
12:47:22:WU01:FS00:Upload 72.40%
12:47:56:WU01:FS00:Upload 76.66%
12:48:16:WU01:FS00:Upload 80.92%
12:48:48:WU01:FS00:Upload 85.17%
12:49:30:WU01:FS00:Upload 89.43%
12:49:47:WU01:FS00:Upload 93.69%
12:50:28:WU01:FS00:Upload 97.95%
12:50:56:WU01:FS00:Upload 100.00%
12:55:00:WU01:FS00:Upload complete
12:55:00:WU01:FS00:Server responded WORK_QUIT (404)
12:55:00:WARNING:WU01:FS00:Server did not like results, dumping
12:55:00:WU01:FS00:Cleaning up

Project: 9026 (Run 64, Clone 6, Gen 15)

Code: Select all

12:01:15:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9026 run:718 clone:5 gen:110 core:0xa4 unit:0x0000007dab40417c55f39c9586800390
12:01:15:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
12:01:15:WU00:FS00:Connecting to 171.64.65.124:8080
12:03:00:WU00:FS00:Upload 8.52%
12:03:41:WU00:FS00:Upload 12.78%
12:04:07:WU00:FS00:Upload 17.04%
12:04:52:WU00:FS00:Upload 21.30%
12:05:07:WU00:FS00:Upload 25.56%
12:05:43:WU00:FS00:Upload 29.82%
12:06:12:WU00:FS00:Upload 34.08%
12:07:06:WU00:FS00:Upload 38.34%
12:07:34:WU00:FS00:Upload 42.60%
12:08:17:WU00:FS00:Upload 46.86%
12:08:57:WU00:FS00:Upload 51.12%
12:10:06:WU00:FS00:Upload 55.38%
12:10:44:WU00:FS00:Upload 59.64%
12:10:58:WU00:FS00:Upload 63.90%
12:11:39:WU00:FS00:Upload 68.16%
12:12:15:WU00:FS00:Upload 72.42%
12:12:46:WU00:FS00:Upload 76.68%
12:13:12:WU00:FS00:Upload 80.94%
12:14:20:WU00:FS00:Upload 85.20%
12:14:49:WU00:FS00:Upload 89.46%
12:15:14:WU00:FS00:Upload 93.73%
12:15:44:WU00:FS00:Upload 97.99%
12:16:05:WU00:FS00:Upload 100.00%
12:19:54:WU00:FS00:Upload complete
12:19:54:WU00:FS00:Server responded WORK_QUIT (404)
12:19:54:WARNING:WU00:FS00:Server did not like results, dumping
12:19:54:WU00:FS00:Cleaning up

Project: 9018 (Run 446, Clone 6, Gen 82)

Code: Select all

11:52:28:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9018 run:446 clone:6 gen:82 core:0xa4 unit:0x0000006cab40417c55b2d381ef35706a
11:52:28:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
11:52:28:WU00:FS00:Connecting to 171.64.65.124:8080
11:52:38:WU00:FS00:Upload 4.26%
11:53:52:WU00:FS00:Upload 8.52%
11:54:15:WU00:FS00:Upload 12.78%
11:54:47:WU00:FS00:Upload 17.04%
11:55:18:WU00:FS00:Upload 21.30%
11:56:18:WU00:FS00:Upload 25.56%
11:57:09:WU00:FS00:Upload 29.81%
11:57:32:WU00:FS00:Upload 34.07%
11:58:15:WU00:FS00:Upload 38.33%
11:58:58:WU00:FS00:Upload 42.59%
11:59:38:WU00:FS00:Upload 46.85%
12:00:10:WU00:FS00:Upload 51.11%
12:00:33:WU00:FS00:Upload 55.37%
12:01:57:WU00:FS00:Upload 59.63%
12:02:25:WU00:FS00:Upload 63.89%
12:03:02:WU00:FS00:Upload 68.15%
12:03:54:WU00:FS00:Upload 72.41%
12:04:20:WU00:FS00:Upload 76.67%
12:05:03:WU00:FS00:Upload 80.92%
12:05:37:WU00:FS00:Upload 85.18%
12:06:04:WU00:FS00:Upload 89.44%
12:07:05:WU00:FS00:Upload 93.70%
12:07:30:WU00:FS00:Upload 97.96%
12:07:51:WU00:FS00:Upload 100.00%
12:12:42:WU00:FS00:Upload complete
12:12:42:WU00:FS00:Server responded WORK_QUIT (404)
12:12:42:WARNING:WU00:FS00:Server did not like results, dumping
12:12:42:WU00:FS00:Cleaning up

[Joe_H]

I have let sryckbos know you have seen additional WU's dumped.

[Ricky]

I didn't make a record of all the WUs that were dumped in night, but here is a recent one:

Code: Select all

16:51:40:WU00:FS00:Download complete
16:51:40:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9017 run:40 clone:2 gen:148 core:0xa4 unit:0x000000b4ab40417c55b26da8295f1e1a
16:51:40:WU00:FS00:Starting
16:51:40:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 3140 -checkpoint 15 -np 11
16:51:40:WU00:FS00:Started FahCore on PID 1072
16:51:40:WU00:FS00:Core PID:1824
16:51:40:WU00:FS00:FahCore 0xa4 started
16:51:41:WU00:FS00:0xa4:
16:51:41:WU00:FS00:0xa4:*------------------------------*
16:51:41:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
16:51:41:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
16:51:41:WU00:FS00:0xa4:
16:51:41:WU00:FS00:0xa4:Preparing to commence simulation
16:51:41:WU00:FS00:0xa4:- Looking at optimizations...
16:51:41:WU00:FS00:0xa4:- Created dyn
16:51:41:WU00:FS00:0xa4:- Files status OK
16:51:41:WU00:FS00:0xa4:- Expanded 824972 -> 1397756 (decompressed 169.4 percent)
16:51:41:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=824972 data_size=1397756, decompressed_data_size=1397756 diff=0
16:51:41:WU00:FS00:0xa4:- Digital signature verified
16:51:41:WU00:FS00:0xa4:
16:51:41:WU00:FS00:0xa4:Project: 9017 (Run 40, Clone 2, Gen 148)
16:51:41:WU00:FS00:0xa4:
16:51:41:WU00:FS00:0xa4:Assembly optimizations on if available.
16:51:41:WU00:FS00:0xa4:Entering M.D.
16:51:46:WU00:FS00:0xa4:Mapping NT from 11 to 10 
16:51:47:WU00:FS00:0xa4:Completed 0 out of 250000 steps  (0%)
16:52:40:WU00:FS00:0xa4:Completed 2500 out of 250000 steps  (1%)
16:53:34:WU00:FS00:0xa4:Completed 5000 out of 250000 steps  (2%)
16:54:28:WU00:FS00:0xa4:Completed 7500 out of 250000 steps  (3%)
16:55:22:WU00:FS00:0xa4:Completed 10000 out of 250000 steps  (4%)
16:56:17:WU00:FS00:0xa4:Completed 12500 out of 250000 steps  (5%)
16:57:11:WU00:FS00:0xa4:Completed 15000 out of 250000 steps  (6%)
16:58:05:WU00:FS00:0xa4:Completed 17500 out of 250000 steps  (7%)
16:58:59:WU00:FS00:0xa4:Completed 20000 out of 250000 steps  (8%)
16:59:53:WU00:FS00:0xa4:Completed 22500 out of 250000 steps  (9%)
17:00:47:WU00:FS00:0xa4:Completed 25000 out of 250000 steps  (10%)
17:01:41:WU00:FS00:0xa4:Completed 27500 out of 250000 steps  (11%)
17:02:36:WU00:FS00:0xa4:Completed 30000 out of 250000 steps  (12%)
17:03:30:WU00:FS00:0xa4:Completed 32500 out of 250000 steps  (13%)
17:04:24:WU00:FS00:0xa4:Completed 35000 out of 250000 steps  (14%)
17:05:18:WU00:FS00:0xa4:Completed 37500 out of 250000 steps  (15%)
17:06:13:WU00:FS00:0xa4:Completed 40000 out of 250000 steps  (16%)
17:07:07:WU00:FS00:0xa4:Completed 42500 out of 250000 steps  (17%)
17:08:02:WU00:FS00:0xa4:Completed 45000 out of 250000 steps  (18%)
17:08:56:WU00:FS00:0xa4:Completed 47500 out of 250000 steps  (19%)
17:09:51:WU00:FS00:0xa4:Completed 50000 out of 250000 steps  (20%)
17:10:45:WU00:FS00:0xa4:Completed 52500 out of 250000 steps  (21%)
17:11:39:WU00:FS00:0xa4:Completed 55000 out of 250000 steps  (22%)
17:12:33:WU00:FS00:0xa4:Completed 57500 out of 250000 steps  (23%)
17:13:27:WU00:FS00:0xa4:Completed 60000 out of 250000 steps  (24%)
17:14:21:WU00:FS00:0xa4:Completed 62500 out of 250000 steps  (25%)
17:15:15:WU00:FS00:0xa4:Completed 65000 out of 250000 steps  (26%)
17:16:10:WU00:FS00:0xa4:Completed 67500 out of 250000 steps  (27%)
17:17:04:WU00:FS00:0xa4:Completed 70000 out of 250000 steps  (28%)
17:17:58:WU00:FS00:0xa4:Completed 72500 out of 250000 steps  (29%)
17:18:52:WU00:FS00:0xa4:Completed 75000 out of 250000 steps  (30%)
17:19:46:WU00:FS00:0xa4:Completed 77500 out of 250000 steps  (31%)
17:20:41:WU00:FS00:0xa4:Completed 80000 out of 250000 steps  (32%)
17:21:35:WU00:FS00:0xa4:Completed 82500 out of 250000 steps  (33%)
17:22:29:WU00:FS00:0xa4:Completed 85000 out of 250000 steps  (34%)
17:23:24:WU00:FS00:0xa4:Completed 87500 out of 250000 steps  (35%)
17:24:18:WU00:FS00:0xa4:Completed 90000 out of 250000 steps  (36%)
17:25:12:WU00:FS00:0xa4:Completed 92500 out of 250000 steps  (37%)
17:26:07:WU00:FS00:0xa4:Completed 95000 out of 250000 steps  (38%)
17:27:01:WU00:FS00:0xa4:Completed 97500 out of 250000 steps  (39%)
17:27:55:WU00:FS00:0xa4:Completed 100000 out of 250000 steps  (40%)
17:28:50:WU00:FS00:0xa4:Completed 102500 out of 250000 steps  (41%)
17:29:44:WU00:FS00:0xa4:Completed 105000 out of 250000 steps  (42%)
17:30:38:WU00:FS00:0xa4:Completed 107500 out of 250000 steps  (43%)
17:31:33:WU00:FS00:0xa4:Completed 110000 out of 250000 steps  (44%)
17:32:27:WU00:FS00:0xa4:Completed 112500 out of 250000 steps  (45%)
17:33:21:WU00:FS00:0xa4:Completed 115000 out of 250000 steps  (46%)
17:34:15:WU00:FS00:0xa4:Completed 117500 out of 250000 steps  (47%)
17:35:10:WU00:FS00:0xa4:Completed 120000 out of 250000 steps  (48%)
17:36:04:WU00:FS00:0xa4:Completed 122500 out of 250000 steps  (49%)
17:36:58:WU00:FS00:0xa4:Completed 125000 out of 250000 steps  (50%)
17:37:52:WU00:FS00:0xa4:Completed 127500 out of 250000 steps  (51%)
17:38:46:WU00:FS00:0xa4:Completed 130000 out of 250000 steps  (52%)
17:39:40:WU00:FS00:0xa4:Completed 132500 out of 250000 steps  (53%)
17:40:34:WU00:FS00:0xa4:Completed 135000 out of 250000 steps  (54%)
17:41:28:WU00:FS00:0xa4:Completed 137500 out of 250000 steps  (55%)
17:42:22:WU00:FS00:0xa4:Completed 140000 out of 250000 steps  (56%)
17:43:16:WU00:FS00:0xa4:Completed 142500 out of 250000 steps  (57%)
17:44:10:WU00:FS00:0xa4:Completed 145000 out of 250000 steps  (58%)
17:45:04:WU00:FS00:0xa4:Completed 147500 out of 250000 steps  (59%)
17:45:59:WU00:FS00:0xa4:Completed 150000 out of 250000 steps  (60%)
17:46:53:WU00:FS00:0xa4:Completed 152500 out of 250000 steps  (61%)
17:47:47:WU00:FS00:0xa4:Completed 155000 out of 250000 steps  (62%)
17:48:41:WU00:FS00:0xa4:Completed 157500 out of 250000 steps  (63%)
17:49:35:WU00:FS00:0xa4:Completed 160000 out of 250000 steps  (64%)
17:50:30:WU00:FS00:0xa4:Completed 162500 out of 250000 steps  (65%)
17:51:24:WU00:FS00:0xa4:Completed 165000 out of 250000 steps  (66%)
17:52:19:WU00:FS00:0xa4:Completed 167500 out of 250000 steps  (67%)
17:53:13:WU00:FS00:0xa4:Completed 170000 out of 250000 steps  (68%)
17:54:07:WU00:FS00:0xa4:Completed 172500 out of 250000 steps  (69%)
17:55:02:WU00:FS00:0xa4:Completed 175000 out of 250000 steps  (70%)
17:55:56:WU00:FS00:0xa4:Completed 177500 out of 250000 steps  (71%)
17:56:50:WU00:FS00:0xa4:Completed 180000 out of 250000 steps  (72%)
17:57:45:WU00:FS00:0xa4:Completed 182500 out of 250000 steps  (73%)
17:58:39:WU00:FS00:0xa4:Completed 185000 out of 250000 steps  (74%)
17:59:33:WU00:FS00:0xa4:Completed 187500 out of 250000 steps  (75%)
18:00:28:WU00:FS00:0xa4:Completed 190000 out of 250000 steps  (76%)
18:01:22:WU00:FS00:0xa4:Completed 192500 out of 250000 steps  (77%)
18:02:16:WU00:FS00:0xa4:Completed 195000 out of 250000 steps  (78%)
18:03:10:WU00:FS00:0xa4:Completed 197500 out of 250000 steps  (79%)
18:04:04:WU00:FS00:0xa4:Completed 200000 out of 250000 steps  (80%)
18:04:59:WU00:FS00:0xa4:Completed 202500 out of 250000 steps  (81%)
18:05:53:WU00:FS00:0xa4:Completed 205000 out of 250000 steps  (82%)
18:06:47:WU00:FS00:0xa4:Completed 207500 out of 250000 steps  (83%)
18:07:41:WU00:FS00:0xa4:Completed 210000 out of 250000 steps  (84%)
18:08:36:WU00:FS00:0xa4:Completed 212500 out of 250000 steps  (85%)
18:09:30:WU00:FS00:0xa4:Completed 215000 out of 250000 steps  (86%)
18:10:24:WU00:FS00:0xa4:Completed 217500 out of 250000 steps  (87%)
18:11:18:WU00:FS00:0xa4:Completed 220000 out of 250000 steps  (88%)
18:12:12:WU00:FS00:0xa4:Completed 222500 out of 250000 steps  (89%)
18:13:07:WU00:FS00:0xa4:Completed 225000 out of 250000 steps  (90%)
18:14:01:WU00:FS00:0xa4:Completed 227500 out of 250000 steps  (91%)
18:14:55:WU00:FS00:0xa4:Completed 230000 out of 250000 steps  (92%)
18:15:50:WU00:FS00:0xa4:Completed 232500 out of 250000 steps  (93%)
18:16:44:WU00:FS00:0xa4:Completed 235000 out of 250000 steps  (94%)
18:17:38:WU00:FS00:0xa4:Completed 237500 out of 250000 steps  (95%)
18:18:32:WU00:FS00:0xa4:Completed 240000 out of 250000 steps  (96%)
18:19:26:WU00:FS00:0xa4:Completed 242500 out of 250000 steps  (97%)
18:20:20:WU00:FS00:0xa4:Completed 245000 out of 250000 steps  (98%)
18:21:14:WU00:FS00:0xa4:Completed 247500 out of 250000 steps  (99%)
18:22:09:WU00:FS00:0xa4:Completed 250000 out of 250000 steps  (100%)
18:22:09:WU00:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
18:22:09:WU01:FS00:Connecting to 171.67.108.45:8080
18:22:09:WU01:FS00:Assigned to work server 171.64.65.124
18:22:09:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:11 from 171.64.65.124
18:22:09:WU01:FS00:Connecting to 171.64.65.124:8080
18:22:19:WU00:FS00:0xa4:
18:22:19:WU00:FS00:0xa4:Finished Work Unit:
18:22:19:WU00:FS00:0xa4:- Reading up to 811512 from "00/wudata_01.trr": Read 811512
18:22:19:WU00:FS00:0xa4:trr file hash check passed.
18:22:19:WU00:FS00:0xa4:- Reading up to 746120 from "00/wudata_01.xtc": Read 746120
18:22:19:WU00:FS00:0xa4:xtc file hash check passed.
18:22:19:WU00:FS00:0xa4:edr file hash check passed.
18:22:19:WU00:FS00:0xa4:logfile size: 22842
18:22:19:WU00:FS00:0xa4:Leaving Run
18:22:21:WU00:FS00:0xa4:- Writing 1582962 bytes of core data to disk...
18:22:21:WU00:FS00:0xa4:Done: 1582450 -> 1537778 (compressed to 97.1 percent)
18:22:21:WU00:FS00:0xa4:  ... Done.
18:22:22:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:22:22:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9017 run:40 clone:2 gen:148 core:0xa4 unit:0x000000b4ab40417c55b26da8295f1e1a
18:22:22:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
18:22:22:WU00:FS00:Connecting to 171.64.65.124:8080
18:22:30:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
18:22:30:WU01:FS00:Connecting to 171.64.65.124:80
18:22:43:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
18:22:43:WU00:FS00:Connecting to 171.64.65.124:80
18:22:43:WU00:FS00:Upload 4.26%
18:23:24:WU01:FS00:Downloading 805.95KiB
18:23:41:WU01:FS00:Download 7.94%
18:24:03:WU01:FS00:Download 15.88%
18:24:41:WU01:FS00:Download 23.82%
18:24:45:WU00:FS00:Upload 12.78%
18:24:58:WU01:FS00:Download 31.76%
18:25:08:WU01:FS00:Download 47.65%
18:25:14:WU01:FS00:Download 63.53%
18:25:14:WU00:FS00:Upload 17.04%
18:25:24:WU01:FS00:Download 79.41%
18:25:36:WU01:FS00:Download 87.35%
18:25:44:WU01:FS00:Download 95.29%
18:25:48:WU01:FS00:Download complete
18:25:48:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9017 run:326 clone:6 gen:7 core:0xa4 unit:0x00000012ab40417c55b2c9afa141f099
18:25:48:WU01:FS00:Starting
18:25:48:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 01 -suffix 01 -version 704 -lifeline 3140 -checkpoint 15 -np 11
18:25:48:WU01:FS00:Started FahCore on PID 240
18:25:48:WU01:FS00:Core PID:4884
18:25:48:WU01:FS00:FahCore 0xa4 started
18:25:48:WU01:FS00:0xa4:
18:25:48:WU01:FS00:0xa4:*------------------------------*
18:25:48:WU01:FS00:0xa4:Folding@Home Gromacs GB Core
18:25:48:WU01:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
18:25:48:WU01:FS00:0xa4:
18:25:48:WU01:FS00:0xa4:Preparing to commence simulation
18:25:48:WU01:FS00:0xa4:- Looking at optimizations...
18:25:48:WU01:FS00:0xa4:- Created dyn
18:25:48:WU01:FS00:0xa4:- Files status OK
18:25:48:WU01:FS00:0xa4:- Expanded 824780 -> 1397756 (decompressed 169.4 percent)
18:25:48:WU01:FS00:0xa4:Called DecompressByteArray: compressed_data_size=824780 data_size=1397756, decompressed_data_size=1397756 diff=0
18:25:48:WU01:FS00:0xa4:- Digital signature verified
18:25:48:WU01:FS00:0xa4:
18:25:48:WU01:FS00:0xa4:Project: 9017 (Run 326, Clone 6, Gen 7)
18:25:48:WU01:FS00:0xa4:
18:25:48:WU01:FS00:0xa4:Assembly optimizations on if available.
18:25:48:WU01:FS00:0xa4:Entering M.D.
18:25:54:WU01:FS00:0xa4:Mapping NT from 11 to 10 
18:25:54:WU00:FS00:Upload 21.30%
18:25:54:WU01:FS00:0xa4:Completed 0 out of 250000 steps  (0%)
18:26:19:WU00:FS00:Upload 25.56%
18:26:48:WU01:FS00:0xa4:Completed 2500 out of 250000 steps  (1%)
18:27:38:WU00:FS00:Upload 29.82%
18:27:42:WU01:FS00:0xa4:Completed 5000 out of 250000 steps  (2%)
18:27:45:WU00:FS00:Upload 34.08%
18:28:36:WU01:FS00:0xa4:Completed 7500 out of 250000 steps  (3%)
18:28:58:WU00:FS00:Upload 38.34%
18:29:24:WU00:FS00:Upload 42.60%
18:29:30:WU01:FS00:0xa4:Completed 10000 out of 250000 steps  (4%)
18:29:36:WU00:FS00:Upload 46.86%
18:29:52:WU00:FS00:Upload 51.12%
18:30:24:WU01:FS00:0xa4:Completed 12500 out of 250000 steps  (5%)
18:31:08:WU00:FS00:Upload 55.38%
18:31:18:WU01:FS00:0xa4:Completed 15000 out of 250000 steps  (6%)
18:31:23:WU00:FS00:Upload 59.64%
18:31:48:WU00:FS00:Upload 63.90%
18:32:11:WU01:FS00:0xa4:Completed 17500 out of 250000 steps  (7%)
18:32:43:WU00:FS00:Upload 68.17%
18:33:05:WU01:FS00:0xa4:Completed 20000 out of 250000 steps  (8%)
18:33:12:WU00:FS00:Upload 72.43%
18:33:32:WU00:FS00:Upload 76.69%
18:33:49:WU00:FS00:Upload 80.95%
18:33:59:WU01:FS00:0xa4:Completed 22500 out of 250000 steps  (9%)
18:33:59:WU00:FS00:Upload 85.21%
18:34:39:WU00:FS00:Upload 89.47%
18:34:53:WU01:FS00:0xa4:Completed 25000 out of 250000 steps  (10%)
18:34:57:WU00:FS00:Upload 93.73%
18:35:47:WU01:FS00:0xa4:Completed 27500 out of 250000 steps  (11%)
18:36:21:WU00:FS00:Upload 97.99%
18:36:41:WU01:FS00:0xa4:Completed 30000 out of 250000 steps  (12%)
18:36:50:WU00:FS00:Upload 100.00%
18:37:35:WU01:FS00:0xa4:Completed 32500 out of 250000 steps  (13%)
18:38:24:WU00:FS00:Upload complete
18:38:24:WU00:FS00:Server responded WORK_ACK (400)
18:38:24:WU00:FS00:Final credit estimate, 1896.00 points
18:38:24:WU00:FS00:Cleaning up
18:38:29:WU01:FS00:0xa4:Completed 35000 out of 250000 steps  (14%)
18:39:23:WU01:FS00:0xa4:Completed 37500 out of 250000 steps  (15%)
18:40:17:WU01:FS00:0xa4:Completed 40000 out of 250000 steps  (16%)
18:41:11:WU01:FS00:0xa4:Completed 42500 out of 250000 steps  (17%)
18:42:05:WU01:FS00:0xa4:Completed 45000 out of 250000 steps  (18%)
18:42:59:WU01:FS00:0xa4:Completed 47500 out of 250000 steps  (19%)
18:43:53:WU01:FS00:0xa4:Completed 50000 out of 250000 steps  (20%)
18:44:47:WU01:FS00:0xa4:Completed 52500 out of 250000 steps  (21%)
18:45:41:WU01:FS00:0xa4:Completed 55000 out of 250000 steps  (22%)
18:46:35:WU01:FS00:0xa4:Completed 57500 out of 250000 steps  (23%)
18:47:29:WU01:FS00:0xa4:Completed 60000 out of 250000 steps  (24%)
18:48:23:WU01:FS00:0xa4:Completed 62500 out of 250000 steps  (25%)
18:49:17:WU01:FS00:0xa4:Completed 65000 out of 250000 steps  (26%)
18:50:11:WU01:FS00:0xa4:Completed 67500 out of 250000 steps  (27%)
18:51:05:WU01:FS00:0xa4:Completed 70000 out of 250000 steps  (28%)
18:51:59:WU01:FS00:0xa4:Completed 72500 out of 250000 steps  (29%)
18:52:53:WU01:FS00:0xa4:Completed 75000 out of 250000 steps  (30%)
18:53:47:WU01:FS00:0xa4:Completed 77500 out of 250000 steps  (31%)
18:54:41:WU01:FS00:0xa4:Completed 80000 out of 250000 steps  (32%)
18:55:35:WU01:FS00:0xa4:Completed 82500 out of 250000 steps  (33%)
18:56:29:WU01:FS00:0xa4:Completed 85000 out of 250000 steps  (34%)
18:57:23:WU01:FS00:0xa4:Completed 87500 out of 250000 steps  (35%)
18:58:17:WU01:FS00:0xa4:Completed 90000 out of 250000 steps  (36%)
18:59:11:WU01:FS00:0xa4:Completed 92500 out of 250000 steps  (37%)
19:00:05:WU01:FS00:0xa4:Completed 95000 out of 250000 steps  (38%)
19:00:59:WU01:FS00:0xa4:Completed 97500 out of 250000 steps  (39%)
19:01:53:WU01:FS00:0xa4:Completed 100000 out of 250000 steps  (40%)
19:02:47:WU01:FS00:0xa4:Completed 102500 out of 250000 steps  (41%)
19:03:41:WU01:FS00:0xa4:Completed 105000 out of 250000 steps  (42%)
19:04:35:WU01:FS00:0xa4:Completed 107500 out of 250000 steps  (43%)
19:05:29:WU01:FS00:0xa4:Completed 110000 out of 250000 steps  (44%)
19:06:23:WU01:FS00:0xa4:Completed 112500 out of 250000 steps  (45%)
19:07:17:WU01:FS00:0xa4:Completed 115000 out of 250000 steps  (46%)
19:08:11:WU01:FS00:0xa4:Completed 117500 out of 250000 steps  (47%)
19:09:05:WU01:FS00:0xa4:Completed 120000 out of 250000 steps  (48%)
19:09:59:WU01:FS00:0xa4:Completed 122500 out of 250000 steps  (49%)
19:10:53:WU01:FS00:0xa4:Completed 125000 out of 250000 steps  (50%)
19:11:47:WU01:FS00:0xa4:Completed 127500 out of 250000 steps  (51%)
19:12:42:WU01:FS00:0xa4:Completed 130000 out of 250000 steps  (52%)
19:13:36:WU01:FS00:0xa4:Completed 132500 out of 250000 steps  (53%)
19:14:29:WU01:FS00:0xa4:Completed 135000 out of 250000 steps  (54%)
19:15:23:WU01:FS00:0xa4:Completed 137500 out of 250000 steps  (55%)
19:16:18:WU01:FS00:0xa4:Completed 140000 out of 250000 steps  (56%)
19:17:11:WU01:FS00:0xa4:Completed 142500 out of 250000 steps  (57%)
19:18:06:WU01:FS00:0xa4:Completed 145000 out of 250000 steps  (58%)
19:19:00:WU01:FS00:0xa4:Completed 147500 out of 250000 steps  (59%)
19:19:55:WU01:FS00:0xa4:Completed 150000 out of 250000 steps  (60%)
19:20:49:WU01:FS00:0xa4:Completed 152500 out of 250000 steps  (61%)
19:21:43:WU01:FS00:0xa4:Completed 155000 out of 250000 steps  (62%)
19:22:37:WU01:FS00:0xa4:Completed 157500 out of 250000 steps  (63%)
19:23:31:WU01:FS00:0xa4:Completed 160000 out of 250000 steps  (64%)
19:24:25:WU01:FS00:0xa4:Completed 162500 out of 250000 steps  (65%)
19:25:19:WU01:FS00:0xa4:Completed 165000 out of 250000 steps  (66%)
19:26:13:WU01:FS00:0xa4:Completed 167500 out of 250000 steps  (67%)
19:27:06:WU01:FS00:0xa4:Completed 170000 out of 250000 steps  (68%)
19:28:00:WU01:FS00:0xa4:Completed 172500 out of 250000 steps  (69%)
19:28:54:WU01:FS00:0xa4:Completed 175000 out of 250000 steps  (70%)
19:29:48:WU01:FS00:0xa4:Completed 177500 out of 250000 steps  (71%)
19:30:41:WU01:FS00:0xa4:Completed 180000 out of 250000 steps  (72%)
19:31:35:WU01:FS00:0xa4:Completed 182500 out of 250000 steps  (73%)
19:32:29:WU01:FS00:0xa4:Completed 185000 out of 250000 steps  (74%)
19:33:24:WU01:FS00:0xa4:Completed 187500 out of 250000 steps  (75%)
19:34:18:WU01:FS00:0xa4:Completed 190000 out of 250000 steps  (76%)
19:35:11:WU01:FS00:0xa4:Completed 192500 out of 250000 steps  (77%)
19:36:05:WU01:FS00:0xa4:Completed 195000 out of 250000 steps  (78%)
******************************* Date: 2016-01-07 *******************************
19:36:59:WU01:FS00:0xa4:Completed 197500 out of 250000 steps  (79%)
19:37:54:WU01:FS00:0xa4:Completed 200000 out of 250000 steps  (80%)
19:38:47:WU01:FS00:0xa4:Completed 202500 out of 250000 steps  (81%)
19:39:41:WU01:FS00:0xa4:Completed 205000 out of 250000 steps  (82%)
19:40:35:WU01:FS00:0xa4:Completed 207500 out of 250000 steps  (83%)
19:41:30:WU01:FS00:0xa4:Completed 210000 out of 250000 steps  (84%)
19:42:23:WU01:FS00:0xa4:Completed 212500 out of 250000 steps  (85%)
19:43:17:WU01:FS00:0xa4:Completed 215000 out of 250000 steps  (86%)
19:44:12:WU01:FS00:0xa4:Completed 217500 out of 250000 steps  (87%)
19:45:06:WU01:FS00:0xa4:Completed 220000 out of 250000 steps  (88%)
19:46:00:WU01:FS00:0xa4:Completed 222500 out of 250000 steps  (89%)
19:46:53:WU01:FS00:0xa4:Completed 225000 out of 250000 steps  (90%)
19:47:47:WU01:FS00:0xa4:Completed 227500 out of 250000 steps  (91%)
19:48:41:WU01:FS00:0xa4:Completed 230000 out of 250000 steps  (92%)
19:49:35:WU01:FS00:0xa4:Completed 232500 out of 250000 steps  (93%)
19:50:29:WU01:FS00:0xa4:Completed 235000 out of 250000 steps  (94%)
19:51:23:WU01:FS00:0xa4:Completed 237500 out of 250000 steps  (95%)
19:52:17:WU01:FS00:0xa4:Completed 240000 out of 250000 steps  (96%)
19:53:11:WU01:FS00:0xa4:Completed 242500 out of 250000 steps  (97%)
19:54:05:WU01:FS00:0xa4:Completed 245000 out of 250000 steps  (98%)
19:54:59:WU01:FS00:0xa4:Completed 247500 out of 250000 steps  (99%)
19:55:53:WU01:FS00:0xa4:Completed 250000 out of 250000 steps  (100%)
19:55:53:WU01:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
19:55:54:WU02:FS00:Connecting to 171.67.108.45:8080
19:55:55:WU02:FS00:Assigned to work server 171.64.65.124
19:55:55:WU02:FS00:Requesting new work unit for slot 00: RUNNING cpu:11 from 171.64.65.124
19:55:55:WU02:FS00:Connecting to 171.64.65.124:8080
19:56:03:WU01:FS00:0xa4:
19:56:03:WU01:FS00:0xa4:Finished Work Unit:
19:56:03:WU01:FS00:0xa4:- Reading up to 811512 from "01/wudata_01.trr": Read 811512
19:56:03:WU01:FS00:0xa4:trr file hash check passed.
19:56:03:WU01:FS00:0xa4:- Reading up to 746072 from "01/wudata_01.xtc": Read 746072
19:56:03:WU01:FS00:0xa4:xtc file hash check passed.
19:56:03:WU01:FS00:0xa4:edr file hash check passed.
19:56:03:WU01:FS00:0xa4:logfile size: 22762
19:56:03:WU01:FS00:0xa4:Leaving Run
19:56:04:WU01:FS00:0xa4:- Writing 1582834 bytes of core data to disk...
19:56:04:WU01:FS00:0xa4:Done: 1582322 -> 1537682 (compressed to 97.1 percent)
19:56:04:WU01:FS00:0xa4:  ... Done.
19:56:04:WU01:FS00:0xa4:- Shutting down core
19:56:04:WU01:FS00:0xa4:
19:56:04:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
19:56:04:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
19:56:04:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9017 run:326 clone:6 gen:7 core:0xa4 unit:0x00000012ab40417c55b2c9afa141f099
19:56:04:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
19:56:04:WU01:FS00:Connecting to 171.64.65.124:8080
19:56:16:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
19:56:16:WU02:FS00:Connecting to 171.64.65.124:80
19:56:37:ERROR:WU02:FS00:Exception: Failed to connect to 171.64.65.124:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
19:56:37:WU02:FS00:Connecting to 171.67.108.45:8080
19:56:37:WU02:FS00:Assigned to work server 155.247.166.219
19:56:37:WU02:FS00:Requesting new work unit for slot 00: READY cpu:11 from 155.247.166.219
19:56:37:WU02:FS00:Connecting to 155.247.166.219:8080
19:56:38:WU02:FS00:Downloading 116.32KiB
19:56:38:WU02:FS00:Download complete
19:56:38:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:6397 run:46 clone:31 gen:335 core:0xa4 unit:0x0000016d0002894b5462c9b9f1a6a037
19:56:38:WU02:FS00:Starting
19:56:38:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/win2/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 3140 -checkpoint 15 -np 11
19:56:38:WU02:FS00:Started FahCore on PID 3552
19:56:38:WU02:FS00:Core PID:3044
19:56:38:WU02:FS00:FahCore 0xa4 started
19:56:39:WU02:FS00:0xa4:
19:56:39:WU02:FS00:0xa4:*------------------------------*
19:56:39:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
19:56:39:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
19:56:39:WU02:FS00:0xa4:
19:56:39:WU02:FS00:0xa4:Preparing to commence simulation
19:56:39:WU02:FS00:0xa4:- Looking at optimizations...
19:56:39:WU02:FS00:0xa4:- Created dyn
19:56:39:WU02:FS00:0xa4:- Files status OK
19:56:39:WU02:FS00:0xa4:- Expanded 118595 -> 268860 (decompressed 226.7 percent)
19:56:39:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=118595 data_size=268860, decompressed_data_size=268860 diff=0
19:56:39:WU02:FS00:0xa4:- Digital signature verified
19:56:39:WU02:FS00:0xa4:
19:56:39:WU02:FS00:0xa4:Project: 6397 (Run 46, Clone 31, Gen 335)
19:56:39:WU02:FS00:0xa4:
19:56:39:WU02:FS00:0xa4:Assembly optimizations on if available.
19:56:39:WU02:FS00:0xa4:Entering M.D.
19:56:44:WU02:FS00:0xa4:Mapping NT from 11 to 10 
19:56:44:WU02:FS00:0xa4:Completed 0 out of 5000000 steps  (0%)
19:58:04:WU02:FS00:0xa4:Completed 50000 out of 5000000 steps  (1%)
19:58:14:WU01:FS00:Upload 12.78%
19:58:39:WU01:FS00:Upload 17.04%
19:59:05:WU01:FS00:Upload 21.30%
19:59:24:WU02:FS00:0xa4:Completed 100000 out of 5000000 steps  (2%)
20:00:32:WU01:FS00:Upload 25.56%
20:00:44:WU02:FS00:0xa4:Completed 150000 out of 5000000 steps  (3%)
20:00:55:WU01:FS00:Upload 29.82%
20:01:33:WU01:FS00:Upload 34.08%
20:02:04:WU02:FS00:0xa4:Completed 200000 out of 5000000 steps  (4%)
20:02:39:WU01:FS00:Upload 38.35%
20:03:16:WU01:FS00:Upload 42.61%
20:03:24:WU02:FS00:0xa4:Completed 250000 out of 5000000 steps  (5%)
20:04:25:WU01:FS00:Upload 46.87%
20:04:44:WU02:FS00:0xa4:Completed 300000 out of 5000000 steps  (6%)
20:04:56:WU01:FS00:Upload 51.13%
20:05:26:WU01:FS00:Upload 55.39%
20:06:04:WU02:FS00:0xa4:Completed 350000 out of 5000000 steps  (7%)
20:06:46:WU01:FS00:Upload 59.65%
20:07:24:WU02:FS00:0xa4:Completed 400000 out of 5000000 steps  (8%)
20:07:33:WU01:FS00:Upload 63.91%
20:08:44:WU02:FS00:0xa4:Completed 450000 out of 5000000 steps  (9%)
20:09:02:WU01:FS00:Upload 68.17%
20:09:41:WU01:FS00:Upload 72.43%
20:10:04:WU02:FS00:0xa4:Completed 500000 out of 5000000 steps  (10%)
20:10:33:WU01:FS00:Upload 76.69%
20:11:25:WU02:FS00:0xa4:Completed 550000 out of 5000000 steps  (11%)
20:11:42:WU01:FS00:Upload 80.95%
20:12:23:WU01:FS00:Upload 85.21%
20:12:45:WU02:FS00:0xa4:Completed 600000 out of 5000000 steps  (12%)
20:13:20:WU01:FS00:Upload 89.47%
20:13:48:WU01:FS00:Upload 93.73%
20:14:03:WU01:FS00:Upload 97.99%
20:14:05:WU02:FS00:0xa4:Completed 650000 out of 5000000 steps  (13%)
20:14:17:WU01:FS00:Upload 100.00%
20:15:25:WU02:FS00:0xa4:Completed 700000 out of 5000000 steps  (14%)
20:16:45:WU02:FS00:0xa4:Completed 750000 out of 5000000 steps  (15%)
20:17:41:WU01:FS00:Upload complete
20:17:41:WU01:FS00:Server responded WORK_QUIT (404)
20:17:41:WARNING:WU01:FS00:Server did not like results, dumping
20:17:41:WU01:FS00:Cleaning up
20:18:05:WU02:FS00:0xa4:Completed 800000 out of 5000000 steps  (16%)

[parkut]

Another WU dumped in the last 90 minutes

Project: 9017 (Run 501, Clone 5, Gen 80)

Code: Select all

18:54:39:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9017 run:501 clone:5 gen:80 core:0xa4 unit:0x00000067ab40417c55b2cb00252ebcc3
18:54:39:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
18:54:39:WU01:FS00:Connecting to 171.64.65.124:8080
18:57:34:WU01:FS00:Upload 4.26%
18:58:16:WU01:FS00:Upload 8.52%
18:59:31:WU01:FS00:Upload 12.78%
18:59:50:WU01:FS00:Upload 17.04%
19:00:24:WU01:FS00:Upload 21.30%
19:01:48:WU01:FS00:Upload 25.56%
19:02:17:WU01:FS00:Upload 29.82%
19:03:21:WU01:FS00:Upload 34.08%
19:04:09:WU01:FS00:Upload 38.34%
19:04:38:WU01:FS00:Upload 42.60%
19:05:53:WU01:FS00:Upload 46.86%
19:06:16:WU01:FS00:Upload 51.12%
19:06:36:WU01:FS00:Upload 55.38%
19:07:03:WU01:FS00:Upload 59.64%
19:08:19:WU01:FS00:Upload 63.91%
19:08:35:WU01:FS00:Upload 68.17%
19:08:51:WU01:FS00:Upload 72.43%
19:09:25:WU01:FS00:Upload 76.69%
19:10:46:WU01:FS00:Upload 80.95%
19:11:07:WU01:FS00:Upload 85.21%
19:11:21:WU01:FS00:Upload 89.47%
19:11:48:WU01:FS00:Upload 93.73%
19:12:59:WU01:FS00:Upload 97.99%
19:13:05:WU01:FS00:Upload 100.00%
19:16:17:WU01:FS00:Upload complete
19:16:17:WU01:FS00:Server responded WORK_QUIT (404)
19:16:17:WARNING:WU01:FS00:Server did not like results, dumping
19:16:17:WU01:FS00:Cleaning up

[n1np]

Same story. I just re-started on of the machines here: (yes, I just noticed that the clock is off by one hour)

Code: Select all

*********************** Log Started 2016-01-07T22:14:52Z ***********************
22:14:52:************************* Folding@home Client *************************
22:14:52:    Website: http://folding.stanford.edu/
22:14:52:  Copyright: (c) 2009-2014 Stanford University
22:14:52:     Author: Joseph Coffland <[email protected]>
22:14:52:       Args: --config core744.xml --client-type=advanced --cpus=8
22:14:52:             --command-allow=192.168.0.0/24 --command-port=36330
22:14:52:             --password=********** --next-unit-percentage=100
22:14:52:             --core-priority=low --cpu-affinity=true
22:14:52:     Config: /home/akula/fah/7-4-4/core744.xml
22:14:52:******************************** Build ********************************
22:14:52:    Version: 7.4.4
22:14:52:       Date: Mar 4 2014
22:14:52:       Time: 12:01:17
22:14:52:    SVN Rev: 4130
22:14:52:     Branch: fah/trunk/client
22:14:52:   Compiler: GNU 4.1.2 20080704 (Red Hat 4.1.2-46)
22:14:52:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
22:14:52:             -fno-unsafe-math-optimizations -msse2
22:14:52:   Platform: linux2 2.6.18-164.11.1.el5
22:14:52:       Bits: 64
22:14:52:       Mode: Release
22:14:52:******************************* System ********************************
22:14:52:        CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz
22:14:52:     CPU ID: GenuineIntel Family 6 Model 23 Stepping 6
22:14:52:       CPUs: 8
22:14:52:     Memory: 7.79GiB
22:14:52:Free Memory: 7.59GiB
22:14:52:    Threads: POSIX_THREADS
22:14:52: OS Version: 3.2
22:14:52:Has Battery: false
22:14:52: On Battery: false
22:14:52: UTC Offset: -5
22:14:52:        PID: 1554
22:14:52:        CWD: /home/akula/fah/7-4-4
22:14:52:         OS: Linux 3.2.29area63_core0 x86_64
22:14:52:    OS Arch: AMD64
22:14:52:       GPUs: 1
22:14:52:      GPU 0: UNSUPPORTED: [ES1000]
22:14:52:       CUDA: Not detected
22:14:52:***********************************************************************
22:14:52:<config>
22:14:52:  <!-- User Information -->
22:14:52:  <passkey v='********************************'/>
22:14:52:  <team v='12912'/>
22:14:52:  <user v='n1np'/>
22:14:52:
22:14:52:  <!-- Folding Slots -->
22:14:52:  <slot id='0' type='CPU'/>
22:14:52:</config>
22:14:52:Trying to access database...
22:14:52:Successfully acquired database lock
22:14:52:Enabled folding slot 00: READY cpu:8
22:14:52:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:9026 run:423 clone:10 gen:3 core:0xa4 unit:0x00000005ab40417c55f39e157185807d
22:14:52:WU03:FS00:Uploading 1.47MiB to 171.64.65.124
22:14:52:WU03:FS00:Connecting to 171.64.65.124:8080
22:14:52:WU00:FS00:Connecting to 171.67.108.45:8080
22:14:53:WU00:FS00:Assigned to work server 171.64.65.124
22:14:53:WU00:FS00:Requesting new work unit for slot 00: READY cpu:8 from 171.64.65.124
22:14:53:WU00:FS00:Connecting to 171.64.65.124:8080
22:16:28:WU00:FS00:Downloading 806.92KiB
22:16:32:WU03:FS00:Upload 12.78%
22:16:44:WU00:FS00:Download 7.93%
22:16:52:WU00:FS00:Download 15.86%
22:16:58:WU00:FS00:Download 23.79%
22:17:07:WU00:FS00:Download 31.73%
22:17:35:WU00:FS00:Download 39.66%
22:18:24:WU00:FS00:Download 47.59%
22:18:24:WU03:FS00:Upload 17.04%
22:19:28:WU00:FS00:Download 55.52%
22:19:45:WU00:FS00:Download 63.45%
22:19:45:WU03:FS00:Upload 21.30%
[91m22:19:47:ERROR:WU00:FS00:Exception: Transfer failed[0m
[93m22:19:47:WARNING:WU03:FS00:Exception: Failed to send results to work server: Transfer failed[0m
22:19:47:WU03:FS00:Trying to send results to collection server
22:19:47:WU03:FS00:Uploading 1.47MiB to 171.67.108.46
22:19:47:WU03:FS00:Connecting to 171.67.108.46:8080
22:19:47:WU00:FS00:Connecting to 171.67.108.45:8080
22:19:48:WU00:FS00:Assigned to work server 171.64.65.124
22:19:48:WU00:FS00:Requesting new work unit for slot 00: READY cpu:8 from 171.64.65.124
22:19:48:WU00:FS00:Connecting to 171.64.65.124:8080
[93m22:19:48:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80[0m
22:19:48:WU00:FS00:Connecting to 171.64.65.124:80
[91m22:19:48:ERROR:WU00:FS00:Exception: Failed to connect to 171.64.65.124:80: Connection refused[0m
22:19:53:WU03:FS00:Upload 85.21%
22:19:54:WU03:FS00:Upload complete
22:19:54:WU03:FS00:Server responded WORK_ACK (400)
22:19:54:WU03:FS00:Final credit estimate, 440.00 points
22:19:54:WU03:FS00:Cleaning up
22:20:47:WU00:FS00:Connecting to 171.67.108.45:8080
22:20:48:WU00:FS00:Assigned to work server 171.64.65.124
22:20:48:WU00:FS00:Requesting new work unit for slot 00: READY cpu:8 from 171.64.65.124
22:20:48:WU00:FS00:Connecting to 171.64.65.124:8080
22:20:56:WU00:FS00:Downloading 806.35KiB
22:21:02:WU00:FS00:Download 7.94%
22:21:08:WU00:FS00:Download 31.75%
22:21:14:WU00:FS00:Download 55.56%
22:21:23:WU00:FS00:Download 79.37%
22:21:26:WU00:FS00:Download complete
22:21:26:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9029 run:708 clone:4 gen:135 core:0xa4 unit:0x0000009dab40417c55f38d7ca0db9e2e
22:21:26:WU00:FS00:Starting
22:21:26:WU00:FS00:Running FahCore: /home/akula/fah/7-4-4/FAHCoreWrapper /home/akula/fah/7-4-4/cores/web.stanford.edu/~pande/Linux/AMD64/Core_a4.fah/FahCore_a4 -dir 00 -suffix 01 -version 704 -lifeline 1554 -checkpoint 15 -np 8
22:21:26:WU00:FS00:Started FahCore on PID 1563
22:21:26:WU00:FS00:Core PID:1567
22:21:26:WU00:FS00:FahCore 0xa4 started
22:21:27:WU00:FS00:0xa4:
22:21:27:WU00:FS00:0xa4:*------------------------------*
22:21:27:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
22:21:27:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
22:21:27:WU00:FS00:0xa4:
22:21:27:WU00:FS00:0xa4:Preparing to commence simulation
22:21:27:WU00:FS00:0xa4:- Looking at optimizations...
22:21:27:WU00:FS00:0xa4:- Created dyn
22:21:27:WU00:FS00:0xa4:- Files status OK
22:21:27:WU00:FS00:0xa4:- Expanded 825187 -> 1398040 (decompressed 169.4 percent)
22:21:27:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825187 data_size=1398040, decompressed_data_size=1398040 diff=0
22:21:27:WU00:FS00:0xa4:- Digital signature verified
22:21:27:WU00:FS00:0xa4:
22:21:27:WU00:FS00:0xa4:Project: 9029 (Run 708, Clone 4, Gen 135)
22:21:27:WU00:FS00:0xa4:
22:21:27:WU00:FS00:0xa4:Assembly optimizations on if available.
22:21:27:WU00:FS00:0xa4:Entering M.D.
22:21:34:WU00:FS00:0xa4:Completed 0 out of 250000 steps  (0%)
22:22:53:WU00:FS00:0xa4:Completed 2500 out of 250000 steps  (1%)
22:24:12:WU00:FS00:0xa4:Completed 5000 out of 250000 steps  (2%)
22:25:31:WU00:FS00:0xa4:Completed 7500 out of 250000 steps  (3%)

And it looks like at least 6 other machines here have WU stuck in the queue for this server. Some of those I know uploaded 100% but did not receive a ACK from the server. Some uploaded less than 100% and stuck there with no error. They will all try to re-send if I re-start the clients.

[sryckbos]

Ugh this is a separate problem. But I know how to fix this one. Everything should be okay shortly. I'm on it.

[davidcoton]

Work server is failing to assign WUs:

Code: Select all

23:42:20:WU01:FS00:Connecting to 171.67.108.45:8080
23:42:21:WU01:FS00:Assigned to work server 171.64.65.124
23:42:21:WU01:FS00:Requesting new work unit for slot 00: READY cpu:5 from 171.64.65.124
23:42:21:WU01:FS00:Connecting to 171.64.65.124:8080
23:42:23:ERROR:WU01:FS00:Exception: Server did not assign work unit

[Rinse and Repeat]

[sryckbos]

So I should have fixed both of those problems.

Sorry about all this. This WS is definitely on the old side and I'm in the process of getting everything to a newer one with more storage and everything. That should be happening soon, which should alleviate the stress on this server, which is by far the most active one on FAH right now.

[parkut]

Two more WU's were dumped about 18 hours ago. However the upload speed was very quick

Project: 9022 (Run 857, Clone 1, Gen 2)

Code: Select all

22:38:50:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
22:38:57:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:38:57:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9022 run:857 clone:1 gen:2 core:0xa4 unit:0x00000006ab40417c55e89fa07ac6ed3f
22:38:57:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
22:38:57:WU01:FS00:Connecting to 171.64.65.124:8080
22:39:02:WU01:FS00:Upload complete
22:39:02:WU01:FS00:Server responded WORK_QUIT (404)
22:39:02:WARNING:WU01:FS00:Server did not like results, dumping
22:39:02:WU01:FS00:Cleaning up

Project: 9023 (Run 433, Clone 7, Gen 131)

Code: Select all

22:13:08:WU00:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
22:13:08:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:13:08:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9023 run:433 clone:7 gen:131 core:0xa4 unit:0x00000098ab40417c55f710dfb95d897d
22:13:08:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
22:13:08:WU00:FS00:Connecting to 171.64.65.124:8080
22:13:13:WU00:FS00:Upload complete
22:13:13:WU00:FS00:Server responded WORK_QUIT (404)
22:13:13:WARNING:WU00:FS00:Server did not like results, dumping
22:13:13:WU00:FS00:Cleaning up

[davidcoton]

Problems continue.
Please stop assigning if you can't take the returns. You are wasting our time, effort, and power
(This log is filtered manually because FAHClient only filters on current WUs )

Code: Select all

16:26:12:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9024 run:646 clone:10 gen:48 core:0xa4 unit:0x00000037ab40417c55f3a70c13f96c85
16:26:12:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
16:26:12:WU00:FS00:Connecting to 171.64.65.124:8080
16:27:17:WU00:FS00:Upload 12.78%
16:27:40:WU00:FS00:Upload 17.04%
16:28:33:WU00:FS00:Upload 21.30%
16:28:53:WU00:FS00:Upload 25.56%
16:29:32:WU00:FS00:Upload 29.82%
16:30:12:WU00:FS00:Upload 34.08%
16:30:24:WU00:FS00:Upload 38.34%
16:31:17:WU00:FS00:Upload 42.60%
16:32:09:WU00:FS00:Upload 46.86%
16:32:28:WU00:FS00:Upload 51.13%
16:35:38:WU00:FS00:Upload 55.39%
16:36:01:WU00:FS00:Upload 59.65%
16:36:36:WU00:FS00:Upload 63.91%
16:37:04:WU00:FS00:Upload 68.17%
16:37:38:WU00:FS00:Upload 72.43%
16:38:39:WU00:FS00:Upload 76.69%
16:38:57:WU00:FS00:Upload 80.95%
16:39:17:WU00:FS00:Upload 85.21%
16:40:20:WU00:FS00:Upload 89.47%
16:40:37:WU00:FS00:Upload 93.73%
16:41:33:WU00:FS00:Upload 97.99%
16:45:34:WU00:FS00:Upload complete
16:45:34:WU00:FS00:Server responded WORK_QUIT (404)
16:45:34:WARNING:WU00:FS00:Server did not like results, dumping
16:45:34:WU00:FS00:Cleaning up

[parkut]

Two More WU's not accepted, and now back to slow uploading time

Project: 9026 (Run 32, Clone 12, Gen 8) - about 30 minutes ago

Code: Select all

17:12:10:WU00:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
17:12:10:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:12:10:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9026 run:32 clone:12 gen:8 core:0xa4 unit:0x0000000dab40417c55f3a028292afad4
17:12:10:WU00:FS00:Uploading 1.47MiB to 171.64.65.124
17:12:10:WU00:FS00:Connecting to 171.64.65.124:8080
17:13:30:WU00:FS00:Upload 12.78%
17:14:14:WU00:FS00:Upload 17.04%
17:14:35:WU00:FS00:Upload 21.30%
17:14:45:WU00:FS00:Upload 25.56%
17:15:06:WU00:FS00:Upload 29.82%
17:15:57:WU00:FS00:Upload 34.08%
17:16:21:WU00:FS00:Upload 38.34%
17:16:51:WU00:FS00:Upload 42.60%
17:17:18:WU00:FS00:Upload 46.86%
17:17:58:WU00:FS00:Upload 51.12%
17:18:45:WU00:FS00:Upload 55.38%
17:18:59:WU00:FS00:Upload 59.64%
17:19:11:WU00:FS00:Upload 63.90%
17:19:37:WU00:FS00:Upload 68.16%
17:20:07:WU00:FS00:Upload 72.42%
17:20:42:WU00:FS00:Upload 76.67%
17:21:08:WU00:FS00:Upload 80.93%
17:21:17:WU00:FS00:Upload 85.19%
17:21:59:WU00:FS00:Upload 89.45%
17:22:28:WU00:FS00:Upload 93.71%
17:22:45:WU00:FS00:Upload 97.97%
17:26:11:WU00:FS00:Upload complete
17:26:11:WU00:FS00:Server responded WORK_QUIT (404)
17:26:11:WARNING:WU00:FS00:Server did not like results, dumping
17:26:11:WU00:FS00:Cleaning up

Project: 9024 (Run 290, Clone 1, Gen 62) - about 2 hours ago

Code: Select all

15:49:20:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
15:49:20:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
15:49:20:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9024 run:290 clone:1 gen:62 core:0xa4 unit:0x0000004cab40417c55f3a93a910a2309
15:49:20:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
15:49:20:WU01:FS00:Connecting to 171.64.65.124:8080
15:50:44:WU01:FS00:Upload 12.78%
15:51:07:WU01:FS00:Upload 17.04%
15:51:37:WU01:FS00:Upload 21.31%
15:52:23:WU01:FS00:Upload 25.57%
15:52:41:WU01:FS00:Upload 29.83%
15:53:04:WU01:FS00:Upload 34.09%
15:53:55:WU01:FS00:Upload 38.35%
15:54:26:WU01:FS00:Upload 42.61%
15:54:38:WU01:FS00:Upload 46.87%
15:54:59:WU01:FS00:Upload 51.13%
15:55:40:WU01:FS00:Upload 55.39%
15:55:57:WU01:FS00:Upload 59.65%
15:56:37:WU01:FS00:Upload 63.92%
15:57:24:WU01:FS00:Upload 68.18%
15:58:08:WU01:FS00:Upload 72.44%
15:58:41:WU01:FS00:Upload 76.70%
15:59:09:WU01:FS00:Upload 80.96%
15:59:53:WU01:FS00:Upload 85.22%
16:00:22:WU01:FS00:Upload 89.48%
16:00:46:WU01:FS00:Upload 93.74%
16:01:38:WU01:FS00:Upload 98.00%
16:01:52:WU01:FS00:Upload 100.00%
16:05:19:WU01:FS00:Upload complete
16:05:19:WU01:FS00:Server responded WORK_QUIT (404)
16:05:19:WARNING:WU01:FS00:Server did not like results, dumping
16:05:19:WU01:FS00:Cleaning up

[sryckbos]

I'm sorry about this. I'm actively working towards getting the more capable WS up right now. That should be up and running soon. The problem is that with so few other CPU projects currently available when I stop assigning there are not nearly enough CPU jobs to keep up with demand. That's why I've been reluctant to stop assigning and have kept doing stopgap fixes while I'm trying to get the new WS running. Obviously this has kind of come to a head the last few days where the stopgap fixes aren't enough. I'm sorry about that. I've talked with other people in the PG and there should be more CPU jobs coming in the next few days to help tide over demand while I fix this. But in the mean time it might be hard to get jobs for a little while.

Obviously this is better than the alternative of you running jobs and then having them dumped when uploaded back to my WS, even if it is annoying to not be able to do work temporarily.

[sryckbos]

So it looks like for the time being gbowman has enough CPU jobs on brodgar (128.252.203.2) that I can stop assigning and get everything in much better shape without losing your guys' ability to get jobs. I'll let you know more updates as they happen.

[parkut]

Two more WU's the server did not like. Note the fast upload time.
This was about 16 hours ago.

Project: 9023 (Run 160, Clone 14, Gen 4)

Code: Select all

20:47:21:WU01:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
20:47:22:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:47:22:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9023 run:160 clone:14 gen:4 core:0xa4 unit:0x00000004ab40417c55f712e957cc783e
20:47:22:WU01:FS00:Uploading 1.47MiB to 171.64.65.124
20:47:22:WU01:FS00:Connecting to 171.64.65.124:8080
20:47:26:WU01:FS00:Upload complete
20:47:26:WU01:FS00:Server responded WORK_QUIT (404)
20:47:26:WARNING:WU01:FS00:Server did not like results, dumping
20:47:26:WU01:FS00:Cleaning up

Project: 9021 (Run 110, Clone 11, Gen 4)

Code: Select all

20:59:23:WU02:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
20:59:31:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:59:31:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:9021 run:110 clone:11 gen:4 core:0xa4 unit:0x00000007ab40417c55e82123350191de
20:59:31:WU02:FS00:Uploading 1.47MiB to 171.64.65.124
20:59:31:WU02:FS00:Connecting to 171.64.65.124:8080
20:59:36:WU02:FS00:Upload complete
20:59:36:WU02:FS00:Server responded WORK_QUIT (404)
20:59:36:WARNING:WU02:FS00:Server did not like results, dumping
20:59:36:WU02:FS00:Cleaning up

[sryckbos]

All I can say is that my new work server is almost ready for the big time and going to be launched in the next couple of days, and so after that a bunch of these problems should be alleviated. Thanks for keeping me up to date with these problems, and for your patience.