Restricting maxwell core18

If you think it might be a driver problem, see viewforum.php?f=79

Moderators: Site Moderators, FAHC Science Team

rwh202
Posts: 410
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Restricting maxwell core18

Post by rwh202 »

Ok, here are some parsed logs similar to Bruce's (courtesy of GREP) for four different GTX-750ti from when I started them up again after Christmas break (using advanced flag). They are all under windows using a mix of drivers.

Code: Select all

00:53:57:WU02:FS00:0x17:Project: 9201 (Run 439, Clone 4, Gen 97)
08:54:34:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
08:54:42:WU00:FS00:0x15:Project: 7621 (Run 395, Clone 0, Gen 309)
21:15:13:WU00:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
21:15:26:WU01:FS00:0x17:*********************** Log Started 2014-12-28T21:15:26Z ***********************
21:15:26:WU01:FS00:0x17:Project: 9201 (Run 425, Clone 0, Gen 300)
05:17:55:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
05:18:03:WU00:FS00:0x17:*********************** Log Started 2014-12-29T05:18:03Z ***********************
05:18:03:WU00:FS00:0x17:Project: 9201 (Run 277, Clone 4, Gen 100)
13:21:16:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
13:21:24:WU02:FS00:0x15:Project: 7624 (Run 159, Clone 0, Gen 427)
01:23:34:WU02:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
01:23:46:WU00:FS00:0x15:Project: 7627 (Run 115, Clone 0, Gen 190)
13:24:21:WU00:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
13:24:56:WU01:FS00:0x17:*********************** Log Started 2014-12-30T13:24:56Z ***********************
13:24:56:WU01:FS00:0x17:Project: 9201 (Run 784, Clone 4, Gen 105)
21:27:27:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
21:27:37:WU02:FS00:0x17:*********************** Log Started 2014-12-30T21:27:36Z ***********************
21:27:37:WU02:FS00:0x17:Project: 9201 (Run 812, Clone 3, Gen 128)
05:31:32:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
05:31:41:WU00:FS00:0x17:*********************** Log Started 2014-12-31T05:31:41Z ***********************
05:31:41:WU00:FS00:0x17:Project: 9201 (Run 983, Clone 4, Gen 100)
13:34:39:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
13:34:48:WU01:FS00:0x17:*********************** Log Started 2014-12-31T13:34:47Z ***********************
13:34:48:WU01:FS00:0x17:Project: 9201 (Run 810, Clone 2, Gen 136)
21:37:32:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
21:37:45:WU02:FS00:0x17:*********************** Log Started 2014-12-31T21:37:45Z ***********************
21:37:45:WU02:FS00:0x17:Project: 9201 (Run 554, Clone 2, Gen 100)
05:40:56:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
05:41:04:WU01:FS00:0x17:*********************** Log Started 2015-01-01T05:41:03Z ***********************
05:41:04:WU01:FS00:0x17:Project: 9201 (Run 615, Clone 3, Gen 98)
13:43:36:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
13:43:48:WU00:FS00:0x17:*********************** Log Started 2015-01-01T13:43:48Z ***********************
13:43:48:WU00:FS00:0x17:Project: 9201 (Run 643, Clone 3, Gen 128)
21:47:43:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
21:47:54:WU02:FS00:0x17:*********************** Log Started 2015-01-01T21:47:53Z ***********************
21:47:54:WU02:FS00:0x17:Project: 9201 (Run 681, Clone 4, Gen 127)
05:50:09:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
05:50:17:WU01:FS00:0x17:*********************** Log Started 2015-01-02T05:50:17Z ***********************
05:50:17:WU01:FS00:0x17:Project: 9201 (Run 163, Clone 4, Gen 84)
13:53:40:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
13:53:49:WU00:FS00:0x15:Project: 7621 (Run 816, Clone 0, Gen 314)
01:51:48:WU00:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
01:52:02:WU01:FS00:0x17:*********************** Log Started 2015-01-03T01:52:02Z ***********************
01:52:02:WU01:FS00:0x17:Project: 9201 (Run 887, Clone 3, Gen 119)
09:56:14:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
09:56:22:WU02:FS00:0x17:*********************** Log Started 2015-01-03T09:56:22Z ***********************
09:56:22:WU02:FS00:0x17:Project: 9201 (Run 175, Clone 3, Gen 124)
17:59:51:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
17:59:59:WU00:FS00:0x15:Project: 7626 (Run 122, Clone 0, Gen 469)
06:03:21:WU00:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
06:03:37:WU01:FS00:0x17:*********************** Log Started 2015-01-04T06:03:36Z ***********************
06:03:37:WU01:FS00:0x17:Project: 9201 (Run 827, Clone 2, Gen 140)
14:06:35:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
14:06:44:WU02:FS00:0x17:*********************** Log Started 2015-01-04T14:06:44Z ***********************
14:06:44:WU02:FS00:0x17:Project: 9201 (Run 594, Clone 2, Gen 121)
22:10:22:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
22:10:30:WU00:FS00:0x17:*********************** Log Started 2015-01-04T22:10:30Z ***********************
22:10:30:WU00:FS00:0x17:Project: 9201 (Run 976, Clone 3, Gen 97)
06:14:33:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
06:14:43:WU01:FS00:0x15:Project: 7621 (Run 195, Clone 0, Gen 254)
18:06:09:WU01:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
18:06:20:WU00:FS00:0x17:*********************** Log Started 2015-01-05T18:06:20Z ***********************
18:06:20:WU00:FS00:0x17:Project: 9201 (Run 594, Clone 2, Gen 124)
02:10:06:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
02:10:16:WU02:FS00:0x17:*********************** Log Started 2015-01-06T02:10:15Z ***********************
02:10:16:WU02:FS00:0x17:Project: 9201 (Run 610, Clone 4, Gen 65)
10:12:39:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
10:12:47:WU01:FS00:0x17:*********************** Log Started 2015-01-06T10:12:46Z ***********************
10:12:47:WU01:FS00:0x17:Project: 9201 (Run 197, Clone 3, Gen 123)
18:15:56:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
18:16:05:WU00:FS00:0x17:*********************** Log Started 2015-01-06T18:16:04Z ***********************
18:16:05:WU00:FS00:0x17:Project: 9201 (Run 931, Clone 4, Gen 125)
02:19:33:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
02:19:43:WU02:FS00:0x17:*********************** Log Started 2015-01-07T02:19:43Z ***********************
02:19:43:WU02:FS00:0x17:Project: 9201 (Run 253, Clone 4, Gen 80)
10:23:08:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
10:23:16:WU01:FS00:0x15:Project: 7621 (Run 47, Clone 0, Gen 257)
22:14:24:WU01:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
22:14:37:WU00:FS00:0x15:Project: 7627 (Run 611, Clone 0, Gen 143)
11:33:57:WU00:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
11:34:11:WU02:FS00:0x17:*********************** Log Started 2015-01-08T11:34:11Z ***********************
11:34:11:WU02:FS00:0x17:Project: 9201 (Run 806, Clone 2, Gen 98)
19:37:36:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
19:37:44:WU01:FS00:0x17:*********************** Log Started 2015-01-08T19:37:43Z ***********************
19:37:44:WU01:FS00:0x17:Project: 9201 (Run 422, Clone 3, Gen 119)
03:40:29:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
03:40:38:WU00:FS00:0x17:*********************** Log Started 2015-01-09T03:40:37Z ***********************
03:40:38:WU00:FS00:0x17:Project: 9201 (Run 698, Clone 4, Gen 190)
11:43:56:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
11:44:09:WU02:FS00:0x17:*********************** Log Started 2015-01-09T11:44:09Z ***********************
11:44:09:WU02:FS00:0x17:Project: 9201 (Run 952, Clone 1, Gen 182)
19:48:12:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
19:48:21:WU01:FS00:0x17:*********************** Log Started 2015-01-09T19:48:21Z ***********************
19:48:21:WU01:FS00:0x17:Project: 9201 (Run 719, Clone 4, Gen 78)
03:50:52:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
03:51:00:WU00:FS00:0x17:*********************** Log Started 2015-01-10T03:50:59Z ***********************
03:51:00:WU00:FS00:0x17:Project: 9201 (Run 920, Clone 4, Gen 135)
11:53:41:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
11:53:50:WU02:FS00:0x17:*********************** Log Started 2015-01-10T11:53:50Z ***********************
11:53:50:WU02:FS00:0x17:Project: 9201 (Run 563, Clone 3, Gen 145)
19:56:03:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
19:56:12:WU01:FS00:0x17:*********************** Log Started 2015-01-10T19:56:12Z ***********************
19:56:12:WU01:FS00:0x17:Project: 9201 (Run 201, Clone 4, Gen 131)
03:57:37:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
03:57:44:WU02:FS00:0x15:Project: 7622 (Run 508, Clone 0, Gen 170)
16:04:24:WU02:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
16:04:37:WU01:FS00:0x17:*********************** Log Started 2015-01-11T16:04:37Z ***********************
16:04:37:WU01:FS00:0x17:Project: 9201 (Run 909, Clone 4, Gen 131)
00:08:09:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
00:08:17:WU00:FS00:0x17:*********************** Log Started 2015-01-12T00:08:16Z ***********************
00:08:17:WU00:FS00:0x17:Project: 9201 (Run 301, Clone 3, Gen 132)
08:12:49:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
08:12:57:WU02:FS00:0x15:Project: 7620 (Run 617, Clone 0, Gen 199)
21:41:26:WU02:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
21:41:41:WU01:FS00:0x15:Project: 7622 (Run 169, Clone 0, Gen 136)
11:00:16:WU01:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
11:00:30:WU00:FS00:0x17:*********************** Log Started 2015-01-13T11:00:30Z ***********************
11:00:30:WU00:FS00:0x17:Project: 9201 (Run 946, Clone 4, Gen 150)
19:04:52:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
19:05:08:WU02:FS00:0x17:*********************** Log Started 2015-01-13T19:05:08Z ***********************
19:05:08:WU02:FS00:0x17:Project: 9201 (Run 178, Clone 4, Gen 153)
03:09:06:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
03:09:14:WU01:FS00:0x15:Project: 7621 (Run 633, Clone 0, Gen 312)
16:27:58:WU01:FS00:0x15:Completed  40000000 out of 40000000 steps (100%).
16:28:13:WU00:FS00:0x17:*********************** Log Started 2015-01-14T16:28:12Z ***********************
16:28:13:WU00:FS00:0x17:Project: 9201 (Run 850, Clone 4, Gen 145)
00:30:36:WU00:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
00:30:45:WU02:FS00:0x17:*********************** Log Started 2015-01-15T00:30:44Z ***********************
00:30:45:WU02:FS00:0x17:Project: 9201 (Run 755, Clone 4, Gen 184)
08:33:12:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
08:33:23:WU01:FS00:0x18:*********************** Log Started 2015-01-15T08:33:23Z ***********************
08:33:23:WU01:FS00:0x18:Project: 9110 (Run 2, Clone 2, Gen 78)
00:34:31:WU01:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
00:34:48:WU00:FS00:0x18:*********************** Log Started 2015-01-16T00:34:47Z ***********************
00:34:48:WU00:FS00:0x18:Project: 9106 (Run 38, Clone 7, Gen 2)
16:30:15:WU00:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
16:30:32:WU01:FS00:0x18:*********************** Log Started 2015-01-16T16:30:31Z ***********************
16:30:32:WU01:FS00:0x18:Project: 9109 (Run 41, Clone 6, Gen 1)
08:26:56:WU01:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
08:27:13:WU00:FS00:0x17:*********************** Log Started 2015-01-17T08:27:12Z ***********************
08:27:13:WU00:FS00:0x17:Project: 13001 (Run 178, Clone 1, Gen 77)
08:29:35:WU00:FS00:0x17:ERROR:exception: Force RMSE error of 454.098 with threshold of 5
08:29:35:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:29:44:WU01:FS00:0x17:*********************** Log Started 2015-01-17T08:29:43Z ***********************
08:29:44:WU01:FS00:0x17:Project: 13001 (Run 284, Clone 1, Gen 20)
08:32:06:WU01:FS00:0x17:ERROR:exception: Force RMSE error of 449.296 with threshold of 5
08:32:06:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:32:15:WU00:FS00:0x17:*********************** Log Started 2015-01-17T08:32:14Z ***********************
08:32:15:WU00:FS00:0x17:Project: 13000 (Run 67, Clone 4, Gen 31)
08:34:37:WU00:FS00:0x17:ERROR:exception: Force RMSE error of 450.555 with threshold of 5
08:34:38:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:34:44:WU01:FS00:0x17:*********************** Log Started 2015-01-17T08:34:44Z ***********************
08:34:44:WU01:FS00:0x17:Project: 9201 (Run 823, Clone 4, Gen 160)
16:38:40:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
16:38:48:WU02:FS00:0x17:*********************** Log Started 2015-01-17T16:38:48Z ***********************
16:38:48:WU02:FS00:0x17:Project: 9201 (Run 465, Clone 4, Gen 147)
00:42:07:WU02:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
00:42:16:WU01:FS00:0x17:*********************** Log Started 2015-01-18T00:42:16Z ***********************
00:42:16:WU01:FS00:0x17:Project: 13001 (Run 274, Clone 3, Gen 115)
00:44:35:WU01:FS00:0x17:ERROR:exception: Force RMSE error of 455.123 with threshold of 5
00:44:36:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
00:44:49:WU02:FS00:0x17:*********************** Log Started 2015-01-18T00:44:48Z ***********************
00:44:49:WU02:FS00:0x17:Project: 10469 (Run 0, Clone 277, Gen 70)
00:47:10:WU02:FS00:0x17:ERROR:exception: Force RMSE error of 453.459 with threshold of 5
00:47:10:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
00:47:24:WU01:FS00:0x17:*********************** Log Started 2015-01-18T00:47:23Z ***********************
00:47:24:WU01:FS00:0x17:Project: 9201 (Run 597, Clone 3, Gen 162)
08:50:38:WU01:FS00:0x17:Completed 5000000 out of 5000000 steps (100%)
08:50:49:WU02:FS00:0x18:*********************** Log Started 2015-01-18T08:50:48Z ***********************
08:50:49:WU02:FS00:0x18:Project: 9111 (Run 12, Clone 7, Gen 10)
01:15:35:WU02:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
01:15:52:WU01:FS00:0x18:*********************** Log Started 2015-01-19T01:15:52Z ***********************
01:15:52:WU01:FS00:0x18:Project: 9107 (Run 13, Clone 2, Gen 8)
15:59:58:WU01:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
16:32:44:WU00:FS00:0x18:*********************** Log Started 2015-01-19T16:32:44Z ***********************
16:32:44:WU00:FS00:0x18:Project: 9107 (Run 5, Clone 6, Gen 17)

Code: Select all

16:50:53:WU00:FS01:0x17:*********************** Log Started 2014-12-27T16:50:53Z ***********************
16:50:53:WU00:FS01:0x17:Project: 9201 (Run 418, Clone 2, Gen 95)
00:55:58:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
00:56:11:WU03:FS01:0x17:Project: 9201 (Run 302, Clone 4, Gen 108)
09:02:53:WU03:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
09:03:02:WU01:FS01:0x17:*********************** Log Started 2014-12-28T09:03:01Z ***********************
09:03:02:WU01:FS01:0x17:Project: 9201 (Run 48, Clone 4, Gen 26)
17:07:16:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
17:07:24:WU02:FS01:0x15:Project: 7621 (Run 197, Clone 0, Gen 318)
05:30:57:WU02:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
05:31:09:WU01:FS01:0x17:Project: 9201 (Run 24, Clone 2, Gen 130)
13:38:39:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:38:54:WU00:FS01:0x17:*********************** Log Started 2014-12-29T13:38:54Z ***********************
13:38:54:WU00:FS01:0x17:Project: 9201 (Run 303, Clone 3, Gen 93)
21:44:55:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:45:12:WU01:FS01:0x17:*********************** Log Started 2014-12-29T21:45:12Z ***********************
21:45:12:WU01:FS01:0x17:Project: 9201 (Run 346, Clone 1, Gen 192)
05:52:13:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:52:31:WU02:FS01:0x17:*********************** Log Started 2014-12-30T05:52:31Z ***********************
05:52:31:WU02:FS01:0x17:Project: 9201 (Run 803, Clone 3, Gen 116)
13:58:56:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:59:04:WU00:FS01:0x15:Project: 7621 (Run 420, Clone 0, Gen 269)
01:53:45:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
01:54:13:WU01:FS01:0x17:*********************** Log Started 2014-12-31T01:54:12Z ***********************
01:54:13:WU01:FS01:0x17:Project: 9201 (Run 681, Clone 3, Gen 124)
10:00:41:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
10:00:51:WU02:FS01:0x17:*********************** Log Started 2014-12-31T10:00:50Z ***********************
10:00:51:WU02:FS01:0x17:Project: 9201 (Run 232, Clone 1, Gen 232)
18:06:22:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
18:06:30:WU00:FS01:0x15:Project: 7622 (Run 194, Clone 0, Gen 321)
06:01:40:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
06:01:52:WU02:FS01:0x17:*********************** Log Started 2015-01-01T06:01:51Z ***********************
06:01:52:WU02:FS01:0x17:Project: 9201 (Run 753, Clone 0, Gen 316)
14:08:47:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:09:00:WU01:FS01:0x17:*********************** Log Started 2015-01-01T14:08:59Z ***********************
14:09:00:WU01:FS01:0x17:Project: 9201 (Run 909, Clone 3, Gen 109)
22:17:02:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:17:11:WU00:FS01:0x17:*********************** Log Started 2015-01-01T22:17:11Z ***********************
22:17:11:WU00:FS01:0x17:Project: 9201 (Run 582, Clone 0, Gen 301)
06:23:28:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:23:35:WU02:FS01:0x17:*********************** Log Started 2015-01-02T06:23:35Z ***********************
06:23:35:WU02:FS01:0x17:Project: 9201 (Run 32, Clone 4, Gen 109)
14:29:05:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:29:13:WU01:FS01:0x17:*********************** Log Started 2015-01-02T14:29:13Z ***********************
14:29:13:WU01:FS01:0x17:Project: 9201 (Run 785, Clone 3, Gen 104)
22:34:01:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:34:10:WU02:FS01:0x17:*********************** Log Started 2015-01-02T22:34:10Z ***********************
22:34:10:WU02:FS01:0x17:Project: 9201 (Run 176, Clone 3, Gen 72)
06:40:25:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:40:44:WU00:FS01:0x17:*********************** Log Started 2015-01-03T06:40:44Z ***********************
06:40:44:WU00:FS01:0x17:Project: 9201 (Run 301, Clone 4, Gen 98)
14:49:17:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:49:26:WU01:FS01:0x17:*********************** Log Started 2015-01-03T14:49:25Z ***********************
14:49:26:WU01:FS01:0x17:Project: 9201 (Run 41, Clone 4, Gen 130)
22:56:14:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:56:22:WU02:FS01:0x17:*********************** Log Started 2015-01-03T22:56:22Z ***********************
22:56:22:WU02:FS01:0x17:Project: 9201 (Run 246, Clone 3, Gen 59)
07:01:50:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
07:01:58:WU00:FS01:0x17:*********************** Log Started 2015-01-04T07:01:58Z ***********************
07:01:58:WU00:FS01:0x17:Project: 9201 (Run 237, Clone 4, Gen 121)
15:08:06:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
15:08:15:WU01:FS01:0x17:*********************** Log Started 2015-01-04T15:08:15Z ***********************
15:08:15:WU01:FS01:0x17:Project: 9201 (Run 326, Clone 4, Gen 116)
23:16:04:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
23:16:13:WU02:FS01:0x17:*********************** Log Started 2015-01-04T23:16:12Z ***********************
23:16:13:WU02:FS01:0x17:Project: 9201 (Run 990, Clone 4, Gen 123)
07:22:46:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
07:23:00:WU00:FS01:0x17:*********************** Log Started 2015-01-05T07:23:00Z ***********************
07:23:00:WU00:FS01:0x17:Project: 9201 (Run 621, Clone 4, Gen 94)
15:28:13:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
15:28:21:WU02:FS01:0x17:*********************** Log Started 2015-01-05T15:28:21Z ***********************
15:28:21:WU02:FS01:0x17:Project: 9201 (Run 738, Clone 2, Gen 135)
23:34:14:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
23:34:24:WU01:FS01:0x17:*********************** Log Started 2015-01-05T23:34:23Z ***********************
23:34:24:WU01:FS01:0x17:Project: 9201 (Run 264, Clone 3, Gen 156)
07:40:01:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
07:40:09:WU00:FS01:0x17:*********************** Log Started 2015-01-06T07:40:09Z ***********************
07:40:09:WU00:FS01:0x17:Project: 9201 (Run 144, Clone 2, Gen 111)
15:46:33:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
15:46:41:WU02:FS01:0x17:*********************** Log Started 2015-01-06T15:46:40Z ***********************
15:46:41:WU02:FS01:0x17:Project: 9201 (Run 326, Clone 2, Gen 148)
23:54:46:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
23:54:55:WU01:FS01:0x17:*********************** Log Started 2015-01-06T23:54:54Z ***********************
23:54:55:WU01:FS01:0x17:Project: 9201 (Run 199, Clone 1, Gen 172)
08:00:55:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
08:01:03:WU00:FS01:0x17:*********************** Log Started 2015-01-07T08:01:02Z ***********************
08:01:03:WU00:FS01:0x17:Project: 9201 (Run 475, Clone 3, Gen 154)
16:08:34:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
16:08:42:WU02:FS01:0x17:*********************** Log Started 2015-01-07T16:08:42Z ***********************
16:08:42:WU02:FS01:0x17:Project: 9201 (Run 776, Clone 4, Gen 126)
00:16:16:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
00:16:24:WU01:FS01:0x15:Project: 7621 (Run 354, Clone 0, Gen 237)
13:31:02:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
13:31:18:WU00:FS01:0x17:*********************** Log Started 2015-01-08T13:31:17Z ***********************
13:31:18:WU00:FS01:0x17:Project: 9201 (Run 316, Clone 3, Gen 83)
21:37:01:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:37:09:WU02:FS01:0x17:*********************** Log Started 2015-01-08T21:37:09Z ***********************
21:37:09:WU02:FS01:0x17:Project: 9201 (Run 900, Clone 2, Gen 120)
05:44:12:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:44:20:WU01:FS01:0x15:Project: 7623 (Run 262, Clone 0, Gen 337)
17:39:37:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
17:39:49:WU00:FS01:0x17:*********************** Log Started 2015-01-09T17:39:48Z ***********************
17:39:49:WU00:FS01:0x17:Project: 9201 (Run 746, Clone 3, Gen 140)
01:46:21:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
01:46:30:WU02:FS01:0x17:*********************** Log Started 2015-01-10T01:46:30Z ***********************
01:46:30:WU02:FS01:0x17:Project: 9201 (Run 494, Clone 2, Gen 106)
09:53:18:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
09:53:26:WU01:FS01:0x17:*********************** Log Started 2015-01-10T09:53:26Z ***********************
09:53:26:WU01:FS01:0x17:Project: 9201 (Run 626, Clone 0, Gen 326)
17:58:43:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
17:58:51:WU00:FS01:0x15:Project: 7620 (Run 389, Clone 0, Gen 271)
06:09:08:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
06:09:24:WU01:FS01:0x17:*********************** Log Started 2015-01-11T06:09:23Z ***********************
06:09:24:WU01:FS01:0x17:Project: 9201 (Run 481, Clone 0, Gen 321)
14:14:25:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:14:33:WU00:FS01:0x17:*********************** Log Started 2015-01-11T14:14:33Z ***********************
14:14:33:WU00:FS01:0x17:Project: 9201 (Run 851, Clone 3, Gen 109)
22:21:09:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:21:17:WU02:FS01:0x17:*********************** Log Started 2015-01-11T22:21:17Z ***********************
22:21:17:WU02:FS01:0x17:Project: 9201 (Run 96, Clone 4, Gen 101)
06:29:34:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:29:42:WU01:FS01:0x15:Project: 7621 (Run 773, Clone 0, Gen 202)
19:42:49:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
19:43:00:WU00:FS01:0x15:Project: 7626 (Run 197, Clone 0, Gen 301)
09:17:09:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
09:17:21:WU02:FS01:0x17:*********************** Log Started 2015-01-13T09:17:20Z ***********************
09:17:21:WU02:FS01:0x17:Project: 9201 (Run 71, Clone 4, Gen 131)
17:23:10:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
17:23:18:WU01:FS01:0x17:*********************** Log Started 2015-01-13T17:23:18Z ***********************
17:23:18:WU01:FS01:0x17:Project: 9201 (Run 952, Clone 2, Gen 120)
01:31:07:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
01:31:14:WU00:FS01:0x15:Project: 7627 (Run 605, Clone 0, Gen 68)
14:52:31:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
14:52:43:WU02:FS01:0x17:*********************** Log Started 2015-01-14T14:52:43Z ***********************
14:52:43:WU02:FS01:0x17:Project: 9201 (Run 926, Clone 3, Gen 125)
22:58:57:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:59:05:WU01:FS01:0x17:*********************** Log Started 2015-01-14T22:59:05Z ***********************
22:59:05:WU01:FS01:0x17:Project: 9201 (Run 895, Clone 4, Gen 141)
07:05:50:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
07:05:58:WU00:FS01:0x17:*********************** Log Started 2015-01-15T07:05:57Z ***********************
07:05:58:WU00:FS01:0x17:Project: 9201 (Run 689, Clone 4, Gen 137)
15:10:50:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
15:11:03:WU02:FS01:0x18:*********************** Log Started 2015-01-15T15:11:02Z ***********************
15:11:03:WU02:FS01:0x18:Project: 9109 (Run 10, Clone 2, Gen 22)
06:38:56:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
06:39:12:WU01:FS01:0x17:*********************** Log Started 2015-01-16T06:39:12Z ***********************
06:39:12:WU01:FS01:0x17:Project: 9201 (Run 596, Clone 4, Gen 138)
14:47:34:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:47:54:WU02:FS01:0x18:*********************** Log Started 2015-01-16T14:47:54Z ***********************
14:47:54:WU02:FS01:0x18:Project: 9106 (Run 10, Clone 2, Gen 14)
05:26:11:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
05:26:26:WU00:FS01:0x17:*********************** Log Started 2015-01-17T05:26:26Z ***********************
05:26:26:WU00:FS01:0x17:Project: 13001 (Run 184, Clone 9, Gen 22)
05:28:45:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 452.641 with threshold of 5
05:28:46:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:28:54:WU02:FS01:0x17:*********************** Log Started 2015-01-17T05:28:53Z ***********************
05:28:54:WU02:FS01:0x17:Project: 9201 (Run 521, Clone 3, Gen 169)
13:34:18:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:34:26:WU00:FS01:0x17:*********************** Log Started 2015-01-17T13:34:26Z ***********************
13:34:26:WU00:FS01:0x17:Project: 9201 (Run 930, Clone 4, Gen 140)
21:41:31:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:41:39:WU01:FS01:0x17:*********************** Log Started 2015-01-17T21:41:38Z ***********************
21:41:39:WU01:FS01:0x17:Project: 10467 (Run 0, Clone 188, Gen 103)
21:43:24:WU01:FS01:0x17:ERROR:exception: Force RMSE error of 409.445 with threshold of 5
21:43:24:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
21:43:36:WU00:FS01:0x18:*********************** Log Started 2015-01-17T21:43:35Z ***********************
21:43:36:WU00:FS01:0x18:Project: 9108 (Run 8, Clone 0, Gen 14)
13:28:46:WU00:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
13:29:05:WU01:FS01:0x17:*********************** Log Started 2015-01-18T13:29:05Z ***********************
13:29:05:WU01:FS01:0x17:Project: 9201 (Run 265, Clone 4, Gen 181)
21:36:51:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:37:04:WU00:FS01:0x17:*********************** Log Started 2015-01-18T21:37:03Z ***********************
21:37:04:WU00:FS01:0x17:Project: 9201 (Run 402, Clone 3, Gen 145)
05:45:18:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:45:31:WU02:FS01:0x18:*********************** Log Started 2015-01-19T05:45:31Z ***********************
05:45:31:WU02:FS01:0x18:Project: 9107 (Run 13, Clone 6, Gen 18)
20:33:32:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)

Code: Select all

18:09:02:WU00:FS01:0x17:Project: 9201 (Run 952, Clone 3, Gen 119)
03:02:46:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
03:02:57:WU01:FS01:0x17:*********************** Log Started 2014-12-28T03:02:57Z ***********************
03:02:57:WU01:FS01:0x17:Project: 9201 (Run 817, Clone 3, Gen 112)
11:56:32:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
11:56:38:WU02:FS01:0x15:Project: 7621 (Run 325, Clone 0, Gen 318)
23:48:01:WU02:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
23:48:17:WU00:FS01:0x17:*********************** Log Started 2014-12-28T23:48:16Z ***********************
23:48:17:WU00:FS01:0x17:Project: 9201 (Run 826, Clone 4, Gen 92)
08:45:37:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
08:45:43:WU01:FS01:0x17:*********************** Log Started 2014-12-29T08:45:43Z ***********************
08:45:43:WU01:FS01:0x17:Project: 9201 (Run 461, Clone 2, Gen 141)
17:32:30:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
17:32:38:WU02:FS01:0x17:*********************** Log Started 2014-12-29T17:32:38Z ***********************
17:32:38:WU02:FS01:0x17:Project: 9201 (Run 207, Clone 2, Gen 103)
02:26:01:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
02:26:10:WU00:FS01:0x17:*********************** Log Started 2014-12-30T02:26:09Z ***********************
02:26:10:WU00:FS01:0x17:Project: 9201 (Run 428, Clone 3, Gen 85)
11:19:48:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
11:20:09:WU01:FS01:0x17:*********************** Log Started 2014-12-30T11:20:08Z ***********************
11:20:09:WU01:FS01:0x17:Project: 9201 (Run 286, Clone 2, Gen 116)
20:07:36:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
20:07:51:WU02:FS01:0x17:*********************** Log Started 2014-12-30T20:07:51Z ***********************
20:07:51:WU02:FS01:0x17:Project: 9201 (Run 206, Clone 3, Gen 109)
05:03:18:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:03:28:WU00:FS01:0x17:*********************** Log Started 2014-12-31T05:03:27Z ***********************
05:03:28:WU00:FS01:0x17:Project: 9201 (Run 164, Clone 2, Gen 117)
13:54:14:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:54:24:WU01:FS01:0x17:*********************** Log Started 2014-12-31T13:54:24Z ***********************
13:54:24:WU01:FS01:0x17:Project: 9201 (Run 768, Clone 2, Gen 115)
22:41:50:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
22:41:57:WU02:FS01:0x15:Project: 7622 (Run 18, Clone 0, Gen 307)
10:23:01:WU02:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
10:23:14:WU01:FS01:0x17:*********************** Log Started 2015-01-01T10:23:13Z ***********************
10:23:14:WU01:FS01:0x17:Project: 9201 (Run 632, Clone 2, Gen 114)
19:10:19:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
19:10:28:WU00:FS01:0x17:*********************** Log Started 2015-01-01T19:10:27Z ***********************
19:10:28:WU00:FS01:0x17:Project: 9201 (Run 18, Clone 4, Gen 100)
04:03:34:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
04:03:43:WU02:FS01:0x17:*********************** Log Started 2015-01-02T04:03:43Z ***********************
04:03:43:WU02:FS01:0x17:Project: 9201 (Run 878, Clone 3, Gen 123)
12:57:24:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
12:57:32:WU01:FS01:0x17:*********************** Log Started 2015-01-02T12:57:31Z ***********************
12:57:32:WU01:FS01:0x17:Project: 9201 (Run 560, Clone 4, Gen 127)
21:43:59:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:44:08:WU00:FS01:0x17:*********************** Log Started 2015-01-02T21:44:07Z ***********************
21:44:08:WU00:FS01:0x17:Project: 9201 (Run 448, Clone 4, Gen 131)
06:41:24:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:41:34:WU02:FS01:0x17:*********************** Log Started 2015-01-03T06:41:33Z ***********************
06:41:34:WU02:FS01:0x17:Project: 9201 (Run 218, Clone 4, Gen 139)
15:31:37:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
15:31:46:WU01:FS01:0x17:*********************** Log Started 2015-01-03T15:31:46Z ***********************
15:31:46:WU01:FS01:0x17:Project: 9201 (Run 945, Clone 4, Gen 95)
00:21:14:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
00:21:24:WU00:FS01:0x17:*********************** Log Started 2015-01-04T00:21:23Z ***********************
00:21:24:WU00:FS01:0x17:Project: 9201 (Run 767, Clone 3, Gen 75)
09:19:58:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
09:20:07:WU02:FS01:0x17:*********************** Log Started 2015-01-04T09:20:06Z ***********************
09:20:07:WU02:FS01:0x17:Project: 9201 (Run 632, Clone 4, Gen 118)
18:07:48:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
18:07:55:WU01:FS01:0x17:*********************** Log Started 2015-01-04T18:07:55Z ***********************
18:07:55:WU01:FS01:0x17:Project: 9201 (Run 928, Clone 4, Gen 74)
03:00:26:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
03:00:36:WU00:FS01:0x17:*********************** Log Started 2015-01-05T03:00:36Z ***********************
03:00:36:WU00:FS01:0x17:Project: 9201 (Run 648, Clone 4, Gen 163)
11:54:07:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
11:54:14:WU02:FS01:0x17:*********************** Log Started 2015-01-05T11:54:14Z ***********************
11:54:14:WU02:FS01:0x17:Project: 9201 (Run 150, Clone 3, Gen 113)
20:40:15:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
20:40:25:WU01:FS01:0x17:*********************** Log Started 2015-01-05T20:40:25Z ***********************
20:40:25:WU01:FS01:0x17:Project: 9201 (Run 723, Clone 2, Gen 75)
05:35:29:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:35:38:WU00:FS01:0x17:*********************** Log Started 2015-01-06T05:35:38Z ***********************
05:35:38:WU00:FS01:0x17:Project: 9201 (Run 978, Clone 3, Gen 137)
14:26:55:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:27:07:WU02:FS01:0x17:*********************** Log Started 2015-01-06T14:27:07Z ***********************
14:27:07:WU02:FS01:0x17:Project: 9201 (Run 455, Clone 4, Gen 132)
23:14:11:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
23:14:20:WU01:FS01:0x17:*********************** Log Started 2015-01-06T23:14:20Z ***********************
23:14:20:WU01:FS01:0x17:Project: 9201 (Run 952, Clone 3, Gen 136)
08:13:58:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
08:14:05:WU00:FS01:0x15:Project: 7621 (Run 20, Clone 0, Gen 268)
19:55:42:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
19:55:54:WU01:FS01:0x15:Project: 7621 (Run 175, Clone 0, Gen 265)
07:37:31:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
07:37:43:WU02:FS01:0x17:*********************** Log Started 2015-01-08T07:37:43Z ***********************
07:37:43:WU02:FS01:0x17:Project: 9201 (Run 9, Clone 4, Gen 113)
16:25:38:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
16:25:46:WU00:FS01:0x17:*********************** Log Started 2015-01-08T16:25:45Z ***********************
16:25:46:WU00:FS01:0x17:Project: 9201 (Run 808, Clone 3, Gen 89)
01:11:57:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
01:12:04:WU01:FS01:0x15:Project: 7622 (Run 102, Clone 0, Gen 265)
12:53:36:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
12:53:52:WU02:FS01:0x17:*********************** Log Started 2015-01-09T12:53:51Z ***********************
12:53:52:WU02:FS01:0x17:Project: 9201 (Run 734, Clone 2, Gen 82)
21:41:06:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:41:13:WU00:FS01:0x15:Project: 7627 (Run 221, Clone 0, Gen 176)
09:32:52:WU00:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
09:33:04:WU01:FS01:0x17:*********************** Log Started 2015-01-10T09:33:03Z ***********************
09:33:04:WU01:FS01:0x17:Project: 9201 (Run 32, Clone 4, Gen 129)
18:19:19:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
18:19:27:WU00:FS01:0x17:*********************** Log Started 2015-01-10T18:19:26Z ***********************
18:19:27:WU00:FS01:0x17:Project: 9201 (Run 323, Clone 4, Gen 142)
03:12:24:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
03:12:33:WU02:FS01:0x15:Project: 7621 (Run 371, Clone 0, Gen 219)
15:00:14:WU02:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
15:00:29:WU01:FS01:0x15:Project: 7627 (Run 87, Clone 0, Gen 160)
02:58:47:WU01:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
02:59:02:WU00:FS01:0x17:*********************** Log Started 2015-01-12T02:59:01Z ***********************
02:59:02:WU00:FS01:0x17:Project: 9201 (Run 38, Clone 1, Gen 240)
11:50:41:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
11:50:48:WU01:FS01:0x17:*********************** Log Started 2015-01-12T11:50:47Z ***********************
11:50:48:WU01:FS01:0x17:Project: 9201 (Run 171, Clone 3, Gen 126)
20:38:20:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
20:38:27:WU02:FS01:0x17:*********************** Log Started 2015-01-12T20:38:27Z ***********************
20:38:27:WU02:FS01:0x17:Project: 9201 (Run 969, Clone 3, Gen 137)
05:31:47:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
05:31:55:WU00:FS01:0x17:*********************** Log Started 2015-01-13T05:31:55Z ***********************
05:31:55:WU00:FS01:0x17:Project: 9201 (Run 688, Clone 3, Gen 119)
14:21:20:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
14:21:28:WU01:FS01:0x17:*********************** Log Started 2015-01-13T14:21:28Z ***********************
14:21:28:WU01:FS01:0x17:Project: 9201 (Run 558, Clone 4, Gen 115)
23:09:00:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
23:09:09:WU02:FS01:0x17:*********************** Log Started 2015-01-13T23:09:09Z ***********************
23:09:09:WU02:FS01:0x17:Project: 9201 (Run 830, Clone 2, Gen 155)
08:07:40:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
08:07:49:WU00:FS01:0x17:*********************** Log Started 2015-01-14T08:07:48Z ***********************
08:07:49:WU00:FS01:0x17:Project: 9201 (Run 878, Clone 3, Gen 152)
16:54:58:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
16:55:05:WU01:FS01:0x17:*********************** Log Started 2015-01-14T16:55:05Z ***********************
16:55:05:WU01:FS01:0x17:Project: 9201 (Run 6, Clone 0, Gen 350)
01:45:04:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
01:45:13:WU02:FS01:0x15:Project: 7623 (Run 584, Clone 0, Gen 189)
13:26:14:WU02:FS01:0x15:Completed  40000000 out of 40000000 steps (100%).
13:26:26:WU00:FS01:0x17:*********************** Log Started 2015-01-15T13:26:25Z ***********************
13:26:26:WU00:FS01:0x17:Project: 13000 (Run 1035, Clone 0, Gen 88)
13:28:59:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 457.501 with threshold of 5
13:29:00:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
13:29:13:WU02:FS01:0x18:*********************** Log Started 2015-01-15T13:29:12Z ***********************
13:29:13:WU02:FS01:0x18:Project: 9110 (Run 4, Clone 8, Gen 21)
04:32:44:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
04:33:04:WU01:FS01:0x17:*********************** Log Started 2015-01-16T04:33:04Z ***********************
04:33:04:WU01:FS01:0x17:Project: 9201 (Run 355, Clone 3, Gen 156)
13:25:09:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:25:30:WU02:FS01:0x18:*********************** Log Started 2015-01-16T13:25:30Z ***********************
13:25:30:WU02:FS01:0x18:Project: 9105 (Run 14, Clone 2, Gen 8)
04:12:21:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
04:12:35:WU00:FS01:0x17:*********************** Log Started 2015-01-17T04:12:35Z ***********************
04:12:35:WU00:FS01:0x17:Project: 10468 (Run 0, Clone 110, Gen 55)
04:14:36:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 409.334 with threshold of 5
04:14:37:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
04:14:48:WU02:FS01:0x17:*********************** Log Started 2015-01-17T04:14:48Z ***********************
04:14:48:WU02:FS01:0x17:Project: 13000 (Run 1748, Clone 1, Gen 21)
04:17:21:WU02:FS01:0x17:ERROR:exception: Force RMSE error of 449.618 with threshold of 5
04:17:21:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
04:17:27:WU00:FS01:0x17:*********************** Log Started 2015-01-17T04:17:26Z ***********************
04:17:27:WU00:FS01:0x17:Project: 9201 (Run 317, Clone 2, Gen 117)
13:05:29:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
13:05:36:WU01:FS01:0x17:*********************** Log Started 2015-01-17T13:05:36Z ***********************
13:05:36:WU01:FS01:0x17:Project: 9201 (Run 707, Clone 3, Gen 146)
21:51:54:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
21:52:01:WU00:FS01:0x17:*********************** Log Started 2015-01-17T21:52:01Z ***********************
21:52:01:WU00:FS01:0x17:Project: 9201 (Run 577, Clone 2, Gen 191)
06:51:13:WU00:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:51:25:WU02:FS01:0x18:*********************** Log Started 2015-01-18T06:51:24Z ***********************
06:51:25:WU02:FS01:0x18:Project: 9113 (Run 28, Clone 7, Gen 21)
21:30:55:WU02:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)
21:31:09:WU01:FS01:0x17:*********************** Log Started 2015-01-18T21:31:08Z ***********************
21:31:09:WU01:FS01:0x17:Project: 9201 (Run 845, Clone 0, Gen 261)
06:29:17:WU01:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
06:29:26:WU02:FS01:0x17:*********************** Log Started 2015-01-19T06:29:25Z ***********************
06:29:26:WU02:FS01:0x17:Project: 10469 (Run 0, Clone 218, Gen 83)
06:33:15:WU02:FS01:0x17:ERROR:exception: Force RMSE error of 455.755 with threshold of 5
06:33:16:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:33:22:WU01:FS01:0x17:*********************** Log Started 2015-01-19T06:33:22Z ***********************
06:33:22:WU01:FS01:0x17:Project: 13000 (Run 2317, Clone 0, Gen 82)
06:37:09:WU01:FS01:0x17:ERROR:exception: Force RMSE error of 453.604 with threshold of 5
06:37:09:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:37:24:WU02:FS01:0x17:*********************** Log Started 2015-01-19T06:37:23Z ***********************
06:37:24:WU02:FS01:0x17:Project: 9201 (Run 35, Clone 4, Gen 150)
15:26:16:WU02:FS01:0x17:Completed 5000000 out of 5000000 steps (100%)
16:13:54:WU00:FS01:0x17:*********************** Log Started 2015-01-19T16:13:53Z ***********************
16:13:54:WU00:FS01:0x17:Project: 13001 (Run 283, Clone 0, Gen 39)
16:16:26:WU00:FS01:0x17:ERROR:exception: Force RMSE error of 452.148 with threshold of 5
16:16:27:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:20:01:WU00:FS01:0x18:*********************** Log Started 2015-01-19T16:20:01Z ***********************
16:20:01:WU00:FS01:0x18:Project: 9107 (Run 3, Clone 1, Gen 54)
07:11:08:WU00:FS01:0x18:Completed 2500000 out of 2500000 steps (100%)

Code: Select all

*********************** Log Started 2014-12-27T18:07:36Z ***********************
18:09:13:WU01:FS02:0x17:*********************** Log Started 2014-12-27T18:09:12Z ***********************
18:09:13:WU01:FS02:0x17:Project: 9201 (Run 760, Clone 2, Gen 115)
*********************** Log Started 2014-12-27T18:07:36Z ***********************
18:09:13:WU01:FS02:0x17:*********************** Log Started 2014-12-27T18:09:12Z ***********************
18:09:13:WU01:FS02:0x17:Project: 9201 (Run 760, Clone 2, Gen 115)
02:29:27:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
02:29:36:WU02:FS02:0x17:*********************** Log Started 2014-12-28T02:29:36Z ***********************
02:29:36:WU02:FS02:0x17:Project: 9201 (Run 312, Clone 3, Gen 130)
10:55:56:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
10:56:03:WU00:FS02:0x15:Project: 7627 (Run 67, Clone 0, Gen 168)
22:46:17:WU00:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
22:46:29:WU01:FS02:0x17:*********************** Log Started 2014-12-28T22:46:28Z ***********************
22:46:29:WU01:FS02:0x17:Project: 9201 (Run 727, Clone 3, Gen 89)
07:18:50:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
07:19:04:WU02:FS02:0x17:*********************** Log Started 2014-12-29T07:19:04Z ***********************
07:19:04:WU02:FS02:0x17:Project: 9201 (Run 731, Clone 4, Gen 87)
15:32:33:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
15:32:41:WU00:FS02:0x17:*********************** Log Started 2014-12-29T15:32:40Z ***********************
15:32:41:WU00:FS02:0x17:Project: 9201 (Run 998, Clone 4, Gen 113)
23:47:25:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
23:47:39:WU01:FS02:0x17:*********************** Log Started 2014-12-29T23:47:38Z ***********************
23:47:39:WU01:FS02:0x17:Project: 9201 (Run 463, Clone 2, Gen 102)
08:21:01:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
08:21:24:WU02:FS02:0x17:*********************** Log Started 2014-12-30T08:21:23Z ***********************
08:21:24:WU02:FS02:0x17:Project: 9201 (Run 276, Clone 3, Gen 90)
16:33:50:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
16:34:15:WU00:FS02:0x17:*********************** Log Started 2014-12-30T16:34:14Z ***********************
16:34:15:WU00:FS02:0x17:Project: 9201 (Run 509, Clone 0, Gen 334)
00:50:48:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
00:50:57:WU01:FS02:0x17:*********************** Log Started 2014-12-31T00:50:57Z ***********************
00:50:57:WU01:FS02:0x17:Project: 9201 (Run 490, Clone 2, Gen 95)
09:23:47:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
09:23:54:WU02:FS02:0x17:*********************** Log Started 2014-12-31T09:23:53Z ***********************
09:23:54:WU02:FS02:0x17:Project: 9201 (Run 495, Clone 2, Gen 122)
17:34:20:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
17:34:34:WU00:FS02:0x17:*********************** Log Started 2014-12-31T17:34:34Z ***********************
17:34:34:WU00:FS02:0x17:Project: 9201 (Run 722, Clone 4, Gen 105)
01:52:42:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
01:52:51:WU01:FS02:0x17:*********************** Log Started 2015-01-01T01:52:51Z ***********************
01:52:51:WU01:FS02:0x17:Project: 9201 (Run 365, Clone 4, Gen 104)
10:20:47:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
10:20:56:WU00:FS02:0x17:*********************** Log Started 2015-01-01T10:20:56Z ***********************
10:20:56:WU00:FS02:0x17:Project: 9201 (Run 714, Clone 0, Gen 244)
18:31:37:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
18:31:45:WU02:FS02:0x17:*********************** Log Started 2015-01-01T18:31:44Z ***********************
18:31:45:WU02:FS02:0x17:Project: 9201 (Run 396, Clone 4, Gen 107)
02:54:05:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
02:54:14:WU01:FS02:0x17:*********************** Log Started 2015-01-02T02:54:14Z ***********************
02:54:14:WU01:FS02:0x17:Project: 9201 (Run 716, Clone 4, Gen 132)
11:19:39:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
11:19:47:WU00:FS02:0x17:*********************** Log Started 2015-01-02T11:19:47Z ***********************
11:19:47:WU00:FS02:0x17:Project: 9201 (Run 640, Clone 0, Gen 300)
19:31:42:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
19:31:49:WU02:FS02:0x17:*********************** Log Started 2015-01-02T19:31:48Z ***********************
19:31:49:WU02:FS02:0x17:Project: 9201 (Run 67, Clone 1, Gen 196)
03:57:32:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
03:57:43:WU01:FS02:0x17:*********************** Log Started 2015-01-03T03:57:42Z ***********************
03:57:43:WU01:FS02:0x17:Project: 9201 (Run 181, Clone 4, Gen 115)
12:21:51:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
12:22:00:WU00:FS02:0x17:*********************** Log Started 2015-01-03T12:22:00Z ***********************
12:22:00:WU00:FS02:0x17:Project: 9201 (Run 952, Clone 1, Gen 162)
20:35:13:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
20:35:21:WU02:FS02:0x17:*********************** Log Started 2015-01-03T20:35:21Z ***********************
20:35:21:WU02:FS02:0x17:Project: 9201 (Run 317, Clone 4, Gen 138)
05:05:06:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
05:05:16:WU01:FS02:0x17:*********************** Log Started 2015-01-04T05:05:15Z ***********************
05:05:16:WU01:FS02:0x17:Project: 9201 (Run 933, Clone 4, Gen 102)
13:25:11:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
13:25:40:WU00:FS02:0x17:Project: 9201 (Run 420, Clone 4, Gen 64)
21:37:28:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
21:37:36:WU02:FS02:0x17:*********************** Log Started 2015-01-04T21:37:36Z ***********************
21:37:36:WU02:FS02:0x17:Project: 9201 (Run 53, Clone 4, Gen 92)
06:10:38:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
06:10:53:WU01:FS02:0x17:*********************** Log Started 2015-01-05T06:10:52Z ***********************
06:10:53:WU01:FS02:0x17:Project: 9201 (Run 666, Clone 2, Gen 68)
14:27:58:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
14:28:04:WU00:FS02:0x15:Project: 7620 (Run 694, Clone 0, Gen 281)
02:04:31:WU00:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
02:04:43:WU02:FS02:0x17:*********************** Log Started 2015-01-06T02:04:42Z ***********************
02:04:43:WU02:FS02:0x17:Project: 9201 (Run 265, Clone 1, Gen 256)
10:33:49:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
10:33:57:WU01:FS02:0x17:*********************** Log Started 2015-01-06T10:33:57Z ***********************
10:33:57:WU01:FS02:0x17:Project: 9201 (Run 531, Clone 4, Gen 134)
18:46:37:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
18:46:45:WU00:FS02:0x17:*********************** Log Started 2015-01-06T18:46:44Z ***********************
18:46:45:WU00:FS02:0x17:Project: 9201 (Run 606, Clone 3, Gen 160)
03:10:12:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
03:10:21:WU02:FS02:0x17:*********************** Log Started 2015-01-07T03:10:21Z ***********************
03:10:21:WU02:FS02:0x17:Project: 9201 (Run 348, Clone 3, Gen 126)
11:34:36:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
11:34:44:WU01:FS02:0x17:*********************** Log Started 2015-01-07T11:34:44Z ***********************
11:34:44:WU01:FS02:0x17:Project: 9201 (Run 263, Clone 0, Gen 299)
19:46:03:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
19:46:10:WU02:FS02:0x17:*********************** Log Started 2015-01-07T19:46:09Z ***********************
19:46:10:WU02:FS02:0x17:Project: 9201 (Run 762, Clone 4, Gen 140)
04:10:13:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
04:10:22:WU00:FS02:0x17:*********************** Log Started 2015-01-08T04:10:22Z ***********************
04:10:22:WU00:FS02:0x17:Project: 9201 (Run 535, Clone 4, Gen 112)
12:31:09:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
12:31:15:WU01:FS02:0x17:*********************** Log Started 2015-01-08T12:31:15Z ***********************
12:31:15:WU01:FS02:0x17:Project: 9201 (Run 841, Clone 4, Gen 112)
20:44:21:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
20:44:28:WU02:FS02:0x17:*********************** Log Started 2015-01-08T20:44:27Z ***********************
20:44:28:WU02:FS02:0x17:Project: 9201 (Run 999, Clone 3, Gen 139)
05:05:11:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
05:05:21:WU00:FS02:0x17:*********************** Log Started 2015-01-09T05:05:21Z ***********************
05:05:21:WU00:FS02:0x17:Project: 9201 (Run 957, Clone 3, Gen 188)
13:23:55:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
13:24:01:WU01:FS02:0x15:Project: 7626 (Run 284, Clone 0, Gen 572)
01:09:43:WU01:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
01:09:56:WU02:FS02:0x17:*********************** Log Started 2015-01-10T01:09:56Z ***********************
01:09:56:WU02:FS02:0x17:Project: 9201 (Run 24, Clone 4, Gen 152)
09:40:49:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
09:40:58:WU00:FS02:0x17:*********************** Log Started 2015-01-10T09:40:57Z ***********************
09:40:58:WU00:FS02:0x17:Project: 9201 (Run 641, Clone 2, Gen 138)
17:52:08:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
17:52:15:WU02:FS02:0x17:*********************** Log Started 2015-01-10T17:52:15Z ***********************
17:52:15:WU02:FS02:0x17:Project: 9201 (Run 357, Clone 4, Gen 134)
02:12:28:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
02:12:37:WU01:FS02:0x15:Project: 7620 (Run 259, Clone 0, Gen 233)
13:58:33:WU01:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
13:58:49:WU00:FS02:0x15:Project: 7624 (Run 389, Clone 0, Gen 402)
01:52:46:WU00:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
01:52:59:WU02:FS02:0x15:Project: 7621 (Run 630, Clone 0, Gen 342)
13:34:06:WU02:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
13:34:29:WARNING:WU00:FS02:Failed to get assignment from '171.67.108.200:80': Failed to connect to 171.67.108.200:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
13:34:36:WU00:FS02:0x17:*********************** Log Started 2015-01-12T13:34:35Z ***********************
13:34:36:WU00:FS02:0x17:Project: 9201 (Run 79, Clone 2, Gen 148)
21:44:44:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
21:44:50:WU01:FS02:0x15:Project: 7626 (Run 100, Clone 0, Gen 329)
09:38:58:WU01:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
09:39:09:WU02:FS02:0x17:*********************** Log Started 2015-01-13T09:39:08Z ***********************
09:39:09:WU02:FS02:0x17:Project: 9201 (Run 629, Clone 3, Gen 135)
17:51:36:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
17:51:43:WU00:FS02:0x17:*********************** Log Started 2015-01-13T17:51:43Z ***********************
17:51:43:WU00:FS02:0x17:Project: 9201 (Run 450, Clone 2, Gen 169)
02:13:30:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
02:13:39:WU01:FS02:0x15:Project: 7623 (Run 437, Clone 0, Gen 168)
13:54:39:WU01:FS02:0x15:Completed  40000000 out of 40000000 steps (100%).
13:54:53:WU02:FS02:0x17:*********************** Log Started 2015-01-14T13:54:53Z ***********************
13:54:53:WU02:FS02:0x17:Project: 9201 (Run 948, Clone 4, Gen 130)
22:06:50:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
22:07:03:WU00:FS02:0x17:*********************** Log Started 2015-01-14T22:07:02Z ***********************
22:07:03:WU00:FS02:0x17:Project: 9201 (Run 617, Clone 2, Gen 157)
06:38:16:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
06:38:25:WU01:FS02:0x17:*********************** Log Started 2015-01-15T06:38:25Z ***********************
06:38:25:WU01:FS02:0x17:Project: 9201 (Run 68, Clone 4, Gen 148)
14:55:19:WU01:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
14:55:26:WU00:FS02:0x17:*********************** Log Started 2015-01-15T14:55:25Z ***********************
14:55:26:WU00:FS02:0x17:Project: 9201 (Run 889, Clone 4, Gen 146)
23:06:40:WU00:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
23:06:50:WU01:FS02:0x17:*********************** Log Started 2015-01-15T23:06:49Z ***********************
23:06:50:WU01:FS02:0x17:Project: 13000 (Run 1288, Clone 7, Gen 119)
23:10:38:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
23:10:48:WU00:FS02:0x18:*********************** Log Started 2015-01-15T23:10:48Z ***********************
23:10:48:WU00:FS02:0x18:Project: 9107 (Run 34, Clone 7, Gen 9)
15:18:14:WU00:FS02:0x18:Completed 2500000 out of 2500000 steps (100%)
15:18:29:WU01:FS02:0x18:*********************** Log Started 2015-01-16T15:18:29Z ***********************
15:18:29:WU01:FS02:0x18:Project: 9107 (Run 9, Clone 2, Gen 11)
06:22:24:WU01:FS02:0x18:Completed 2500000 out of 2500000 steps (100%)
06:22:38:WU02:FS02:0x17:*********************** Log Started 2015-01-17T06:22:38Z ***********************
06:22:38:WU02:FS02:0x17:Project: 13001 (Run 55, Clone 2, Gen 67)
06:25:12:WU02:FS02:0x17:ERROR:exception: Force RMSE error of 454.86 with threshold of 5
06:25:12:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:25:20:WU01:FS02:0x17:*********************** Log Started 2015-01-17T06:25:19Z ***********************
06:25:20:WU01:FS02:0x17:Project: 13000 (Run 1256, Clone 0, Gen 75)
06:27:54:WU01:FS02:0x17:ERROR:exception: Force RMSE error of 451.861 with threshold of 5
06:27:54:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:27:59:WU02:FS02:0x17:*********************** Log Started 2015-01-17T06:27:58Z ***********************
06:27:59:WU02:FS02:0x17:Project: 10466 (Run 0, Clone 324, Gen 153)
17:10:33:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
17:10:43:WU00:FS02:0x17:*********************** Log Started 2015-01-17T17:10:43Z ***********************
17:10:43:WU00:FS02:0x17:Project: 13000 (Run 2165, Clone 4, Gen 109)
17:13:17:WU00:FS02:0x17:ERROR:exception: Force RMSE error of 453.924 with threshold of 5
17:13:17:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
17:13:24:WU02:FS02:0x17:*********************** Log Started 2015-01-17T17:13:23Z ***********************
17:13:24:WU02:FS02:0x17:Project: 9201 (Run 723, Clone 3, Gen 99)
01:32:55:WU02:FS02:0x17:Completed 5000000 out of 5000000 steps (100%)
01:33:04:WU01:FS02:0x18:*********************** Log Started 2015-01-18T01:33:04Z ***********************
01:33:04:WU01:FS02:0x18:Project: 9110 (Run 6, Clone 5, Gen 14)
17:47:58:WU01:FS02:0x18:Completed 2500000 out of 2500000 steps (100%)
17:48:13:WU00:FS02:0x18:*********************** Log Started 2015-01-18T17:48:13Z ***********************
17:48:13:WU00:FS02:0x18:Project: 9110 (Run 6, Clone 9, Gen 11)
10:06:36:WU00:FS02:0x18:Completed 2500000 out of 2500000 steps (100%)
10:06:52:WU01:FS02:0x18:*********************** Log Started 2015-01-19T10:06:51Z ***********************
10:06:52:WU01:FS02:0x18:Project: 9108 (Run 10, Clone 5, Gen 12)
00:50:56:WU01:FS02:0x18:Completed 2500000 out of 2500000 steps (100%)
As can be seen from the logs, there was a nice period of just 9201, interspersed with core_15 (that all ran successfully) until the 15th Jan. Then came along core_18 (which all ran successfully - the latest version Bruce is running seems to a regression in this regard) and the 'Maxwell incompatible' core_17 projects that fail on these systems.

Now where it gets interesting, and partially confirms Bollix47, is that yesterday on four Maxwell systems I have done a brand new Mint install with latest drivers (346.35) and no flags and noticed that the GTX970 and GTX980 cards only get 9201 (with 15% more PPD than windows). However, the GTX-750ti have also seen 13000 and 13001 projects - but this is different - they have actually RUN! Obviously somethings are changing behind the scenes again (core version?).
I have just done a fresh Windows7 install on another gtx-750ti system to test the latest and greatest under Windows and see what happens when that encounters a non-9201 core_17, but still waiting an assignment.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Restricting maxwell core18

Post by bruce »

Yes, grep would be nice, but the windows ports that I've seen aren't quite up to par.

As far as a behind-the-scenes core update, that didn't happen. On Windows I've got 0x17:Version 0.0.52 and 0x18:Version 0.0.3 and 0x15:Version 2.25 (Wed May 9 17:03:01 EDT 2012). How about you?

As far as the regression, if you grep that log for 0x18:Project you'll find 35 WUs all of which were finished in spite of three recovered hiccups.

When you have an 0x17:ERROR:exception: Force RMSE error, the WU is not completed. I count 16 in the logs you posted. I'd call that a more important regression that errors which are successfully recovered.

I think it's fair to say that NV still has driver work to be done.
rwh202
Posts: 410
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Restricting maxwell core18

Post by rwh202 »

Thanks for the information Bruce.

On a fresh windows install, various core_17 WUs are now running using latest drivers and 0x17 Version 0.0.52. My previous install was running 0.0.55 and would only do 9201. Core_18 is version 0.0.3 both before and after and hasn't missed a beat so far.

So it looks like in my case the gtx750ti can now fold everything being assigned to it (windows and linux) and that these assignments are probably intentional. The next question is whether the other Maxwell cards perform the same and if their assignment restrictions should be relaxed too?
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Restricting maxwell core18

Post by bruce »

Based on reports here on the forum, the GM107 [GTX 750 Ti, etc.] successfully completes many WUs which fail on GM204 [GTX 970-980]. Even though both are both called Maxwell I've started calling them Maxwell1 and Maxwell2. They're different enough to warrant selectively blocking some projects from one and not the other. If you want more details about how/why they're different you'll have to consult NVidia. If you want more details about which projects are involved, you'll have to consult the project owners at PG concerning their lab tests. (I don't expect you can get either one.)

Note: We're sort of off-topic talking about Core17, but restrictions can apply to individual projects, whether they happen to use Core18 or Core17.
... latest drivers and 0x17 Version 0.0.52. My previous install was running 0.0.55 and would only do 9201.
Exactly what changed? Does that mean (A) assignments did not change, but (B) some non-9201 projects do run successfully on 0.0.52 but 0.0.55 fails to run the same non-9201 projects? Are you now running different drivers or did just Core_17 change?
rwh202
Posts: 410
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Restricting maxwell core18

Post by rwh202 »

bruce wrote:
... latest drivers and 0x17 Version 0.0.52. My previous install was running 0.0.55 and would only do 9201.
Exactly what changed? Does that mean (A) assignments did not change, but (B) some non-9201 projects do run successfully on 0.0.52 but 0.0.55 fails to run the same non-9201 projects? Are you now running different drivers or did just Core_17 change?
Sorry, I was very unscientific and changed two variables at once. Going back and retrying the combinations it would appear that the drivers are indeed to blame.

340.52 fail non-9201 WUs whereas 347.09 run them, regardless of core_17 0.0.52 or 0.0.55

Just to complicate things, I've got a couple of gtx-960 coming on Monday. Do we think they'll be a Maxwell 3 or just a 2.1?
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Restricting maxwell core18

Post by bruce »

rwh202 wrote:Just to complicate things, I've got a couple of gtx-960 coming on Monday. Do we think they'll be a Maxwell 3 or just a 2.1?
At the present time, Stanford has classified it as what I'm calling a maxwell2. It uses a GM206 rather than a GM204 or a GM107 but it seems likely that the FAH classification should follow the third character of the chip designation. I have not seen any technical details from NV answering your question.
strong_epoxy
Posts: 1
Joined: Sun Feb 22, 2015 6:00 am

Re: Restricting maxwell core18

Post by strong_epoxy »

So is it all better now? Can I get the hottest nVidia GPU on Newegg and run FAH at top speed?
stevehat1
Posts: 23
Joined: Fri Jun 06, 2008 12:33 pm

Re: Restricting maxwell core18

Post by stevehat1 »

So in the last few days my GTX 970 has completed 12 Core 18 WU's of the 1047* variety all with no issues. Driver version 347.25 and 347.88 as of yesterday in Win 7 64.

Looks as if the Maxwell restriction has been lifted, anything official on that? Will Maxwell be assigned the full gamut of GPU WU's going forward?

With all the trama getting Maxwell to fold an official statement would be nice if indeed the issue has been resolved.
ImageImage
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Restricting maxwell core18

Post by 7im »

Officially, Nvidia announced the release of updated drivers for Maxwell GPUs with several bug fixes. If you have a Maxwell GPU, please consider upgrading to v347.xx or newer drivers for Windows, or 346.xx drivers or newer for Linux. Nothing official from FAH regarding a change in assignments as yet.
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Restricting maxwell core18

Post by bruce »

Stanford always attempts (A) to avoid assigning projects that will crash and (B) to manage the overall assignment process to maximize TOTAL throughput. The Core_18 restriction is/was not unique except perhaps in the time it has taken to resolve it plus the amount of publicity it has gotten.

Some projects run on OS=x1 with hardware=x2,x3 with client-type=x4 while others do not. There are all kinds of restrictions in place and whenever possible, development works to "fix" whatever caused that restriction in the first place. Once a fix is in place, it's gradually tested and gradually loosened (if possible). New projects and/or new servers also enter in to the process of relaxing restrictions.

Stanford will decide when restrictions need to be added and when they can be relaxed. If Donors decide to add their own restrictions to "protect" their system from "bad" WUs those donors also take on the responsibility to decide when to relax them.
Post Reply