I am getting no points: Server responded WORK_ACK (400)

[Leonetienne]

I am not getting any points for maybe half of my finished work units...
They finish folding, get uploaded and according to the log the server responds with WORK_ACK (400)

Code: Select all

12:15:49:WU02:FS01:Server responded WORK_ACK (400)

Will i get these points eventueally?
Is my gpu time just lost?
Will these results go to the researches at all?

[anandhanju]

Can you post a few of those WU numbers? The line in the log will have a Project, Run, Clone and Gen number than uniquely identifies the WU.

[Yujah]

I was here yesterday inquiring after getting no CPU WUs. Have gotten two since then, but also experiencing not being credited for them. Not in points, and also not in "Total WUs", which is given that I am still < 10 (9 currently, according to the stats, but should be 11 then) is something I was keeping an eye on to see if/how that stuff with the QRB works.

The projects this concerns are 14523 and 13851. In both cases an initial upload fails (to resp. 128.252.203.10 and 40.114.52.201) and a second attempt succeeds (to resp. 155.247.166.220 and 13.82.98.119) but both latter servers then apparently don't communicate with the stats server. Or something.

===
03:00:19:WU00:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
03:00:20:WU00:FS00:0xa7:Saving result file ..\logfile_01.txt
03:00:20:WU00:FS00:0xa7:Saving result file frame6.trr
03:00:20:WU00:FS00:0xa7:Saving result file frame6.xtc
03:00:20:WU00:FS00:0xa7:Saving result file md.log
03:00:20:WU00:FS00:0xa7:Saving result file science.log
03:00:20:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
03:00:20:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:00:20:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14523 run:978 clone:5 gen:6 core:0xa7 unit:0x0000000680fccb0a5e781c5dc26aee64
03:00:20:WU00:FS00:Uploading 1.50MiB to 128.252.203.10
03:00:20:WU00:FS00:Connecting to 128.252.203.10:8080
03:00:41:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
03:00:41:WU00:FS00:Connecting to 128.252.203.10:80
03:01:02:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 128.252.203.10:80: [ ... ]
03:01:02:WU00:FS00:Trying to send results to collection server
03:01:02:WU00:FS00:Uploading 1.50MiB to 155.247.166.220
03:01:02:WU00:FS00:Connecting to 155.247.166.220:8080
03:01:08:WU00:FS00:Upload 41.68%
03:01:14:WU00:FS00:Upload 87.53%
03:01:16:WU00:FS00:Upload complete
03:01:16:WU00:FS00:Server responded WORK_ACK (400)
03:01:16:WU00:FS00:Final credit estimate, 1406.00 points
03:01:16:WU00:FS00:Cleaning up
===

and

===
10:07:37:WU02:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
10:07:39:WU02:FS00:0xa7:Saving result file ..\logfile_01.txt
10:07:39:WU02:FS00:0xa7:Saving result file frame1.edr
10:07:39:WU02:FS00:0xa7:Saving result file frame1.trr
10:07:39:WU02:FS00:0xa7:Saving result file frame1.xtc
10:07:39:WU02:FS00:0xa7:Saving result file md.log
10:07:39:WU02:FS00:0xa7:Saving result file science.log
10:07:39:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
10:07:40:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
10:07:40:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:13851 run:0 clone:31505 gen:1 core:0xa7 unit:0x00000002287234c95e7883719b33f0a6
10:07:40:WU02:FS00:Uploading 2.47MiB to 40.114.52.201
10:07:40:WU02:FS00:Connecting to 40.114.52.201:8080
10:08:01:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
10:08:01:WU02:FS00:Connecting to 40.114.52.201:80
10:08:22:WARNING:WU02:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: [ ... ]
10:08:22:WU02:FS00:Trying to send results to collection server
10:08:22:WU02:FS00:Uploading 2.47MiB to 13.82.98.119
10:08:22:WU02:FS00:Connecting to 13.82.98.119:8080
10:08:28:WU02:FS00:Upload 30.42%
10:08:34:WU02:FS00:Upload 60.83%
10:08:40:WU02:FS00:Upload 88.71%
10:08:43:WU02:FS00:Upload complete
10:08:43:WU02:FS00:Server responded WORK_ACK (400)
10:08:43:WU02:FS00:Final credit estimate, 1000.00 points
10:08:43:WU02:FS00:Cleaning up
===

Note; the GPU WUs I've gotten during that same time do seem to be properly credited.

[Leonetienne]

Can you post a few of those WU numbers? The line in the log will have a Project, Run, Clone and Gen number than uniquely identifies the WU.

The last one should still be in the log.
However the last two were both yesterday, i think.
My CPU workload has also finished since creating this thread. Also failed.

Where exactly do i find these identifiers?
These are the surrounding lines in the logfiles.

Code: Select all

12:58:58:WU01:FS01:Upload 93.26%
12:59:04:WU01:FS01:Upload 94.39%
12:59:10:WU01:FS01:Upload 96.39%
12:59:16:WU01:FS01:Upload 98.77%
12:59:20:WU01:FS01:Upload complete
12:59:20:WU01:FS01:Server responded WORK_ACK (400)
12:59:20:WU01:FS01:Final credit estimate, 15396.00 points
12:59:20:WU01:FS01:Cleaning up
13:00:31:WU00:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
13:02:36:WU00:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
13:03:21:Removing old file 'configs/config-20191130-043557.xml'
13:03:21:Saving configuration to config.xml

[anandhanju]

@Yujah, The first one (project:14523 run:978 clone:5 gen:6) has been credited.

Code: Select all

[{"user":"Yujah","team":223518,"cpuid":"2D48795E307FE1A7","credit":193,"credit_time":"2020-03-25 03:04:48","log_time":"2020-03-25 03:02:16","assign_time":"2020-03-25 00:40:00","days":0.0988,"code":"Ok"}]

Considering this was your 9th WU after the passkey was added, it is reasonable that you only received the base credits and no bonus.

Code: Select all

[{"finished":9,"expired":0,"active":0}]

The second one (project:13851 run:0 clone:31505 gen:1) hasn't been credited yet. It can take a few hours for work server/ collection server validate the result, send it to the stats database and for the stats db to do its thing before it starts showing up.

[anandhanju]

@Leonetienne, the WU identifiers should be a few lines above what you've posted. It will look something like this:

10:07:40:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:13851 run:0 clone:31505 gen:1 core:0xa7

[Leonetienne]

Can you post a few of those WU numbers? The line in the log will have a Project, Run, Clone and Gen number than uniquely identifies the WU.

I think ive found them:
today:

Code: Select all

10:31:09:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11762 run:0 clone:4584 gen:7 core:0x22 unit:0x0000001080fccb0a5e6d810fffc517ef
12:12:17:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11762 run:0 clone:4584 gen:7 core:0x22 unit:0x0000001080fccb0a5e6d810fffc517ef

yesterday:

Code: Select all

09:31:51:WU01:FS01:0x22:Project: 11776 (Run 0, Clone 1689, Gen 9)
14:54:39:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11747 run:0 clone:2491 gen:12 core:0x22 unit:0x0000001b8ca304e75e6a7fd18efb36c4

I got these by using notepad++ to search for "WORK_ACK (400)" and then backwards-searching for the next "clone:"

[Yujah]

@anandhanju:

Thanks very much for checking. I'm pretty much positive that if my current stats include that first CPU WU then it must in fact be the case that one GPU WU has not counted towards the "Total WUs" as currently reported by my stats --- but well, let me not dwell on it. It seems that at least everything's basically working if it's not the case that CPU WUs aren't being credited, period.

[anandhanju]

@Leonetienne, both your WUs have been credited with the the base points only as you've not completed 10 units with the passkey.

Code: Select all

[{"finished":4,"expired":0,"active":0}]

project:11762 run:0 clone:4584 gen:7

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":7608,"credit_time":"2020-03-25 12:21:00","log_time":"2020-03-25 12:15:49","assign_time":"2020-03-25 10:29:58","days":0.0735,"code":"Ok"}

project:11747 run:0 clone:2491 gen:12

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":15396,"credit_time":"2020-03-23 16:09:42","log_time":"2020-03-23 15:58:00","assign_time":"2020-03-23 12:58:30","days":0.1247,"code":"Ok"}

I can see the one from the 23rd (15396) in your EOC stats page. The other one should appear shortly.

[Leonetienne]

Can you post a few of those WU numbers? The line in the log will have a Project, Run, Clone and Gen number than uniquely identifies the WU.

My next finished workunit is uploading currently.
Lets see if it succeeds!

Aaaaaaaaaaand it's gone.

Code: Select all

13:56:40:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11780 run:0 clone:726 gen:14 core:0x22 unit:0x000000130d5a98395e73c58772888697

I am really no smart-ass, but i really see no use in investing hours of cpu/gpu time a day and real money in elictricity (expensive in germany) just for 60-70% of my work-unity to fail uploading.
Not only am i getting no credit for them, i do not even know if they get arrive at the research labs

I still don't know if the ACK-400 fails are gone for good or if i get credit for them eventually?

[Leonetienne]

@Leonetienne, both your WUs have been credited with the the base points only as you've not completed 10 units with the passkey.

Code: Select all

[{"finished":4,"expired":0,"active":0}]

project:11762 run:0 clone:4584 gen:7

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":7608,"credit_time":"2020-03-25 12:21:00","log_time":"2020-03-25 12:15:49","assign_time":"2020-03-25 10:29:58","days":0.0735,"code":"Ok"}

project:11747 run:0 clone:2491 gen:12

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":15396,"credit_time":"2020-03-23 16:09:42","log_time":"2020-03-23 15:58:00","assign_time":"2020-03-23 12:58:30","days":0.1247,"code":"Ok"}

I can see the one from the 23rd (15396) in your EOC stats page. The other one should appear shortly.

How many work units do i have finished according to the server?
Should be at least 5-6 in 2020, including the 70% that ACK_400-failed

[Leonetienne]

@Leonetienne, both your WUs have been credited with the the base points only as you've not completed 10 units with the passkey.

Code: Select all

[{"finished":4,"expired":0,"active":0}]

project:11762 run:0 clone:4584 gen:7

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":7608,"credit_time":"2020-03-25 12:21:00","log_time":"2020-03-25 12:15:49","assign_time":"2020-03-25 10:29:58","days":0.0735,"code":"Ok"}

project:11747 run:0 clone:2491 gen:12

Code: Select all

{"user":"Leon_Etienne","team":225605,"cpuid":"B6F0E15D527A6EC1","credit":15396,"credit_time":"2020-03-23 16:09:42","log_time":"2020-03-23 15:58:00","assign_time":"2020-03-23 12:58:30","days":0.1247,"code":"Ok"}

I can see the one from the 23rd (15396) in your EOC stats page. The other one should appear shortly.

How many work units do i have finished according to the server?
Should be at least 5-6 in 2020, including the 70% that ACK_400-failed

So it seems like they get credited automatically, after a few hours?

[Yujah]

Aaaaaaaaaaand it's gone.

Code: Select all

13:56:40:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11780 run:0 clone:726 gen:14 core:0x22 unit:0x000000130d5a98395e73c58772888697

Leonetienne; there does not seem to be cause to conclude "it's gone" from that. Are you interpreting the "error:NO_ERROR" other than as saying that no error occurred while sending? Because that's what it means.

If you check my bits of quoted log you see the same thing; it's the WORK_ACK lines and"Final credit estimate" ones that are important; if those are also present as you in at least one case showed to be then all's seemingly well (this while noting that the estimate is incorrect until you have 10 WUs logged; until then you only get the base credit whereas the estimate is determined assuming 10 or more if you have a passkey set in the first place).

As mentioned, also still not quite sure I'm not missing one or more WUs total, but, well, yes, any I may be might still come up as soon as some overwhelmed server somewhere gets around to checking/crediting the results. All seems basically well; just very, very busy...

[Leonetienne]

Aaaaaaaaaaand it's gone.

Code: Select all

13:56:40:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11780 run:0 clone:726 gen:14 core:0x22 unit:0x000000130d5a98395e73c58772888697

Leonetienne; there does not seem to be cause to conclude "it's gone" from that. Are you interpreting the "error:NO_ERROR" other than as saying that no error occurred while sending? Because that's what it means.

If you check my bits of quoted log you see the same thing; it's the WORK_ACK lines and"Final credit estimate" ones that are important; if those are also present as you in at least one case showed to be then all's seemingly well (this while noting that the estimate is incorrect until you have 10 WUs logged; until then you only get the base credit whereas the estimate is determined assuming 10 or more if you have a passkey set in the first place).

As mentioned, also still not quite sure I'm not missing one or more WUs total, but, well, yes, any I may be might still come up as soon as some overwhelmed server somewhere gets around to checking/crediting the results. All seems basically well; just very, very busy...

that's nice to hear.
*folding away*

[Rel25917]

If you got the work acknowledged and credit estimate the science done is good. Sometime the info doesn't make it to the stats server and they have to manually re run whatever it is to transfer the info to the stats system. Sadly that may happen a lot with how slammed all the servers are right now but be assured the work done has been recieved.