Most credits missing

ThorstenR · Post by **ThorstenR** » Wed Jan 27, 2021 5:32 pm

Hello all,

for a few days now, most of the credits seems to be missing - since Monday morning, only 4 WUs out of 13 got credited.
This seems to be related either to servers 155.247.166.219 and 129.32.209.201 or to projects 14188 and 16927. The results are successfully uploaded, the server sends WORK_ACK, but they are never credited. Here's are some example snippets from the log file:

Code: Select all

01:30:45:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14188 run:5 clone:1491 gen:8 core:0xa7 unit:0x000005d3000000080000376c00000005
01:32:56:WU00:FS00:Starting
01:32:56:WU00:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/osx/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7" -dir 00 -suffix 01 -version 706 -lifeline 68368 -checkpoint 15 -np 4
01:32:56:WU00:FS00:Started FahCore on PID 2037
01:32:56:WU00:FS00:Core PID:2038
01:32:56:WU00:FS00:FahCore 0xa7 started
01:32:57:WU00:FS00:0xa7:*********************** Log Started 2021-01-25T01:32:56Z ***********************
01:32:57:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:32:57:WU00:FS00:0xa7:       Type: 0xa7
01:32:57:WU00:FS00:0xa7:       Core: Gromacs
01:32:57:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 2037 -checkpoint 15 -np 4
01:32:57:WU00:FS00:0xa7:************************************ CBang *************************************
01:32:57:WU00:FS00:0xa7:       Date: Nov 27 2019
01:32:57:WU00:FS00:0xa7:       Time: 03:27:01
01:32:57:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
01:32:57:WU00:FS00:0xa7:     Branch: master
01:32:57:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
01:32:57:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
01:32:57:WU00:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++ -fPIC
01:32:57:WU00:FS00:0xa7:   Platform: darwin 19.0.0
01:32:57:WU00:FS00:0xa7:       Bits: 64
01:32:57:WU00:FS00:0xa7:       Mode: Release
01:32:57:WU00:FS00:0xa7:************************************ System ************************************
01:32:57:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
01:32:57:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 70 Stepping 1
01:32:57:WU00:FS00:0xa7:       CPUs: 8
01:32:57:WU00:FS00:0xa7:     Memory: 16.00GiB
01:32:57:WU00:FS00:0xa7:Free Memory: 9.86GiB
01:32:57:WU00:FS00:0xa7:    Threads: POSIX_THREADS
01:32:57:WU00:FS00:0xa7: OS Version: 10.11
01:32:57:WU00:FS00:0xa7:Has Battery: true
01:32:57:WU00:FS00:0xa7: On Battery: false
01:32:57:WU00:FS00:0xa7: UTC Offset: 1
01:32:57:WU00:FS00:0xa7:        PID: 2038
01:32:57:WU00:FS00:0xa7:        CWD: /Library/Application Support/FAHClient/work
01:32:57:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
01:32:57:WU00:FS00:0xa7:    Version: 0.0.19
01:32:57:WU00:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
01:32:57:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
01:32:57:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
01:32:57:WU00:FS00:0xa7:       Date: Nov 25 2019
01:32:57:WU00:FS00:0xa7:       Time: 16:41:59
01:32:57:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
01:32:57:WU00:FS00:0xa7:     Branch: master
01:32:57:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
01:32:57:WU00:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
01:32:57:WU00:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++
01:32:57:WU00:FS00:0xa7:   Platform: darwin 19.0.0
01:32:57:WU00:FS00:0xa7:       Bits: 64
01:32:57:WU00:FS00:0xa7:       Mode: Release
01:32:57:WU00:FS00:0xa7:************************************ Build *************************************
01:32:57:WU00:FS00:0xa7:       SIMD: avx_256
01:32:57:WU00:FS00:0xa7:********************************************************************************
01:32:57:WU00:FS00:0xa7:Project: 14188 (Run 5, Clone 1491, Gen 8)
01:32:57:WU00:FS00:0xa7:Unit: 0x00000000000000000000000000000000
01:32:57:WU00:FS00:0xa7:Reading tar file core.xml
01:32:57:WU00:FS00:0xa7:Reading tar file frame8.tpr
01:32:57:WU00:FS00:0xa7:Digital signatures verified
01:32:57:WU00:FS00:0xa7:Calling: mdrun -s frame8.tpr -o frame8.trr -cpt 15 -nt 4
01:32:57:WU00:FS00:0xa7:Steps: first=10000000 total=1250000
<snip>
22:29:33:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
22:29:34:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:29:34:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14188 run:5 clone:1491 gen:8 core:0xa7 unit:0x000005d3000000080000376c00000005
22:29:34:WU00:FS00:Uploading 11.93MiB to 155.247.166.219
22:29:34:WU00:FS00:Connecting to 155.247.166.219:8080
22:29:40:WU00:FS00:Upload 13.62%
22:29:46:WU00:FS00:Upload 26.19%
22:29:52:WU00:FS00:Upload 48.19%
22:29:58:WU00:FS00:Upload 62.34%
22:30:04:WU00:FS00:Upload 71.24%
22:30:10:WU00:FS00:Upload 77.53%
22:30:16:WU00:FS00:Upload 80.67%
22:30:22:WU00:FS00:Upload 86.96%
22:30:28:WU00:FS00:Upload 98.48%
22:30:32:WU00:FS00:Upload complete
22:30:32:WU00:FS00:Server responded WORK_ACK (400)
22:30:32:WU00:FS00:Final credit estimate, 22772.00 points
22:30:32:WU00:FS00:Cleaning up

Code: Select all

07:33:16:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:33:16:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16927 run:10 clone:1115 gen:2 core:0xa7 unit:0x0000045b000000020000421f0000000a
07:33:16:WU00:FS00:Uploading 6.30MiB to 129.32.209.201
07:33:16:WU00:FS00:Connecting to 129.32.209.201:8080
07:33:22:WU00:FS00:Upload 13.89%
07:33:28:WU00:FS00:Upload 49.61%
07:33:34:WU00:FS00:Upload 86.32%
07:33:36:WU00:FS00:Upload complete
07:33:36:WU00:FS00:Server responded WORK_ACK (400)
07:33:36:WU00:FS00:Final credit estimate, 8970.00 points
07:33:36:WU00:FS00:Cleaning up

Anything I can do to help?

Cheers,
Thorsten

Post by **bruce** » Wed Jan 27, 2021 10:22 pm

ThorstenR wrote:
Anything I can do to help?

Cheers,
Thorsten

No. (unfortunately)

There are multiple reports of multiple servers failing to transfer the information about points and WUs to the stats system. I expect this is a temporary condition and they'll get there eventually. The problem has been escalated and I see no responses from the folks that can fix that type of problem.

ThorstenR · Post by **ThorstenR** » Thu Jan 28, 2021 7:31 pm

Just one little observation: both projects (and servers) that failed are hosted at Temple in Philly. Maybe there's a networking thing between Temple and the FAH stats servers?

JimboPalmer · Post by **JimboPalmer** » Thu Jan 28, 2021 7:52 pm

ThorstenR wrote:Just one little observation: both projects (and servers) that failed are hosted at Temple in Philly. Maybe there's a networking thing between Temple and the FAH stats servers?

Just a comment: this can be one reason these can be so slow to resolve, watch the bureaucracy: F@H team at Temple, IT Department at Temple, Help Desk at ISP A, Help Desk at ISP B, IT Department at University with Stats Server, F@H team at University with Stats Server. You know that not all of these groups actually talk to each other. This game of 'Telephone' can tale weeks.

https://en.wikipedia.org/wiki/Chinese_whispers

Neil-B · Post by **Neil-B** » Thu Jan 28, 2021 8:11 pm

.. and even once the issue is resolved the clean up exercise of identifying all the "missed" stats and catching them up takes time

Folding Forum

Most credits missing

Most credits missing

Re: Most credits missing

Re: Most credits missing

Re: Most credits missing

Re: Most credits missing