Not Receiving GPU Work Units from from 192.0.2.1

Moderators: Site Moderators, FAHC Science Team

tinskip
Posts: 3
Joined: Sun Jul 12, 2020 9:37 pm

Not Receiving GPU Work Units from from 192.0.2.1

Post by tinskip »

Hi.

I have configured a slot for my GPU, which I believe is supported, but am not getting work units:

Code: Select all

14:33:26:WU00:FS01:Connecting to assign1.foldingathome.org:80
14:33:27:WU00:FS01:Assigned to work server 192.0.2.1
14:33:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:Tahiti XT [Radeon R9 200/HD 7900/8970] from 192.0.2.1
14:33:27:WU00:FS01:Connecting to 192.0.2.1:8080
14:34:43:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
14:34:43:WU00:FS01:Connecting to 192.0.2.1:80
14:35:58:ERROR:WU00:FS01:Exception: Failed to connect to 192.0.2.1:80: Operation timed out
Any suggestions?

Thanks!
JimboPalmer
Posts: 2522
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: Not Receiving GPU Work Units

Post by JimboPalmer »

192.0.2.1 means the GPU it found does not seem to be able to fold.

If we had the first 200 lines of your log, we could make better guesses why. Here are directions:

viewtopic.php?f=24&t=26036
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
HendricksSA
Posts: 336
Joined: Fri Jun 26, 2009 4:34 am

Re: Not Receiving GPU Work Units

Post by HendricksSA »

Is this right? I thought 192 IP address was within a local network. Not sure why assign1 would send you to an IP address like that.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Not Receiving GPU Work Units

Post by bruce »

We don't have a good explanation for WHY you're being sent to 192.0.2.1 but we do have enough reports to know that you only get sent there when there's a configuration problem with your GPU, as @JimboPalmer has said. Show us the first 200 lines of FAH's log where the GPUs are detected and configured. See below.
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Not Receiving GPU Work Units

Post by Joe_H »

HendricksSA wrote:Is this right? I thought 192 IP address was within a local network. Not sure why assign1 would send you to an IP address like that.
That particular block of 192.0.2.n addresses is reserved for a different purpose, documentation. In theory they could connect to a message giving further explanation, but that would be a later enhancement.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
JimboPalmer
Posts: 2522
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: Not Receiving GPU Work Units

Post by JimboPalmer »

HendricksSA wrote:Is this right? I thought 192 IP address was within a local network. Not sure why assign1 would send you to an IP address like that.
Actually, only 192.168.x.x is for that. I have a great deal of 192.152.17x.x I obtained in 1992. It is a perfectly normal IP range.

https://en.wikipedia.org/wiki/Private_network
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Not Receiving GPU Work Units

Post by bruce »

The DNS for most home-based routers creates what used to be called a Class C network. The first 24 bits are the network number and last 8 bits are the device number. (Your mileage may vary, of course but) I'll bet your network is 192.168.1.x or 1192.168.0.x or something like that where x is between 1 and 254. If that applies to you, then 192.0.2.x is treated as a separate network and it goes nowhere. Most home LANs aren't expected to have more than 254 devices.
CaptJolly
Posts: 2
Joined: Wed May 13, 2020 9:45 am

Re: Not Receiving GPU Work Units

Post by CaptJolly »

Hi,

I have seen something similar.
There seems to be a bug in the fallback logic to port 80 when port 8080 is not open.
If the network/servers are busy then the routine which attempts to get a new GPU WU seems to just give up.
It will never ask for a GPU work unit again. I've seen as long as 7 days. A reboot restores normal service.
I saw this A LOT on maybe 12 PCs during the time port 8080 was closed on my network.
I have since managed to get port 80 open and this issue no longer occurs.
All working now, so can't get an example of the log for this.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Not Receiving GPU Work Units

Post by Neil-B »

@CaptJolly ... Think your issue might be different ... The OP is specifically referring to a scenario that indicates some form of issue with the GPU not being recognized as able to fold - not any form of comms configuration issue.

@tinskip ... Someone may be able to assist if you could post first 200 lines of so of log that show what GPU has been detected and how it is configured ... At present we have no clue as to any of your setup and so are unable to even confirm that your GPU is able to fold let alone assist in getting it working.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Not Receiving GPU Work Units

Post by Joe_H »

CaptJolly wrote:Hi,

I have seen something similar.
There seems to be a bug in the fallback logic to port 80 when port 8080 is not open.
If the network/servers are busy then the routine which attempts to get a new GPU WU seems to just give up.
It will never ask for a GPU work unit again. I've seen as long as 7 days. A reboot restores normal service.
I saw this A LOT on maybe 12 PCs during the time port 8080 was closed on my network.
I have since managed to get port 80 open and this issue no longer occurs.
All working now, so can't get an example of the log for this.
Your problem is totally different from the OP. There is a known bug where the client does not detect a stalled or interrupted connection and then retry the upload or download. It applies to all slots, CPU and GPU. This issue has been around a long time, code improvements in the most recent couple versions of the client- 7.5.1 & 7.6.n - have reduced the frequency of the connection failing to retry. Restarting the FAHClient process or rebooting are the only fixes without additional software usage. Some workarounds using tcpview on Windows or similar software for other OSs have been posted on the forum.

Getting back to the OP, the problem is that an apparently supported GPU is getting no assignment and being redirected to a documentation IP number which is non-routable. That indicates the AS has no GPU WUs that can be assigned to AMD cards in species 5.

Code: Select all

14:33:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:Tahiti XT [Radeon R9 200/HD 7900/8970] from 192.0.2.1
14:33:27:WU00:FS01:Connecting to 192.0.2.1:8080
All of the AMD R9 200, HD 7900/8970 GPUs support FP64 and a sufficient level of OpenCL so could get assignments. Some of the GPU projects were restricted from species 5 AMD cards due to a bug connected with certain ranges of simulation size and GCN based chips, but a recent update to Core_22 should have fixed that. It is possible all available WUs that could be assigned were out or that some projects have not had those restrictions removed.

However there are additional possibilities, one is whether the OP is running on a 32 or 64-bit OS. No GPU assignments are available for 32-bit OSs as I understand it. Posting the first 100-200 lines of the log showing the hardware, software, and client information would go towards answering that and other questions.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
tinskip
Posts: 3
Joined: Sun Jul 12, 2020 9:37 pm

Re: Not Receiving GPU Work Units

Post by tinskip »

Here is the very start of the log:

Code: Select all

*********************** Log Started 2020-07-13T14:31:20Z ***********************
14:31:21:Trying to access database...
14:31:22:Successfully acquired database lock
14:31:23:Read GPUs.txt
14:31:23:Enabled folding slot 00: READY cpu:23
14:31:23:Enabled folding slot 01: READY gpu:1:Tahiti XT [Radeon R9 200/HD 7900/8970]
14:31:23:ERROR:No compute devices matched GPU #1 {
14:31:23:ERROR:  "vendor": 4098,
14:31:23:ERROR:  "device": 26520,
14:31:23:ERROR:  "type": 1,
14:31:23:ERROR:  "species": 5,
14:31:23:ERROR:  "description": "Tahiti XT [Radeon R9 200/HD 7900/8970]"
14:31:23:ERROR:}.  You may need to update your graphics drivers.
14:31:23:****************************** FAHClient ******************************
14:31:23:    Version: 7.6.13
14:31:23:     Author: Joseph Coffland <[email protected]>
14:31:23:  Copyright: 2020 foldingathome.org
14:31:23:   Homepage: https://foldingathome.org/
14:31:23:       Date: Apr 27 2020
14:31:23:       Time: 21:20:45
14:31:23:   Revision: 5a652817f46116b6e135503af97f18e094414e3b
14:31:23:     Branch: master
14:31:23:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
14:31:23:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
14:31:23:             -Wno-unused-local-typedefs -stdlib=libc++
14:31:23:   Platform: darwin 19.2.0
14:31:23:       Bits: 64
14:31:23:       Mode: Release
14:31:23:     Config: /Library/Application Support/FAHClient/config.xml
14:31:23:******************************** CBang ********************************
14:31:23:       Date: Apr 24 2020
14:31:23:       Time: 17:07:50
14:31:23:   Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
14:31:23:     Branch: master
14:31:23:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
14:31:23:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
14:31:23:             -Wno-unused-local-typedefs -stdlib=libc++ -fPIC
14:31:23:   Platform: darwin 19.2.0
14:31:23:       Bits: 64
14:31:23:       Mode: Release
14:31:23:******************************* System ********************************
14:31:23:        CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
14:31:23:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
14:31:23:       CPUs: 24
14:31:23:     Memory: 64.00GiB
14:31:23:Free Memory: 60.59GiB
14:31:23:    Threads: POSIX_THREADS
14:31:23: OS Version: 10.15
14:31:23:Has Battery: false
14:31:23: On Battery: false
14:31:23: UTC Offset: -7
14:31:23:        PID: 135
14:31:23:        CWD: /Library/Application Support/FAHClient
14:31:23:         OS: Darwin 19.5.0 x86_64
14:31:23:    OS Arch: AMD64
14:31:23:       GPUs: 2
14:31:23:      GPU 0: Bus:2 Slot:0 Func:0 AMD:5 Tahiti XT [Radeon R9 200/HD 7900/8970]
14:31:23:      GPU 1: Bus:6 Slot:0 Func:0 AMD:5 Tahiti XT [Radeon R9 200/HD 7900/8970]
14:31:23:       CUDA: Not detected: Failed to open dynamic library 'libcuda.dylib':
14:31:23:             dlopen(libcuda.dylib, 1): image not found
14:31:23:     OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.dylib':
14:31:23:             dlopen(libOpenCL.dylib, 1): image not found
14:31:23:******************************* libFAH ********************************
14:31:23:       Date: Apr 15 2020
14:31:23:       Time: 14:43:28
14:31:23:   Revision: 216968bc7025029c841ed6e36e81a03a316890d3
14:31:23:     Branch: master
14:31:23:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
14:31:23:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
14:31:23:             -Wno-unused-local-typedefs -stdlib=libc++
14:31:23:   Platform: darwin 19.2.0
14:31:23:       Bits: 64
14:31:23:       Mode: Release
14:31:23:***********************************************************************
14:31:23:<config>
14:31:23:  <!-- Folding Slot Configuration -->
14:31:23:  <gpu v='true'/>
14:31:23:
14:31:23:  <!-- Network -->
14:31:23:  <proxy v=':8080'/>
14:31:23:
14:31:23:  <!-- Slot Control -->
14:31:23:  <power v='full'/>
14:31:23:
14:31:23:  <!-- User Information -->
14:31:23:  <passkey v='*****'/>
14:31:23:  <team v='446'/>
14:31:23:  <user v='tinskip'/>
14:31:23:
14:31:23:  <!-- Folding Slots -->
14:31:23:  <slot id='0' type='CPU'/>
14:31:23:  <slot id='1' type='GPU'>
14:31:23:    <gpu-index v='1'/>
14:31:23:    <opencl-index v='0'/>
14:31:23:  </slot>
14:31:23:</config>
14:31:23:WU02:FS00:Starting
14:31:23:WU02:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/osx/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7" -dir 02 -suffix 01 -version 706 -lifeline 135 -checkpoint 15 -np 23
14:31:23:WU02:FS00:Started FahCore on PID 473
14:31:25:WU02:FS00:Core PID:520
14:31:25:WU02:FS00:FahCore 0xa7 started
14:31:35:WU02:FS00:0xa7:*********************** Log Started 2020-07-13T14:31:34Z ***********************
14:31:35:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
14:31:35:WU02:FS00:0xa7:       Type: 0xa7
14:31:35:WU02:FS00:0xa7:       Core: Gromacs
14:31:35:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 706 -lifeline 473 -checkpoint 15 -np 23
14:31:35:WU02:FS00:0xa7:************************************ CBang *************************************
14:31:35:WU02:FS00:0xa7:       Date: Nov 27 2019
14:31:35:WU02:FS00:0xa7:       Time: 03:27:01
14:31:35:WU02:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
14:31:35:WU02:FS00:0xa7:     Branch: master
14:31:35:WU02:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
14:31:35:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
14:31:35:WU02:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++ -fPIC
14:31:35:WU02:FS00:0xa7:   Platform: darwin 19.0.0
14:31:35:WU02:FS00:0xa7:       Bits: 64
14:31:35:WU02:FS00:0xa7:       Mode: Release
14:31:35:WU02:FS00:0xa7:************************************ System ************************************
14:31:35:WU02:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
14:31:35:WU02:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
14:31:35:WU02:FS00:0xa7:       CPUs: 24
14:31:35:WU02:FS00:0xa7:     Memory: 64.00GiB
14:31:35:WU02:FS00:0xa7:Free Memory: 57.50GiB
14:31:35:WU02:FS00:0xa7:    Threads: POSIX_THREADS
14:31:35:WU02:FS00:0xa7: OS Version: 10.15
14:31:35:WU02:FS00:0xa7:Has Battery: false
14:31:35:WU02:FS00:0xa7: On Battery: false
14:31:35:WU02:FS00:0xa7: UTC Offset: -7
14:31:35:WU02:FS00:0xa7:        PID: 520
14:31:35:WU02:FS00:0xa7:        CWD: /Library/Application Support/FAHClient/work
14:31:35:WU02:FS00:0xa7:******************************** Build - libFAH ********************************
14:31:35:WU02:FS00:0xa7:    Version: 0.0.19
14:31:35:WU02:FS00:0xa7:     Author: Joseph Coffland <[email protected]>
14:31:35:WU02:FS00:0xa7:  Copyright: 2019 foldingathome.org
14:31:35:WU02:FS00:0xa7:   Homepage: https://foldingathome.org/
14:31:35:WU02:FS00:0xa7:       Date: Nov 25 2019
14:31:35:WU02:FS00:0xa7:       Time: 16:41:59
14:31:35:WU02:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
14:31:35:WU02:FS00:0xa7:     Branch: master
14:31:35:WU02:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)
14:31:35:WU02:FS00:0xa7:    Options: -std=c++11 -O3 -funroll-loops -mmacosx-version-min=10.7
14:31:35:WU02:FS00:0xa7:             -Wno-unused-local-typedefs -stdlib=libc++
14:31:35:WU02:FS00:0xa7:   Platform: darwin 19.0.0
14:31:35:WU02:FS00:0xa7:       Bits: 64
14:31:35:WU02:FS00:0xa7:       Mode: Release
14:31:35:WU02:FS00:0xa7:************************************ Build *************************************
14:31:35:WU02:FS00:0xa7:       SIMD: avx_256
14:31:35:WU02:FS00:0xa7:********************************************************************************
14:31:35:WU02:FS00:0xa7:Project: 17201 (Run 0, Clone 1250, Gen 24)
14:31:35:WU02:FS00:0xa7:Unit: 0x0000001c031532b95efd3dbd761861c3
14:31:35:WU02:FS00:0xa7:Digital signatures verified
14:31:35:WU02:FS00:0xa7:Reducing thread count from 23 to 22 to avoid domain decomposition by a prime number > 3
14:31:35:WU02:FS00:0xa7:Reducing thread count from 22 to 21 to avoid domain decomposition with large prime factor 11
14:31:36:WU02:FS00:0xa7:Calling: mdrun -s frame24.tpr -o frame24.trr -x frame24.xtc -cpi state.cpt -cpt 15 -nt 21
14:33:23:WU02:FS00:0xa7:Steps: first=6000000 total=250000
14:33:24:WU02:FS00:0xa7:Completed 83521 out of 250000 steps (33%)
14:33:26:WARNING:WU02:FS00:Detected clock skew (1 mins 53 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
14:33:26:WU00:FS01:Connecting to assign1.foldingathome.org:80
14:33:27:WU00:FS01:Assigned to work server 192.0.2.1
14:33:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:Tahiti XT [Radeon R9 200/HD 7900/8970] from 192.0.2.1
14:33:27:WU00:FS01:Connecting to 192.0.2.1:8080
14:33:41:WU02:FS00:0xa7:Completed 85000 out of 250000 steps (34%)
14:34:08:WU02:FS00:0xa7:Completed 87500 out of 250000 steps (35%)
14:34:35:WU02:FS00:0xa7:Completed 90000 out of 250000 steps (36%)
14:34:43:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Not Receiving GPU Work Units

Post by Neil-B »

At least one issue may be a driver one in that it appears OpenCL is not present ... but I am not a GPU guru so I'll let others confirm if this could cause issue you are having and assist further.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Joe_H
Site Admin
Posts: 7937
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Not Receiving GPU Work Units

Post by Joe_H »

GPU folding is not currently supported on OS X:

Code: Select all

14:31:23:        CWD: /Library/Application Support/FAHClient
14:31:23:         OS: Darwin 19.5.0 x86_64
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
JimboPalmer
Posts: 2522
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: Not Receiving GPU Work Units

Post by JimboPalmer »

If you dual boot to Windows or Linux, the Tahiti XT is 9 years old but should fold on those OSs.
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
tinskip
Posts: 3
Joined: Sun Jul 12, 2020 9:37 pm

Re: Not Receiving GPU Work Units

Post by tinskip »

Thanks for letting me know. Sounds like Mac GPU support is pretty much a non-starter for me.
Post Reply