Extremely low performance on project 13409 with Radeon VII

[ThWuensche]

My node is just processing a WU from project 13409, which is stated as:

Project 13409

Testing new core22 release v0.0.5 for our ability to roll out extremely large scale relative binding free energy calculations in support of the COVID Moonshot.

This job shows extremely low PPD, less than 250000, while the average is around 1200000, up to 2000000 on good WUs. So average PPD is about five times as high than PPD for WU of this project. The GPU also shows very low power consumption and % of use, the fan is almost at idle.

My setup is a Ryzen 3700X running Debian Buster, equipped with two Radeon VII. From the log I can't find any hints.

As this seems to be experimental WUs for a new task, I wanted to let you know. If any additional data is required, please let me know.

[Joe_H]

Please post the complete WU description - Project, Run, Clone and Generation otherwise referred to as PRCG. From the posts about this group of projects by John Chodera, some Runs may perform differently compared to others in the same project.

[foldy]

If you want to get full load on your Radeon VII always then you can try to run 2 work units concurrently on each GPU. So you create 4 GPU slots where 1st and 2nd have same GPU index 0 and opencl index 0 and 3rd and 4th GPU slots have same GPU index 1 and OpenCL index 1

This could be a workaround until assignment servers have updated gpu species to only assign big work units with high atom count to AMD many shader GPUs like Radeon VII preferred.

[ThWuensche]

Browsing through the log I found my node processed already two of these WUs, the first one was in the night and therefor went by unrecognized. Here are the relevant parts from the log:

Code: Select all

First WU:

05:38:39:WU03:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:Vega 20 [Radeon VII] from 18.188.125.154
05:38:39:WU03:FS02:Connecting to 18.188.125.154:8080
05:38:40:WU03:FS02:Downloading 437.63KiB
05:38:41:WU03:FS02:Download complete
05:38:41:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13409 run:593 clone:8 gen:1 core:0x22 unit:0x0000000212bc7d9a5ed2f3d863279f7d
05:39:44:WU03:FS02:Starting
05:39:44:WU03:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 8190 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
05:39:44:WU03:FS02:Started FahCore on PID 15830
05:39:44:WU03:FS02:Core PID:15834
05:39:44:WU03:FS02:FahCore 0x22 started
05:39:45:WU03:FS02:0x22:*********************** Log Started 2020-06-02T05:39:44Z ***********************
05:39:45:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
05:39:45:WU03:FS02:0x22:       Type: 0x22
05:39:45:WU03:FS02:0x22:       Core: Core22
05:39:45:WU03:FS02:0x22:    Website: https://foldingathome.org/
05:39:45:WU03:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
05:39:45:WU03:FS02:0x22:     Author: John Chodera <[email protected]> and Rafal Wiewiora
05:39:45:WU03:FS02:0x22:             <[email protected]>
05:39:45:WU03:FS02:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 15830 -checkpoint 15
05:39:45:WU03:FS02:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
05:39:45:WU03:FS02:0x22:     Config: <none>
05:39:45:WU03:FS02:0x22:************************************ Build *************************************
05:39:45:WU03:FS02:0x22:    Version: 0.0.5
05:39:45:WU03:FS02:0x22:       Date: Apr 22 2020
05:39:45:WU03:FS02:0x22:       Time: 03:57:11
05:39:45:WU03:FS02:0x22: Repository: Git
05:39:45:WU03:FS02:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
05:39:45:WU03:FS02:0x22:     Branch: HEAD
05:39:45:WU03:FS02:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
05:39:45:WU03:FS02:0x22:    Options: -std=c++11 -O3 -funroll-loops
05:39:45:WU03:FS02:0x22:   Platform: linux2 4.19.76-linuxkit
05:39:45:WU03:FS02:0x22:       Bits: 64
05:39:45:WU03:FS02:0x22:       Mode: Release
05:39:45:WU03:FS02:0x22:************************************ System ************************************
05:39:45:WU03:FS02:0x22:        CPU: AMD Ryzen 7 3700X 8-Core Processor
05:39:45:WU03:FS02:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
05:39:45:WU03:FS02:0x22:       CPUs: 16
05:39:45:WU03:FS02:0x22:     Memory: 94.27GiB
05:39:45:WU03:FS02:0x22:Free Memory: 78.36GiB
05:39:45:WU03:FS02:0x22:    Threads: POSIX_THREADS
05:39:45:WU03:FS02:0x22: OS Version: 5.5
05:39:45:WU03:FS02:0x22:Has Battery: false
05:39:45:WU03:FS02:0x22: On Battery: false
05:39:45:WU03:FS02:0x22: UTC Offset: 2
05:39:45:WU03:FS02:0x22:        PID: 15834
05:39:45:WU03:FS02:0x22:        CWD: /var/lib/fahclient/work
05:39:45:WU03:FS02:0x22:         OS: Linux 5.5.0-0.bpo.2-amd64 x86_64
05:39:45:WU03:FS02:0x22:    OS Arch: AMD64
05:39:45:WU03:FS02:0x22:********************************************************************************
05:39:45:WU03:FS02:0x22:Project: 13409 (Run 593, Clone 8, Gen 1)
05:39:45:WU03:FS02:0x22:Unit: 0x0000000212bc7d9a5ed2f3d863279f7d
05:39:45:WU03:FS02:0x22:Reading tar file core.xml
05:39:45:WU03:FS02:0x22:Reading tar file integrator.xml
05:39:45:WU03:FS02:0x22:Reading tar file state.xml
05:39:45:WU03:FS02:0x22:Reading tar file system.xml
05:39:45:WU03:FS02:0x22:Digital signatures verified
05:39:45:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
05:39:45:WU03:FS02:0x22:Version 0.0.5
05:39:53:WU03:FS02:0x22:Completed 0 out of 1000000 steps (0%)
05:39:53:WU03:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:40:33:WU03:FS02:0x22:Completed 10000 out of 1000000 steps (1%)

......

06:45:40:WU03:FS02:0x22:Completed 1000000 out of 1000000 steps (100%)
06:45:40:WU03:FS02:0x22:Saving result file ../logfile_01.txt
06:45:40:WU03:FS02:0x22:Saving result file checkpointState.xml
06:45:40:WU03:FS02:0x22:Saving result file checkpt.crc
06:45:40:WU03:FS02:0x22:Saving result file globals.csv
06:45:40:WU03:FS02:0x22:Saving result file positions.xtc
06:45:40:WU03:FS02:0x22:Saving result file science.log
06:45:40:WU03:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
06:45:41:WU03:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:45:41:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:13409 run:593 clone:8 gen:1 core:0x22 unit:0x0000000212bc7d9a5ed2f3d863279f7d
06:45:41:WU03:FS02:Uploading 518.75KiB to 18.188.125.154
06:45:41:WU03:FS02:Connecting to 18.188.125.154:8080
06:45:42:WU03:FS02:Upload complete
06:45:42:WU03:FS02:Server responded WORK_ACK (400)
06:45:42:WU03:FS02:Final credit estimate, 14189.00 points
06:45:42:WU03:FS02:Cleaning up

Second WU:

11:18:47:WU03:FS02:Starting
11:18:47:WU03:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 8190 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
11:18:47:WU03:FS02:Started FahCore on PID 22742
11:18:47:WU03:FS02:Core PID:22746
11:18:47:WU03:FS02:FahCore 0x22 started
11:18:47:WU03:FS02:0x22:*********************** Log Started 2020-06-02T11:18:47Z ***********************
11:18:47:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
11:18:47:WU03:FS02:0x22:       Type: 0x22
11:18:47:WU03:FS02:0x22:       Core: Core22
11:18:47:WU03:FS02:0x22:    Website: https://foldingathome.org/
11:18:47:WU03:FS02:0x22:  Copyright: (c) 2009-2018 foldingathome.org
11:18:47:WU03:FS02:0x22:     Author: John Chodera <[email protected]> and Rafal Wiewiora
11:18:47:WU03:FS02:0x22:             <[email protected]>
11:18:47:WU03:FS02:0x22:       Args: -dir 03 -suffix 01 -version 706 -lifeline 22742 -checkpoint 15
11:18:47:WU03:FS02:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
11:18:47:WU03:FS02:0x22:     Config: <none>
11:18:47:WU03:FS02:0x22:************************************ Build *************************************
11:18:47:WU03:FS02:0x22:    Version: 0.0.5
11:18:47:WU03:FS02:0x22:       Date: Apr 22 2020
11:18:47:WU03:FS02:0x22:       Time: 03:57:11
11:18:47:WU03:FS02:0x22: Repository: Git
11:18:47:WU03:FS02:0x22:   Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
11:18:47:WU03:FS02:0x22:     Branch: HEAD
11:18:47:WU03:FS02:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
11:18:47:WU03:FS02:0x22:    Options: -std=c++11 -O3 -funroll-loops
11:18:47:WU03:FS02:0x22:   Platform: linux2 4.19.76-linuxkit
11:18:47:WU03:FS02:0x22:       Bits: 64
11:18:47:WU03:FS02:0x22:       Mode: Release
11:18:47:WU03:FS02:0x22:************************************ System ************************************
11:18:47:WU03:FS02:0x22:        CPU: AMD Ryzen 7 3700X 8-Core Processor
11:18:47:WU03:FS02:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:18:47:WU03:FS02:0x22:       CPUs: 16
11:18:47:WU03:FS02:0x22:     Memory: 94.27GiB
11:18:47:WU03:FS02:0x22:Free Memory: 77.06GiB
11:18:47:WU03:FS02:0x22:    Threads: POSIX_THREADS
11:18:47:WU03:FS02:0x22: OS Version: 5.5
11:18:47:WU03:FS02:0x22:Has Battery: false
11:18:47:WU03:FS02:0x22: On Battery: false
11:18:47:WU03:FS02:0x22: UTC Offset: 2
11:18:47:WU03:FS02:0x22:        PID: 22746
11:18:47:WU03:FS02:0x22:        CWD: /var/lib/fahclient/work
11:18:47:WU03:FS02:0x22:         OS: Linux 5.5.0-0.bpo.2-amd64 x86_64
11:18:47:WU03:FS02:0x22:    OS Arch: AMD64
11:18:47:WU03:FS02:0x22:********************************************************************************
11:18:47:WU03:FS02:0x22:Project: 13409 (Run 133, Clone 1, Gen 1)
11:18:47:WU03:FS02:0x22:Unit: 0x0000000512bc7d9a5ed2f3d9fa5ca7a2
11:18:47:WU03:FS02:0x22:Reading tar file core.xml
11:18:47:WU03:FS02:0x22:Reading tar file integrator.xml
11:18:47:WU03:FS02:0x22:Reading tar file state.xml
11:18:47:WU03:FS02:0x22:Reading tar file system.xml
11:18:47:WU03:FS02:0x22:Digital signatures verified
11:18:47:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
11:18:47:WU03:FS02:0x22:Version 0.0.5
11:18:57:WU03:FS02:0x22:Completed 0 out of 1000000 steps (0%)
11:18:57:WU03:FS02:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:19:38:WU03:FS02:0x22:Completed 10000 out of 1000000 steps (1%)

....

12:33:16:WU03:FS02:0x22:Completed 1000000 out of 1000000 steps (100%)
12:33:17:WU03:FS02:0x22:Saving result file ../logfile_01.txt
12:33:17:WU03:FS02:0x22:Saving result file checkpointState.xml
12:33:17:WU03:FS02:0x22:Saving result file checkpt.crc
12:33:17:WU03:FS02:0x22:Saving result file globals.csv
12:33:17:WU03:FS02:0x22:Saving result file positions.xtc
12:33:17:WU03:FS02:0x22:Saving result file science.log
12:33:17:WU03:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
12:33:17:WU03:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:33:17:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:13409 run:133 clone:1 gen:1 core:0x22 unit:0x0000000512bc7d9a5ed2f3d9fa5ca7a2
12:33:17:WU03:FS02:Uploading 519.69KiB to 18.188.125.154
12:33:18:WU03:FS02:Upload complete
12:33:18:WU03:FS02:Server responded WORK_ACK (400)
12:33:18:WU03:FS02:Final credit estimate, 13288.00 points

[ThWuensche]

Thanks for the hint with placing two WUs concurrently on one GPU. Will try to do that to give the most to the project.

[ThWuensche]

I tried the suggestion for two concurrent WUs on each GPU, however in that case all slots request Gromacs, type 0xa7 WUs, even though the slots are marked GPU. WUs which had already been started are dumped. So unfortunately this does not work, at least in my setup.

This is the log:

Code: Select all

*********************** Log Started 2020-06-02T21:38:00Z ***********************
21:38:00:Trying to access database...
21:38:00:Successfully acquired database lock
21:38:00:Read GPUs.txt
[91m21:38:00:ERROR:Exception: GPU index 0 already in use[0m
[93m21:38:00:WARNING:WU02:Slot ID 0 no longer exists, migrating to FS01[0m
[91m21:38:00:ERROR:Exception: Unit not found[0m
21:38:00:****************************** FAHClient ******************************
21:38:00:        Version: 7.6.13
21:38:00:         Author: Joseph Coffland <[email protected]>
21:38:00:      Copyright: 2020 foldingathome.org
21:38:00:       Homepage: https://foldingathome.org/
21:38:00:           Date: Apr 28 2020
21:38:00:           Time: 04:20:16
21:38:00:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
21:38:00:         Branch: master
21:38:00:       Compiler: GNU 8.3.0
21:38:00:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:38:00:                 -funroll-loops -fno-pie
21:38:00:       Platform: linux2 4.19.0-5-amd64
21:38:00:           Bits: 64
21:38:00:           Mode: Release
21:38:00:           Args: /etc/fahclient/config.xml
21:38:00:         Config: /etc/fahclient/config.xml
21:38:00:******************************** CBang ********************************
21:38:00:           Date: Apr 25 2020
21:38:00:           Time: 00:07:53
21:38:00:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
21:38:00:         Branch: master
21:38:00:       Compiler: GNU 8.3.0
21:38:00:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:38:00:                 -funroll-loops -fno-pie -fPIC
21:38:00:       Platform: linux2 4.19.0-5-amd64
21:38:00:           Bits: 64
21:38:00:           Mode: Release
21:38:00:******************************* System ********************************
21:38:00:            CPU: AMD Ryzen 7 3700X 8-Core Processor
21:38:00:         CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
21:38:00:           CPUs: 16
21:38:00:         Memory: 94.27GiB
21:38:00:    Free Memory: 77.25GiB
21:38:00:        Threads: POSIX_THREADS
21:38:00:     OS Version: 5.5
21:38:00:    Has Battery: false
21:38:00:     On Battery: false
21:38:00:     UTC Offset: 2
21:38:00:            PID: 4283
21:38:00:            CWD: /var/lib/fahclient
21:38:00:             OS: Linux 5.5.0-0.bpo.2-amd64 x86_64
21:38:00:        OS Arch: AMD64
21:38:00:           GPUs: 2
21:38:00:          GPU 0: Bus:6 Slot:0 Func:0 AMD:5 Vega 20 [Radeon VII]
21:38:00:          GPU 1: Bus:13 Slot:0 Func:0 AMD:5 Vega 20 [Radeon VII]
21:38:00:           CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
21:38:00:                 libcuda.so: cannot open shared object file: No such file or
21:38:00:                 directory
21:38:00:OpenCL Device 0: Platform:0 Device:0 Bus:6 Slot:0 Compute:2.0 Driver:3098.0
21:38:00:OpenCL Device 1: Platform:0 Device:1 Bus:13 Slot:0 Compute:2.0 Driver:3098.0
21:38:00:******************************* libFAH ********************************
21:38:00:           Date: Apr 15 2020
21:38:00:           Time: 21:43:24
21:38:00:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
21:38:00:         Branch: master
21:38:00:       Compiler: GNU 8.3.0
21:38:00:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
21:38:00:                 -funroll-loops -fno-pie
21:38:00:       Platform: linux2 4.19.0-5-amd64
21:38:00:           Bits: 64
21:38:00:           Mode: Release
21:38:00:***********************************************************************
21:38:00:<config>
21:38:00:  <!-- Client Control -->
21:38:00:  <fold-anon v='true'/>
21:38:00:
21:38:00:  <!-- Slot Control -->
21:38:00:  <power v='MEDIUM'/>
21:38:00:
21:38:00:  <!-- User Information -->
21:38:00:  <passkey v='*****'/>
21:38:00:  <team v='265730'/>
21:38:00:  <user v='tw@ems-wuensche'/>
21:38:00:
21:38:00:  <!-- Folding Slots -->
21:38:00:  <slot id='1' type='GPU'>
21:38:00:    <gpu-index v='0'/>
21:38:00:    <opencl-index v='0'/>
21:38:00:    <paused v='true'/>
21:38:00:  </slot>
21:38:00:  <slot id='2' type='GPU'>
21:38:00:    <gpu-index v='0'/>
21:38:00:    <opencl-index v='0'/>
21:38:00:    <paused v='true'/>
21:38:00:  </slot>
21:38:00:  <slot id='3' type='GPU'>
21:38:00:    <gpu-index v='1'/>
21:38:00:    <opencl-index v='1'/>
21:38:00:    <paused v='true'/>
21:38:00:  </slot>
21:38:00:  <slot id='4' type='GPU'>
21:38:00:    <gpu-index v='1'/>
21:38:00:    <opencl-index v='1'/>
21:38:00:    <paused v='true'/>
21:38:00:  </slot>
21:38:00:</config>
21:38:00:WU05:FS03:Starting
21:38:00:WU05:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 05 -suffix 01 -version 706 -lifeline 4283 -checkpoint 15 -gpu-vendor none -opencl-device 1 -gpu 1
21:38:00:WU05:FS03:Started FahCore on PID 4294
21:38:00:WU05:FS03:Core PID:4298
21:38:00:WU05:FS03:FahCore 0xa7 started
21:38:00:WU01:FS01:Starting
21:38:00:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 4283 -checkpoint 15 -gpu-vendor none -opencl-device 0 -gpu 0
21:38:00:WU01:FS01:Started FahCore on PID 4302
21:38:00:WU01:FS01:Core PID:4307
21:38:00:WU01:FS01:FahCore 0xa7 started
21:38:00:WU06:FS04:Starting
21:38:00:WU06:FS04:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 06 -suffix 01 -version 706 -lifeline 4283 -checkpoint 15 -gpu-vendor none -opencl-device 1 -gpu 1
21:38:00:WU06:FS04:Started FahCore on PID 4311
21:38:00:WU06:FS04:Core PID:4315
21:38:00:WU06:FS04:FahCore 0xa7 started
21:38:00:WU04:FS02:Starting
21:38:00:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 04 -suffix 01 -version 706 -lifeline 4283 -checkpoint 15 -gpu-vendor none -opencl-device 0 -gpu 0
21:38:00:WU04:FS02:Started FahCore on PID 4319
21:38:00:WU04:FS02:Core PID:4323
21:38:00:WU04:FS02:FahCore 0xa7 started
21:38:00:WU05:FS03:0xa7:*********************** Log Started 2020-06-02T21:38:00Z ***********************
21:38:00:WU05:FS03:0xa7:************************** Gromacs Folding@home Core ***************************
21:38:00:WU05:FS03:0xa7:       Type: 0xa7
21:38:00:WU05:FS03:0xa7:       Core: Gromacs
21:38:00:WU05:FS03:0xa7:       Args: -dir 05 -suffix 01 -version 706 -lifeline 4294 -checkpoint 15
21:38:00:WU05:FS03:0xa7:             -gpu-vendor none -opencl-device 1 -gpu 1
21:38:00:WU05:FS03:0xa7:************************************ CBang *************************************
21:38:00:WU05:FS03:0xa7:       Date: Nov 5 2019
21:38:00:WU05:FS03:0xa7:       Time: 06:06:57
21:38:00:WU05:FS03:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:38:00:WU05:FS03:0xa7:     Branch: master
21:38:00:WU05:FS03:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU05:FS03:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:38:00:WU05:FS03:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU05:FS03:0xa7:       Bits: 64
21:38:00:WU05:FS03:0xa7:       Mode: Release
21:38:00:WU05:FS03:0xa7:************************************ System ************************************
21:38:00:WU05:FS03:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
21:38:00:WU05:FS03:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
21:38:00:WU05:FS03:0xa7:       CPUs: 16
21:38:00:WU05:FS03:0xa7:     Memory: 94.27GiB
21:38:00:WU05:FS03:0xa7:Free Memory: 77.23GiB
21:38:00:WU05:FS03:0xa7:    Threads: POSIX_THREADS
21:38:00:WU05:FS03:0xa7: OS Version: 5.5
21:38:00:WU05:FS03:0xa7:Has Battery: false
21:38:00:WU05:FS03:0xa7: On Battery: false
21:38:00:WU05:FS03:0xa7: UTC Offset: 2
21:38:00:WU05:FS03:0xa7:        PID: 4298
21:38:00:WU05:FS03:0xa7:        CWD: /var/lib/fahclient/work
21:38:00:WU05:FS03:0xa7:******************************** Build - libFAH ********************************
21:38:00:WU05:FS03:0xa7:    Version: 0.0.18
21:38:00:WU05:FS03:0xa7:     Author: Joseph Coffland <[email protected]>
21:38:00:WU05:FS03:0xa7:  Copyright: 2019 foldingathome.org
21:38:00:WU05:FS03:0xa7:   Homepage: https://foldingathome.org/
21:38:00:WU05:FS03:0xa7:       Date: Nov 5 2019
21:38:00:WU05:FS03:0xa7:       Time: 06:13:26
21:38:00:WU05:FS03:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:38:00:WU05:FS03:0xa7:     Branch: master
21:38:00:WU05:FS03:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU05:FS03:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:38:00:WU05:FS03:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU05:FS03:0xa7:       Bits: 64
21:38:00:WU05:FS03:0xa7:       Mode: Release
21:38:00:WU05:FS03:0xa7:************************************ Build *************************************
21:38:00:WU05:FS03:0xa7:       SIMD: avx_256
21:38:00:WU05:FS03:0xa7:********************************************************************************
21:38:00:WU05:FS03:0xa7:Project: 14258 (Run 0, Clone 625, Gen 53)
21:38:00:WU05:FS03:0xa7:Unit: 0x00000043cedfaa925eac9e5e7c1a57a5
21:38:00:WU05:FS03:0xa7:Digital signatures verified
21:38:00:WU05:FS03:0xa7:Calling: mdrun -s frame53.tpr -o frame53.trr -x frame53.xtc -cpi state.cpt -cpt 15 -nt 1
21:38:00:WU05:FS03:0xa7:Steps: first=13250000 total=250000
21:38:00:WU05:FS03:0xa7:Completed 2412 out of 250000 steps (0%)
21:38:00:WU01:FS01:0xa7:*********************** Log Started 2020-06-02T21:38:00Z ***********************
21:38:00:WU01:FS01:0xa7:************************** Gromacs Folding@home Core ***************************
21:38:00:WU01:FS01:0xa7:       Type: 0xa7
21:38:00:WU01:FS01:0xa7:       Core: Gromacs
21:38:00:WU01:FS01:0xa7:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4302 -checkpoint 15
21:38:00:WU01:FS01:0xa7:             -gpu-vendor none -opencl-device 0 -gpu 0
21:38:00:WU01:FS01:0xa7:************************************ CBang *************************************
21:38:00:WU01:FS01:0xa7:       Date: Nov 5 2019
21:38:00:WU01:FS01:0xa7:       Time: 06:06:57
21:38:00:WU01:FS01:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:38:00:WU01:FS01:0xa7:     Branch: master
21:38:00:WU01:FS01:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU01:FS01:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:38:00:WU01:FS01:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU01:FS01:0xa7:       Bits: 64
21:38:00:WU01:FS01:0xa7:       Mode: Release
21:38:00:WU01:FS01:0xa7:************************************ System ************************************
21:38:00:WU01:FS01:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
21:38:00:WU01:FS01:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
21:38:00:WU01:FS01:0xa7:       CPUs: 16
21:38:00:WU01:FS01:0xa7:     Memory: 94.27GiB
21:38:00:WU01:FS01:0xa7:Free Memory: 77.21GiB
21:38:00:WU01:FS01:0xa7:    Threads: POSIX_THREADS
21:38:00:WU01:FS01:0xa7: OS Version: 5.5
21:38:00:WU01:FS01:0xa7:Has Battery: false
21:38:00:WU01:FS01:0xa7: On Battery: false
21:38:00:WU01:FS01:0xa7: UTC Offset: 2
21:38:00:WU01:FS01:0xa7:        PID: 4307
21:38:00:WU01:FS01:0xa7:        CWD: /var/lib/fahclient/work
21:38:00:WU01:FS01:0xa7:******************************** Build - libFAH ********************************
21:38:00:WU01:FS01:0xa7:    Version: 0.0.18
21:38:00:WU01:FS01:0xa7:     Author: Joseph Coffland <[email protected]>
21:38:00:WU01:FS01:0xa7:  Copyright: 2019 foldingathome.org
21:38:00:WU01:FS01:0xa7:   Homepage: https://foldingathome.org/
21:38:00:WU01:FS01:0xa7:       Date: Nov 5 2019
21:38:00:WU01:FS01:0xa7:       Time: 06:13:26
21:38:00:WU01:FS01:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:38:00:WU01:FS01:0xa7:     Branch: master
21:38:00:WU01:FS01:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU01:FS01:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:38:00:WU01:FS01:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU01:FS01:0xa7:       Bits: 64
21:38:00:WU01:FS01:0xa7:       Mode: Release
21:38:00:WU01:FS01:0xa7:************************************ Build *************************************
21:38:00:WU01:FS01:0xa7:       SIMD: avx_256
21:38:00:WU01:FS01:0xa7:********************************************************************************
21:38:00:WU01:FS01:0xa7:Project: 13851 (Run 0, Clone 2635, Gen 173)
21:38:00:WU01:FS01:0xa7:Unit: 0x000000cf287234c95e72586d763dc879
21:38:00:WU01:FS01:0xa7:Digital signatures verified
21:38:00:WU01:FS01:0xa7:Calling: mdrun -s frame173.tpr -o frame173.trr -x frame173.xtc -e frame173.edr -cpi state.cpt -cpt 15 -nt 1
21:38:00:WU01:FS01:0xa7:Steps: first=86500000 total=500000
21:38:00:WU01:FS01:0xa7:Completed 742 out of 500000 steps (0%)
21:38:00:WU06:FS04:0xa7:*********************** Log Started 2020-06-02T21:38:00Z ***********************
21:38:00:WU06:FS04:0xa7:************************** Gromacs Folding@home Core ***************************
21:38:00:WU06:FS04:0xa7:       Type: 0xa7
21:38:00:WU06:FS04:0xa7:       Core: Gromacs
21:38:00:WU06:FS04:0xa7:       Args: -dir 06 -suffix 01 -version 706 -lifeline 4311 -checkpoint 15
21:38:00:WU06:FS04:0xa7:             -gpu-vendor none -opencl-device 1 -gpu 1
21:38:00:WU06:FS04:0xa7:************************************ CBang *************************************
21:38:00:WU06:FS04:0xa7:       Date: Nov 5 2019
21:38:00:WU06:FS04:0xa7:       Time: 06:06:57
21:38:00:WU06:FS04:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:38:00:WU06:FS04:0xa7:     Branch: master
21:38:00:WU06:FS04:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU06:FS04:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:38:00:WU06:FS04:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU06:FS04:0xa7:       Bits: 64
21:38:00:WU06:FS04:0xa7:       Mode: Release
21:38:00:WU06:FS04:0xa7:************************************ System ************************************
21:38:00:WU06:FS04:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
21:38:00:WU06:FS04:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
21:38:00:WU06:FS04:0xa7:       CPUs: 16
21:38:00:WU06:FS04:0xa7:     Memory: 94.27GiB
21:38:00:WU06:FS04:0xa7:Free Memory: 77.21GiB
21:38:00:WU06:FS04:0xa7:    Threads: POSIX_THREADS
21:38:00:WU06:FS04:0xa7: OS Version: 5.5
21:38:00:WU06:FS04:0xa7:Has Battery: false
21:38:00:WU06:FS04:0xa7: On Battery: false
21:38:00:WU06:FS04:0xa7: UTC Offset: 2
21:38:00:WU06:FS04:0xa7:        PID: 4315
21:38:00:WU06:FS04:0xa7:        CWD: /var/lib/fahclient/work
21:38:00:WU06:FS04:0xa7:******************************** Build - libFAH ********************************
21:38:00:WU06:FS04:0xa7:    Version: 0.0.18
21:38:00:WU06:FS04:0xa7:     Author: Joseph Coffland <[email protected]>
21:38:00:WU06:FS04:0xa7:  Copyright: 2019 foldingathome.org
21:38:00:WU06:FS04:0xa7:   Homepage: https://foldingathome.org/
21:38:00:WU06:FS04:0xa7:       Date: Nov 5 2019
21:38:00:WU06:FS04:0xa7:       Time: 06:13:26
21:38:00:WU06:FS04:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:38:00:WU06:FS04:0xa7:     Branch: master
21:38:00:WU06:FS04:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU06:FS04:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:38:00:WU06:FS04:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU06:FS04:0xa7:       Bits: 64
21:38:00:WU06:FS04:0xa7:       Mode: Release
21:38:00:WU06:FS04:0xa7:************************************ Build *************************************
21:38:00:WU06:FS04:0xa7:       SIMD: avx_256
21:38:00:WU06:FS04:0xa7:********************************************************************************
21:38:00:WU06:FS04:0xa7:Project: 14719 (Run 1690, Clone 1, Gen 41)
21:38:00:WU06:FS04:0xa7:Unit: 0x0000002d2879986c5ea965f74bd9d362
21:38:00:WU06:FS04:0xa7:Digital signatures verified
21:38:00:WU06:FS04:0xa7:Calling: mdrun -s frame41.tpr -o frame41.trr -cpi state.cpt -cpt 15 -nt 1
21:38:00:WU06:FS04:0xa7:Steps: first=0 total=250000
21:38:00:WU06:FS04:0xa7:Completed 416 out of 250000 steps (0%)
21:38:00:WU04:FS02:0xa7:*********************** Log Started 2020-06-02T21:38:00Z ***********************
21:38:00:WU04:FS02:0xa7:************************** Gromacs Folding@home Core ***************************
21:38:00:WU04:FS02:0xa7:       Type: 0xa7
21:38:00:WU04:FS02:0xa7:       Core: Gromacs
21:38:00:WU04:FS02:0xa7:       Args: -dir 04 -suffix 01 -version 706 -lifeline 4319 -checkpoint 15
21:38:00:WU04:FS02:0xa7:             -gpu-vendor none -opencl-device 0 -gpu 0
21:38:00:WU04:FS02:0xa7:************************************ CBang *************************************
21:38:00:WU04:FS02:0xa7:       Date: Nov 5 2019
21:38:00:WU04:FS02:0xa7:       Time: 06:06:57
21:38:00:WU04:FS02:0xa7:   Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
21:38:00:WU04:FS02:0xa7:     Branch: master
21:38:00:WU04:FS02:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU04:FS02:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
21:38:00:WU04:FS02:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU04:FS02:0xa7:       Bits: 64
21:38:00:WU04:FS02:0xa7:       Mode: Release
21:38:00:WU04:FS02:0xa7:************************************ System ************************************
21:38:00:WU04:FS02:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
21:38:00:WU04:FS02:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
21:38:00:WU04:FS02:0xa7:       CPUs: 16
21:38:00:WU04:FS02:0xa7:     Memory: 94.27GiB
21:38:00:WU04:FS02:0xa7:Free Memory: 77.20GiB
21:38:00:WU04:FS02:0xa7:    Threads: POSIX_THREADS
21:38:00:WU04:FS02:0xa7: OS Version: 5.5
21:38:00:WU04:FS02:0xa7:Has Battery: false
21:38:00:WU04:FS02:0xa7: On Battery: false
21:38:00:WU04:FS02:0xa7: UTC Offset: 2
21:38:00:WU04:FS02:0xa7:        PID: 4323
21:38:00:WU04:FS02:0xa7:        CWD: /var/lib/fahclient/work
21:38:00:WU04:FS02:0xa7:******************************** Build - libFAH ********************************
21:38:00:WU04:FS02:0xa7:    Version: 0.0.18
21:38:00:WU04:FS02:0xa7:     Author: Joseph Coffland <[email protected]>
21:38:00:WU04:FS02:0xa7:  Copyright: 2019 foldingathome.org
21:38:00:WU04:FS02:0xa7:   Homepage: https://foldingathome.org/
21:38:00:WU04:FS02:0xa7:       Date: Nov 5 2019
21:38:00:WU04:FS02:0xa7:       Time: 06:13:26
21:38:00:WU04:FS02:0xa7:   Revision: 490c9aa2957b725af319379424d5c5cb36efb656
21:38:00:WU04:FS02:0xa7:     Branch: master
21:38:00:WU04:FS02:0xa7:   Compiler: GNU 8.3.0
21:38:00:WU04:FS02:0xa7:    Options: -std=c++11 -O3 -funroll-loops -fno-pie
21:38:00:WU04:FS02:0xa7:   Platform: linux2 4.19.0-5-amd64
21:38:00:WU04:FS02:0xa7:       Bits: 64
21:38:00:WU04:FS02:0xa7:       Mode: Release
21:38:00:WU04:FS02:0xa7:************************************ Build *************************************
21:38:00:WU04:FS02:0xa7:       SIMD: avx_256
21:38:00:WU04:FS02:0xa7:********************************************************************************
21:38:00:WU04:FS02:0xa7:Project: 14217 (Run 1612, Clone 0, Gen 53)
21:38:00:WU04:FS02:0xa7:Unit: 0x00000041cedfaa925eab715417daee7e
21:38:00:WU04:FS02:0xa7:Digital signatures verified
21:38:00:WU04:FS02:0xa7:Calling: mdrun -s frame53.tpr -o frame53.trr -x frame53.xtc -cpi state.cpt -cpt 15 -nt 1
21:38:00:WU04:FS02:0xa7:Steps: first=3312500 total=62500
21:38:03:WU04:FS02:0xa7:Completed 262 out of 62500 steps (0%)
21:38:07:WU05:FS03:0xa7:Completed 2500 out of 250000 steps (1%)

[ThWuensche]

That's my GPU usage, now with two WUs from project 13408. FAH leaves about 70%-80% of the capacity unused, which is a pity for

extremely large scale relative binding free energy calculations

Code: Select all

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    76.0c  39.0W   1546Mhz  800Mhz  22.75%  auto  250.0W   13%   4%    
1    54.0c  40.0W   700Mhz   350Mhz  22.75%  auto  250.0W    1%   45%   
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    78.0c  36.0W   1546Mhz  800Mhz  22.75%  auto  250.0W   13%   59%   
1    55.0c  34.0W   1546Mhz  800Mhz  23.92%  auto  250.0W    1%   76%   
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK    MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    78.0c  29.0W   700Mhz  800Mhz  22.75%  auto  250.0W   13%   0%    
1    55.0c  23.0W   700Mhz  350Mhz  23.92%  auto  250.0W    1%   1%    
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK    MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    78.0c  29.0W   700Mhz  800Mhz  24.71%  auto  250.0W   13%   0%    
1    54.0c  32.0W   700Mhz  350Mhz  26.67%  auto  250.0W    1%   46%   
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK     Fan     Perf  PwrCap  VRAM%  GPU%  
0    76.0c  163.0W  1801Mhz  1000Mhz  23.92%  auto  250.0W   13%   89%   
1    55.0c  81.0W   1801Mhz  1000Mhz  22.75%  auto  250.0W    1%   66%   
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK    MCLK     Fan     Perf  PwrCap  VRAM%  GPU%  
0    75.0c  46.0W   700Mhz  1000Mhz  23.92%  auto  250.0W   13%   62%   
1    54.0c  19.0W   700Mhz  800Mhz   22.75%  auto  250.0W    1%   0%    
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    75.0c  39.0W   1546Mhz  800Mhz  22.75%  auto  250.0W   13%   4%    
1    53.0c  21.0W   1546Mhz  350Mhz  22.75%  auto  250.0W    1%   11%   
================================================================================
==============================End of ROCm SMI Log ==============================

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    76.0c  39.0W   1546Mhz  800Mhz  22.75%  auto  250.0W   13%   19%   
1    54.0c  20.0W   1546Mhz  800Mhz  22.75%  auto  250.0W    1%   0%    
================================================================================
==============================End of ROCm SMI Log ==============================

Here is data for the WUs:

Code: Select all

16:51:39:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:13408 run:596 clone:143 gen:1 core:0x22 unit:0x0000000112bc7d9a5ed4a8b94e7d1a01
16:51:39:WU02:FS01:0x22:Project: 13408 (Run 283, Clone 163, Gen 0)
18:13:57:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13408 run:622 clone:165 gen:0 core:0x22 unit:0x0000000312bc7d9a5ed4a8b7f2a02870
18:15:27:WU03:FS02:0x22:Project: 13408 (Run 622, Clone 165, Gen 0)

On the other hand there seem to be WUs with rather high PPD from the same project.

[Crawdaddy79]

Hey - I accidentally noticed the same thing on my Vega 64.

My GPU didn't even get above 100W when I'm used to 200W+. Also the GPU typically doesn't get less than 800k PPD - usually 1.0 - 1.2M PPD.



The saving grace is that it's a quick return.

[ThWuensche]

I think making best use of the available hardware is in the interest of the project. Just a few thoughts to help with the issue, unfortunately contribution of real analysis is not possible, as the source is not available. So instead of an analysis I can contribute only guesses:

First possible reason might be that the problem does not scale well to highly parallel architectures, parts of code with parallelization are intermixed with and dependent on parts which can not be parallelized, that way limiting the overall performance by those parts which can not be executed in parallel. This probably would be hard to circumvent unless the algorithm may be changed. Only help in that case would be to run WUs in parallel on a GPU.

Second possible reason might be intensive sequential interaction between CPU and GPU, which could lead to latencies effectively blocking GPU performance. I don't know whether such interaction during processing of a WU takes parts or whether the jobs are once loaded to the GPU in the beginning and then processed to the end without further interaction with the CPU. Mention of the requirement to have free CPU ressources not to slow down GPU processing makes me expect that interactions exist. In this case maybe more or bigger chunks could be loaded to the GPU in one run? I also have the impression that the tasks with low GPU usage also have rather low GPU memory usage.

BTW the figures mentioned by crawdaddy for average PPD look identical to what I see, while the Radeon VII should be somewhat faster than the Vega64. This might indicate that the limitation comes from something else, not the GPU.

[PantherX]

F@H uses OpenMM which is open source so you can look into that code base to see how it operates: https://github.com/openmm/openmm

Also, there's a new version of FahCore_22 in development that may or may not address this issue. There's no ETA on when that FahCore_22 will be released.

[ThWuensche]

@PantherX

Thank you for the hint to openMM. So I assume how work is distributed between CPU and GPU in core22 is taken from openMM as available in git and not changed in core22?

My feedback was meant as test result for the new version of core22, to help find weaknesses before general rollout. The WUs had indicated:

"Testing new core22 release v0.0.5"

so I guess it is the new version you mention.

[PantherX]

FahCore_22 uses the code from OpenMM and does additional stuff to make it useful for F@H. However, any optimizations and features would be taken from OpenMM if they are useful for the researchers in F@H.

There's a new version of FahCore_22 version 0.0.6 which is under development. There's no ETA except that it will be soon (see my signature above to understand how long that is).

[BobWilliams757]

I think the complexity of these WU's make it next to impossible for the researchers to really account for all the various hardware running them.

In this case, at least two folders with highish end GPU's have low PPD returns with this WU. But with my meager Vega 11 onboard graphics, it returned about 80% higher than average PPD. It seems that small atom count WU's as a trend run really great on this little chip, maybe due to the way the memory allocation can change.


Regardless of the hardware, I think there will be WU's that here and there it either struggles with, or excels at. But in the end it should balance out, and if they poured too much time into how they are allocated it would just slow the science down.




But look on the positive side. The variance you guys with the good GPUs are seeing is still by far more points than my little GPU will fold in a day. Even a good day, with a higher than average PPD return WU.

[Crawdaddy79]

It doesn't bother me much - as I said it's a quick turnaround at about an hour fifteen.

Just got two in a row though -for the 2nd one, I paused the CPU and after several percent, TPF went from 51.0 sec to 43.0 sec; I took this screenshot showing nearly 275k PPD SCREENSHOT

A far cry from what I'm used to, but meh. I have other issues I'm dealing with that are affecting my points to a larger degree.

[ThWuensche]

As for the PPD, I don't care much. What I care about is the volume of calculations required to support scientific results. I just bought another PC and two more Radeon VII to support the scientific results (not the PPD). As I understand the support of project moonshot will need a lot of computational power and it's sad if big part of it stays unused (for the electricity bill it's good, but ...)

What I conclude from the observations from Crawdaddy and BobWilliams supports the second of the reasons mentioned above, latency in interaction with the CPU. For a low end GPU, the GPU is limiting and not the interaction with the CPU, for a high end GPU, the interaction with the CPU becomes limiting. Thus latency would be a big show-stopper for high-end GPUs, but mostly go by unnoticed on smaller GPUs. The latency issue seems also confirmed from Crawdaddy's test with stopped CPU folding. The low memory footprint I observed is good for GPUs with limited memory and memory bandwidth, but leaves the benefit of the 16G fast RAM on the Radeon VII unused. So bigger work packages and less interaction might help to make good use of high-end GPUs (if that is possible from the nature of the WUs). Don't know about behaviour of the bigger NVidia GPUs in this situation, any ideas?