GPU WU for 14415 causes crashes
Posted: Sat Apr 11, 2020 10:56 pm
I've been folding successfully with this machine for several weeks. This morning I woke up to a garbled display and a no longer running FAHClient. I am running a Ryzen 2600X with an RX Vega 56 both at stock clocks on Arch Linux, using the AMDGPU-PRO OpenCL driver (packaged as opencl-amd in the arch repositories), with kernel 5.4.31. Within seconds after starting FAHClient, the gpu crashes and my displays show only garbage, and FAHClient is killed.
Is there anything I can do to troubleshoot this? Should I just dump the WU?
FAH log
Core log
clinfo output
Is there anything I can do to troubleshoot this? Should I just dump the WU?
FAH log
Code: Select all
*********************** Log Started 2020-04-11T22:03:17Z ***********************
22:03:17:************************* Folding@home Client *************************
22:03:17: Website: https://foldingathome.org/
22:03:17: Copyright: (c) 2009-2018 foldingathome.org
22:03:17: Author: Joseph Coffland <[email protected]>
22:03:17: Args:
22:03:17: Config: /home/sanae/config.xml
22:03:17:******************************** Build ********************************
22:03:17: Version: 7.5.1
22:03:17: Date: May 11 2018
22:03:17: Time: 19:59:04
22:03:17: Repository: Git
22:03:17: Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
22:03:17: Branch: master
22:03:17: Compiler: GNU 6.3.0 20170516
22:03:17: Options: -std=gnu++98 -O3 -funroll-loops
22:03:17: Platform: linux2 4.14.0-3-amd64
22:03:17: Bits: 64
22:03:17: Mode: Release
22:03:17:******************************* System ********************************
22:03:17: CPU: AMD Ryzen 5 2600X Six-Core Processor
22:03:17: CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
22:03:17: CPUs: 12
22:03:17: Memory: 15.57GiB
22:03:17: Free Memory: 5.17GiB
22:03:17: Threads: POSIX_THREADS
22:03:17: OS Version: 5.4
22:03:17: Has Battery: false
22:03:17: On Battery: false
22:03:17: UTC Offset: -7
22:03:17: PID: 12969
22:03:17: CWD: /home/sanae
22:03:17: OS: Linux 5.4.31-0-lts x86_64
22:03:17: OS Arch: AMD64
22:03:17: GPUs: 1
22:03:17: GPU 0: Bus:11 Slot:0 Func:0 AMD:5 Vega 10 XL/XT [Radeon RX Vega 56/64]
22:03:17: CUDA: Not detected: Failed to open dynamic library 'libcuda.so':
22:03:17: libcuda.so: cannot open shared object file: No such file or
22:03:17: directory
22:03:17:OpenCL Device 0: Platform:0 Device:0 Bus:11 Slot:0 Compute:2.0 Driver:3004.6
22:03:17:***********************************************************************
22:03:17:<config>
22:03:17: <!-- Network -->
22:03:17: <proxy v=':8080'/>
22:03:17:
22:03:17: <!-- Slot Control -->
22:03:17: <power v='full'/>
22:03:17:
22:03:17: <!-- User Information -->
22:03:17: <passkey v='********************************'/>
22:03:17: <team v='236565'/>
22:03:17: <user v='ky0ko'/>
22:03:17:
22:03:17: <!-- Folding Slots -->
22:03:17: <slot id='0' type='CPU'/>
22:03:17: <slot id='1' type='GPU'/>
22:03:17:</config>
22:03:17:Trying to access database...
22:03:17:Successfully acquired database lock
22:03:17:Enabled folding slot 00: READY cpu:11
22:03:17:Enabled folding slot 01: READY gpu:0:Vega 10 XL/XT [Radeon RX Vega 56/64]
22:03:17:WU00:FS01:Starting
22:03:17:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /home/sanae/cores/cores.foldingathome.org/v7/lin/64bit/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 705 -lifeline 12969 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
22:03:17:WU00:FS01:Started FahCore on PID 12993
22:03:17:WU00:FS01:Core PID:12997
22:03:17:WU00:FS01:FahCore 0x22 started
22:03:17:WU01:FS00:Starting
22:03:17:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /home/sanae/cores/cores.foldingathome.org/v7/lin/64bit/avx/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 705 -lifeline 12969 -checkpoint 15 -np 11
22:03:17:WU01:FS00:Started FahCore on PID 13000
22:03:17:WU01:FS00:Core PID:13004
22:03:17:WU01:FS00:FahCore 0xa7 started
22:03:17:WU00:FS01:0x22:*********************** Log Started 2020-04-11T22:03:17Z ***********************
22:03:17:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:03:17:WU00:FS01:0x22: Type: 0x22
22:03:17:WU00:FS01:0x22: Core: Core22
22:03:17:WU00:FS01:0x22: Website: https://foldingathome.org/
22:03:17:WU00:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
22:03:17:WU00:FS01:0x22: Author: John Chodera <[email protected]> and Rafal Wiewiora
22:03:17:WU00:FS01:0x22: <[email protected]>
22:03:17:WU00:FS01:0x22: Args: -dir 00 -suffix 01 -version 705 -lifeline 12993 -checkpoint 15
22:03:17:WU00:FS01:0x22: -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
22:03:17:WU00:FS01:0x22: Config: <none>
22:03:17:WU00:FS01:0x22:************************************ Build *************************************
22:03:17:WU00:FS01:0x22: Version: 0.0.2
22:03:17:WU00:FS01:0x22: Date: Dec 6 2019
22:03:17:WU00:FS01:0x22: Time: 21:20:17
22:03:17:WU00:FS01:0x22: Repository: Git
22:03:17:WU00:FS01:0x22: Revision: f87d92b58abdf7e6bf2e173cfbc4dc3e837c7042
22:03:17:WU00:FS01:0x22: Branch: core22
22:03:17:WU00:FS01:0x22: Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
22:03:17:WU00:FS01:0x22: Options: -std=gnu++98 -O3 -funroll-loops
22:03:17:WU00:FS01:0x22: Platform: linux2 4.9.87-linuxkit-aufs
22:03:17:WU00:FS01:0x22: Bits: 64
22:03:17:WU00:FS01:0x22: Mode: Release
22:03:17:WU00:FS01:0x22:************************************ System ************************************
22:03:17:WU00:FS01:0x22: CPU: AMD Ryzen 5 2600X Six-Core Processor
22:03:17:WU00:FS01:0x22: CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
22:03:17:WU00:FS01:0x22: CPUs: 12
22:03:17:WU00:FS01:0x22: Memory: 15.57GiB
22:03:17:WU00:FS01:0x22:Free Memory: 5.11GiB
22:03:17:WU00:FS01:0x22: Threads: POSIX_THREADS
22:03:17:WU00:FS01:0x22: OS Version: 5.4
22:03:17:WU00:FS01:0x22:Has Battery: false
22:03:17:WU00:FS01:0x22: On Battery: false
22:03:17:WU00:FS01:0x22: UTC Offset: -7
22:03:17:WU00:FS01:0x22: PID: 12997
22:03:17:WU00:FS01:0x22: CWD: /home/sanae/work
22:03:17:WU00:FS01:0x22: OS: Linux 5.4.31-0-lts x86_64
22:03:17:WU00:FS01:0x22: OS Arch: AMD64
22:03:17:WU00:FS01:0x22:********************************************************************************
22:03:17:WU00:FS01:0x22:Project: 14415 (Run 0, Clone 719, Gen 6)
22:03:17:WU00:FS01:0x22:Unit: 0x000000080d5262775e839e5ce79f8bf0
22:03:17:WU00:FS01:0x22:Reading tar file core.xml
22:03:17:WU00:FS01:0x22:Reading tar file integrator.xml
22:03:17:WU00:FS01:0x22:Reading tar file state.xml
22:03:17:WU00:FS01:0x22:Reading tar file system.xml
22:03:17:WU01:FS00:0xa7:*********************** Log Started 2020-04-11T22:03:17Z ***********************
22:03:17:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
22:03:17:WU01:FS00:0xa7: Type: 0xa7
22:03:17:WU01:FS00:0xa7: Core: Gromacs
22:03:17:WU01:FS00:0xa7: Args: -dir 01 -suffix 01 -version 705 -lifeline 13000 -checkpoint 15 -np
22:03:17:WU01:FS00:0xa7: 11
22:03:17:WU01:FS00:0xa7:************************************ CBang *************************************
22:03:17:WU01:FS00:0xa7: Date: Nov 5 2019
22:03:17:WU01:FS00:0xa7: Time: 06:06:57
22:03:17:WU01:FS00:0xa7: Revision: 46c96f1aa8419571d83f3e63f9c99a0d602f6da9
22:03:17:WU01:FS00:0xa7: Branch: master
22:03:17:WU01:FS00:0xa7: Compiler: GNU 8.3.0
22:03:17:WU01:FS00:0xa7: Options: -std=c++11 -O3 -funroll-loops -fno-pie -fPIC
22:03:17:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
22:03:17:WU01:FS00:0xa7: Bits: 64
22:03:17:WU01:FS00:0xa7: Mode: Release
22:03:17:WU01:FS00:0xa7:************************************ System ************************************
22:03:17:WU01:FS00:0xa7: CPU: AMD Ryzen 5 2600X Six-Core Processor
22:03:17:WU01:FS00:0xa7: CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
22:03:17:WU01:FS00:0xa7: CPUs: 12
22:03:17:WU01:FS00:0xa7: Memory: 15.57GiB
22:03:17:WU01:FS00:0xa7:Free Memory: 5.07GiB
22:03:17:WU01:FS00:0xa7: Threads: POSIX_THREADS
22:03:17:WU01:FS00:0xa7: OS Version: 5.4
22:03:17:WU01:FS00:0xa7:Has Battery: false
22:03:17:WU01:FS00:0xa7: On Battery: false
22:03:17:WU01:FS00:0xa7: UTC Offset: -7
22:03:17:WU01:FS00:0xa7: PID: 13004
22:03:17:WU01:FS00:0xa7: CWD: /home/sanae/work
22:03:17:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
22:03:17:WU01:FS00:0xa7: Version: 0.0.18
22:03:17:WU01:FS00:0xa7: Author: Joseph Coffland <[email protected]>
22:03:17:WU01:FS00:0xa7: Copyright: 2019 foldingathome.org
22:03:17:WU01:FS00:0xa7: Homepage: https://foldingathome.org/
22:03:17:WU01:FS00:0xa7: Date: Nov 5 2019
22:03:17:WU01:FS00:0xa7: Time: 06:13:26
22:03:17:WU01:FS00:0xa7: Revision: 490c9aa2957b725af319379424d5c5cb36efb656
22:03:17:WU01:FS00:0xa7: Branch: master
22:03:17:WU01:FS00:0xa7: Compiler: GNU 8.3.0
22:03:17:WU01:FS00:0xa7: Options: -std=c++11 -O3 -funroll-loops -fno-pie
22:03:17:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
22:03:17:WU01:FS00:0xa7: Bits: 64
22:03:17:WU01:FS00:0xa7: Mode: Release
22:03:17:WU01:FS00:0xa7:************************************ Build *************************************
22:03:17:WU01:FS00:0xa7: SIMD: avx_256
22:03:17:WU01:FS00:0xa7:********************************************************************************
22:03:17:WU01:FS00:0xa7:Project: 14625 (Run 1271, Clone 4, Gen 5)
22:03:17:WU01:FS00:0xa7:Unit: 0x000000059bf7a4d65e88df0038db82b5
22:03:17:WU01:FS00:0xa7:Digital signatures verified
22:03:17:WU01:FS00:0xa7:Reducing thread count from 11 to 10 to avoid domain decomposition by a prime number > 3
22:03:17:WU01:FS00:0xa7:Calling: mdrun -s frame5.tpr -o frame5.trr -cpi state.cpt -cpt 15 -nt 10
22:03:17:WU01:FS00:0xa7:Steps: first=0 total=250000
22:03:18:WU00:FS01:0x22:Digital signatures verified
22:03:18:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:03:18:WU00:FS01:0x22:Version 0.0.2
22:03:18:WU01:FS00:0xa7:Completed 30119 out of 250000 steps (12%)
22:04:11:WU01:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
22:04:24:Caught signal SIGTERM(15) on PID 12969
22:04:24:Exiting, please wait. . .
22:04:24:WU00:FS01:0x22:Caught signal SIGTERM(15) on PID 12997
22:04:24:WU00:FS01:0x22:Exiting, please wait. . .
22:04:24:WU01:FS00:0xa7:Caught signal SIGTERM(15) on PID 13004
22:04:24:WU01:FS00:0xa7:Exiting, please wait. . .
22:04:24:WU01:FS00:0xa7:Folding@home Core Shutdown: INTERRUPTED
Code: Select all
*********************** Log Started 2020-04-11T22:03:17Z ***********************
*************************** Core22 Folding@home Core ***************************
Type: 0x22
Core: Core22
Website: https://foldingathome.org/
Copyright: (c) 2009-2018 foldingathome.org
Author: John Chodera <[email protected]> and Rafal Wiewiora
<[email protected]>
Args: -dir 00 -suffix 01 -version 705 -lifeline 12993 -checkpoint 15
-gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
Config: <none>
************************************ Build *************************************
Version: 0.0.2
Date: Dec 6 2019
Time: 21:20:17
Repository: Git
Revision: f87d92b58abdf7e6bf2e173cfbc4dc3e837c7042
Branch: core22
Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
Options: -std=gnu++98 -O3 -funroll-loops
Platform: linux2 4.9.87-linuxkit-aufs
Bits: 64
Mode: Release
************************************ System ************************************
CPU: AMD Ryzen 5 2600X Six-Core Processor
CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
CPUs: 12
Memory: 15.57GiB
Free Memory: 5.11GiB
Threads: POSIX_THREADS
OS Version: 5.4
Has Battery: false
On Battery: false
UTC Offset: -7
PID: 12997
CWD: /home/sanae/work
OS: Linux 5.4.31-0-lts x86_64
OS Arch: AMD64
********************************************************************************
Project: 14415 (Run 0, Clone 719, Gen 6)
Unit: 0x000000080d5262775e839e5ce79f8bf0
Reading tar file core.xml
Reading tar file integrator.xml
Reading tar file state.xml
Reading tar file system.xml
Digital signatures verified
Folding@home GPU Core22 Folding@home Core
Version 0.0.2
Caught signal SIGTERM(15) on PID 12997
Exiting, please wait. . .
Caught signal SIGTERM(15) on PID 12997
WARNING:Next signal will force exit
Code: Select all
sanae@akarui ~/fah> clinfo
Number of platforms 1
Platform Name AMD Accelerated Parallel Processing
Platform Vendor Advanced Micro Devices, Inc.
Platform Version OpenCL 2.1 AMD-APP (3004.6)
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
Platform Host timer resolution 1ns
Platform Extensions function suffix AMD
Platform Name AMD Accelerated Parallel Processing
Number of devices 1
Device Name gfx900
Device Vendor Advanced Micro Devices, Inc.
Device Vendor ID 0x1002
Device Version OpenCL 2.0 AMD-APP (3004.6)
Driver Version 3004.6 (PAL,HSAIL)
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Board Name (AMD) Radeon RX Vega
Device Topology (AMD) PCI-E, 0b:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 56
SIMD per compute unit (AMD) 4
SIMD width (AMD) 16
SIMD instruction width (AMD) 1
Max clock frequency 1590MHz
Graphics IP (AMD) 9.0
Device Partition (core)
Max number of sub-devices 56
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 1024x1024x1024
Max work group size 256
Preferred work group size (AMD) 256
Max work group size (AMD) 1024
Preferred work group size multiple 64
Wavefront width (AMD) 64
Preferred / native vector sizes
char 4 / 4
short 2 / 2
int 1 / 1
long 1 / 1
half 1 / 1 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (cl_khr_fp16)
Denormals No
Infinity and NANs No
Round to nearest No
Round to zero No
Round to infinity No
IEEE754-2008 fused multiply-add No
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals No
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 8573157376 (7.984GiB)
Global free memory (AMD) 8306688 (7.922GiB)
Global memory channels (AMD) 64
Global memory banks per channel (AMD) 4
Global memory bank width (AMD) 256 bytes
Error Correction support No
Max memory allocation 4244635648 (3.953GiB)
Unified memory for Host and Device No
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing Yes
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 2048 bits (256 bytes)
Preferred alignment for atomics
SVM 0 bytes
Global 0 bytes
Local 0 bytes
Max size for global variable 3820172032 (3.558GiB)
Preferred total size of global vars 8573157376 (7.984GiB)
Global Memory cache type Read/Write
Global Memory cache size 16384 (16KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 2048 images
Base address alignment for 2D image buffers 256 bytes
Pitch alignment for 2D image buffers 256 pixels
Max 2D image size 16384x16384 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 128
Max number of write image args 64
Max number of read/write image args 64
Max number of pipe args 16
Max active pipe reservations 16
Max pipe packet size 4244635648 (3.953GiB)
Local memory type Local
Local memory size 65536 (64KiB)
Local memory syze per CU (AMD) 65536 (64KiB)
Local memory banks (AMD) 32
Max number of constant args 8
Max constant buffer size 4244635648 (3.953GiB)
Preferred constant buffer size (AMD) 16384 (16KiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution No
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 262144 (256KiB)
Max size 8388608 (8MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Number of P2P devices (AMD) 0
P2P devices (AMD) <printDeviceInfo:147: get number of CL_DEVICE_P2P_DEVICES_AMD : error -30>
Profiling timer resolution 1ns
Profiling timer offset since Epoch (AMD) 1586642702235317192ns (Sat Apr 11 15:05:02 2020)
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Thread trace supported (AMD) Yes
Number of async queues (AMD) 4
Max real-time compute queues (AMD) 0
Max real-time compute units (AMD) 0
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_khr_gl_depth_images cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_amd_copy_buffer_p2p
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) AMD Accelerated Parallel Processing
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [AMD]
clCreateContext(NULL, ...) [default] Success [AMD]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx900
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx900
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx900
ICD loader properties
ICD loader Name OpenCL ICD Loader
ICD loader Vendor OCL Icd free software
ICD loader Version 2.2.12
ICD loader Profile OpenCL 2.2