Docker /fah write issue
Posted: Mon Dec 14, 2020 5:04 pm
I can't seem to get the F@H Docker image to write to the /fah volume flagged for it. It can read the config file but it doesnt seem to want to write checkpoints into the folder. This is being run on Google Cloud in a preemptive instance which is why the writing of checkpoints would be nice between shutdowns.
Code: Select all
@linuxfolding-preemptive:~$ sudo docker run --gpus all --name fah0 --volume $HOME/fah:/fah foldingathome/fah-gpu
16:11:39:Downloading GPUs.txt from assign1.foldingathome.org:80
16:11:39:Connecting to assign1.foldingathome.org:80
16:11:39:Read GPUs.txt
16:11:40:******************************* libFAH ********************************
16:11:40: Date: Oct 20 2020
16:11:40: Time: 20:36:39
16:11:40: Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
16:11:40: Branch: master
16:11:40: Compiler: GNU 8.3.0
16:11:40: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:11:40: -fdata-sections -O3 -funroll-loops -fno-pie
16:11:40: Platform: linux2 5.8.0-1-amd64
16:11:40: Bits: 64
16:11:40: Mode: Release
16:11:40:****************************** FAHClient ******************************
16:11:40: Version: 7.6.21
16:11:40: Author: Joseph Coffland <[email protected]>
16:11:40: Copyright: 2020 foldingathome.org
16:11:40: Homepage: https://foldingathome.org/
16:11:40: Date: Oct 20 2020
16:11:40: Time: 20:39:00
16:11:40: Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
16:11:40: Branch: master
16:11:40: Compiler: GNU 8.3.0
16:11:40: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:11:40: -fdata-sections -O3 -funroll-loops -fno-pie
16:11:40: Platform: linux2 5.8.0-1-amd64
16:11:40: Bits: 64
16:11:40: Mode: Release
16:11:40: Args: --chdir /fah
16:11:40: Config: /fah/config.xml
16:11:40:******************************** CBang ********************************
16:11:40: Date: Oct 20 2020
16:11:40: Time: 18:37:59
16:11:40: Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
16:11:40: Branch: master
16:11:40: Compiler: GNU 8.3.0
16:11:40: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
16:11:40: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
16:11:40: Platform: linux2 5.8.0-1-amd64
16:11:40: Bits: 64
16:11:40: Mode: Release
16:11:40:******************************* System ********************************
16:11:40: CPU: Intel(R) Xeon(R) CPU @ 2.30GHz
16:11:40: CPU ID: GenuineIntel Family 6 Model 63 Stepping 0
16:11:40: CPUs: 2
16:11:40: Memory: 1.74GiB
16:11:40: Free Memory: 876.16MiB
16:11:40: Threads: POSIX_THREADS
16:11:40: OS Version: 5.4
16:11:40: Has Battery: false
16:11:40: On Battery: false
16:11:40: UTC Offset: 0
16:11:40: PID: 1
16:11:40: CWD: /fah
16:11:40: OS: Linux 5.4.0-1032-gcp x86_64
16:11:40: OS Arch: AMD64
16:11:40: GPUs: 1
16:11:40: GPU 0: Bus:0 Slot:4 Func:0 NVIDIA:7 GP100GL [Tesla P100 16GB] 9340
16:11:40: CUDA Device 0: Platform:0 Device:0 Bus:0 Slot:4 Compute:6.0 Driver:11.0
16:11:40:OpenCL Device 0: Platform:0 Device:0 Bus:0 Slot:0 Compute:1.2 Driver:450.80
16:11:40:***********************************************************************
16:11:40:<config>
16:11:40: <!-- Folding Slot Configuration -->
16:11:40: <disable-viz v='true'/>
16:11:40:
16:11:40: <!-- GUI -->
16:11:40: <gui-enabled v='false'/>
16:11:40:
16:11:40: <!-- User Information -->
16:11:40: <passkey v='*****'/>
16:11:40: <team v='**'/>
16:11:40: <user v='**'/>
16:11:40:
16:11:40: <!-- Web Server -->
16:11:40: <web-enable v='false'/>
16:11:40:
16:11:40: <!-- Folding Slots -->
16:11:40: <slot id='0' type='GPU'/>
16:11:40: <slot id='1' type='CPU'>
16:11:40: <cpus v='2'/>
16:11:40: </slot>
16:11:40:</config>
16:11:40:Trying to access database...
16:11:40:Successfully acquired database lock
16:11:40:WARNING:FS00:Guessing ambiguous GPU to OpenCL device mapping for 00: gpu:0:4 GP100GL [Tesla P100 16GB] 9340. Consider upgrading your graphics driver or manually setting ``opencl-index`` in this slot's configuration.
16:11:40:FS00:Set client configured
16:11:40:FS00:Initialized folding slot 00: gpu:0:4 GP100GL [Tesla P100 16GB] 9340
16:11:40:FS01:Initialized folding slot 01: cpu:2
16:11:40:WU00:FS00:Connecting to assign1.foldingathome.org:80
16:11:40:WU01:FS01:Connecting to assign1.foldingathome.org:80
16:11:40:WU01:FS01:Connecting to assign1.foldingathome.org:80
16:11:40:WU00:FS00:Connecting to assign1.foldingathome.org:80
16:11:40:WU01:FS01:Assigned to work server 69.94.66.6
16:11:40:WU01:FS01:Requesting new work unit for slot 01: cpu:2 from 69.94.66.6
16:11:40:WU01:FS01:Connecting to 69.94.66.6:8080
16:11:40:WU00:FS00:Assigned to work server 206.223.170.146
16:11:40:WU00:FS00:Requesting new work unit for slot 00: gpu:0:4 GP100GL [Tesla P100 16GB] 9340 from 206.223.170.146
16:11:40:WU00:FS00:Connecting to 206.223.170.146:8080
16:11:41:WU00:FS00:Downloading 13.78MiB
16:11:41:WU01:FS01:Downloading 2.83MiB
16:11:42:WU00:FS00:Download complete
16:11:42:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:17427 run:0 clone:1030 gen:114 core:0x22 unit:0x00000095cedfaa920000000000000406
16:11:42:WU00:FS00:Downloading core from http://cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah
16:11:42:WU00:FS00:Connecting to cores.foldingathome.org:80
16:11:42:WU00:FS00:FahCore 22: Downloading 79.02MiB
16:11:43:WU01:FS01:Download complete
16:11:43:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14814 run:2135 clone:1 gen:89 core:0xa7 unit:0x00000064455e42060000000008570001
16:11:43:WU01:FS01:Downloading core from http://cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah
16:11:43:WU01:FS01:Connecting to cores.foldingathome.org:80
16:11:43:WU01:FS01:FahCore a7: Downloading 8.17MiB
16:11:45:WU01:FS01:FahCore a7: Download complete
16:11:45:WU01:FS01:Valid core signature
16:11:45:WU01:FS01:Unpacked 18.52MiB to cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7
16:11:45:WU01:FS01:Starting
16:11:45:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /fah/cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 1 -checkpoint 15 -np 2
16:11:45:WU01:FS01:Started FahCore on PID 15
16:11:45:WU01:FS01:Core PID:19
16:11:45:WU01:FS01:FahCore 0xa7 started
16:11:45:WU01:FS01:0xa7:*********************** Log Started 2020-12-14T16:11:45Z ***********************
16:11:45:WU01:FS01:0xa7:************************** Gromacs Folding@home Core ***************************
16:11:45:WU01:FS01:0xa7: Type: 0xa7
16:11:45:WU01:FS01:0xa7: Core: Gromacs
16:11:45:WU01:FS01:0xa7: Args: -dir 01 -suffix 01 -version 706 -lifeline 15 -checkpoint 15 -np 2
16:11:45:WU01:FS01:0xa7:************************************ CBang *************************************
16:11:45:WU01:FS01:0xa7: Date: Nov 27 2019
16:11:45:WU01:FS01:0xa7: Time: 11:26:54
16:11:45:WU01:FS01:0xa7: Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
16:11:45:WU01:FS01:0xa7: Branch: master
16:11:45:WU01:FS01:0xa7: Compiler: GNU 8.3.0
16:11:45:WU01:FS01:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
16:11:45:WU01:FS01:0xa7: -fno-pie -fPIC
16:11:45:WU01:FS01:0xa7: Platform: linux2 4.19.0-5-amd64
16:11:45:WU01:FS01:0xa7: Bits: 64
16:11:45:WU01:FS01:0xa7: Mode: Release
16:11:45:WU01:FS01:0xa7:************************************ System ************************************
16:11:45:WU01:FS01:0xa7: CPU: Intel(R) Xeon(R) CPU @ 2.30GHz
16:11:45:WU01:FS01:0xa7: CPU ID: GenuineIntel Family 6 Model 63 Stepping 0
16:11:45:WU01:FS01:0xa7: CPUs: 2
16:11:45:WU01:FS01:0xa7: Memory: 1.74GiB
16:11:45:WU01:FS01:0xa7:Free Memory: 779.97MiB
16:11:45:WU01:FS01:0xa7: Threads: POSIX_THREADS
16:11:45:WU01:FS01:0xa7: OS Version: 5.4
16:11:45:WU01:FS01:0xa7:Has Battery: false
16:11:45:WU01:FS01:0xa7: On Battery: false
16:11:45:WU01:FS01:0xa7: UTC Offset: 0
16:11:45:WU01:FS01:0xa7: PID: 19
16:11:45:WU01:FS01:0xa7: CWD: /fah/work
16:11:45:WU01:FS01:0xa7:******************************** Build - libFAH ********************************
16:11:45:WU01:FS01:0xa7: Version: 0.0.19
16:11:45:WU01:FS01:0xa7: Author: Joseph Coffland <[email protected]>
16:11:45:WU01:FS01:0xa7: Copyright: 2019 foldingathome.org
16:11:45:WU01:FS01:0xa7: Homepage: https://foldingathome.org/
16:11:45:WU01:FS01:0xa7: Date: Nov 26 2019
16:11:45:WU01:FS01:0xa7: Time: 00:41:42
16:11:45:WU01:FS01:0xa7: Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
16:11:45:WU01:FS01:0xa7: Branch: master
16:11:45:WU01:FS01:0xa7: Compiler: GNU 8.3.0
16:11:45:WU01:FS01:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
16:11:45:WU01:FS01:0xa7: -fno-pie
16:11:45:WU01:FS01:0xa7: Platform: linux2 4.19.0-5-amd64
16:11:45:WU01:FS01:0xa7: Bits: 64
16:11:45:WU01:FS01:0xa7: Mode: Release
16:11:45:WU01:FS01:0xa7:************************************ Build *************************************
16:11:45:WU01:FS01:0xa7: SIMD: avx_256
16:11:45:WU01:FS01:0xa7:********************************************************************************
16:11:45:WU01:FS01:0xa7:Project: 14814 (Run 2135, Clone 1, Gen 89)
16:11:45:WU01:FS01:0xa7:Unit: 0x00000064455e42060000000008570001
16:11:45:WU01:FS01:0xa7:Reading tar file core.xml
16:11:45:WU01:FS01:0xa7:Reading tar file frame89.tpr
16:11:45:WU01:FS01:0xa7:Digital signatures verified
16:11:45:WU01:FS01:0xa7:Calling: mdrun -s frame89.tpr -o frame89.trr -cpt 15 -nt 2
16:11:45:WU01:FS01:0xa7:Steps: first=0 total=250000
16:11:48:WU00:FS00:FahCore 22: 45.48%
16:11:52:WU01:FS01:0xa7:Completed 1 out of 250000 steps (0%)
16:11:54:WU00:FS00:FahCore 22: 91.20%
16:11:55:WU00:FS00:FahCore 22: Download complete
16:11:55:WU00:FS00:Valid core signature
16:11:55:WU00:FS00:Unpacked 5.21MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22
16:11:55:WU00:FS00:Unpacked 65.05KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMPME.so
16:11:55:WU00:FS00:Unpacked 2.72MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMOpenCL.so
16:11:55:WU00:FS00:Unpacked 32.98KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCudaCompiler.so
16:11:55:WU00:FS00:Unpacked 2.54MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCUDA.so
16:11:55:WU00:FS00:Unpacked 84.05MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libcufft.so.9.2
16:11:55:WU00:FS00:Unpacked 570.08KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCPU.so
16:11:55:WU00:FS00:Unpacked 3.21MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libnvrtc-builtins.so
16:11:55:WU00:FS00:Unpacked 3.22MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMM.so
16:11:55:WU00:FS00:Unpacked 30.73KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libfftw3f_threads.so.3
16:11:55:WU00:FS00:Unpacked 1.57MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libfftw3f.so.3
16:11:55:WU00:FS00:Unpacked 19.32MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libnvrtc.so.9.2
16:11:56:WU00:FS00:Starting
16:11:56:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /fah/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 1 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
16:11:56:WU00:FS00:Started FahCore on PID 24
16:11:56:WU00:FS00:Core PID:28
16:11:56:WU00:FS00:FahCore 0x22 started
16:11:56:WU00:FS00:0x22:*********************** Log Started 2020-12-14T16:11:56Z ***********************
16:11:56:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:11:56:WU00:FS00:0x22: Core: Core22
16:11:56:WU00:FS00:0x22: Type: 0x22
16:11:56:WU00:FS00:0x22: Version: 0.0.13
16:11:56:WU00:FS00:0x22: Author: Joseph Coffland <[email protected]>
16:11:56:WU00:FS00:0x22: Copyright: 2020 foldingathome.org
16:11:56:WU00:FS00:0x22: Homepage: https://foldingathome.org/
16:11:56:WU00:FS00:0x22: Date: Sep 19 2020
16:11:56:WU00:FS00:0x22: Time: 01:10:35
16:11:56:WU00:FS00:0x22: Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:11:56:WU00:FS00:0x22: Branch: core22-0.0.13
16:11:56:WU00:FS00:0x22: Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:11:56:WU00:FS00:0x22: Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:11:56:WU00:FS00:0x22: -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
16:11:56:WU00:FS00:0x22: Platform: linux2 4.19.76-linuxkit
16:11:56:WU00:FS00:0x22: Bits: 64
16:11:56:WU00:FS00:0x22: Mode: Release
16:11:56:WU00:FS00:0x22:Maintainers: John Chodera <[email protected]> and Peter Eastman
16:11:56:WU00:FS00:0x22: <[email protected]>
16:11:56:WU00:FS00:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 24 -checkpoint 15
16:11:56:WU00:FS00:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
16:11:56:WU00:FS00:0x22: nvidia -gpu 0 -gpu-usage 100
16:11:56:WU00:FS00:0x22:************************************ libFAH ************************************
16:11:56:WU00:FS00:0x22: Date: Sep 15 2020
16:11:56:WU00:FS00:0x22: Time: 05:14:43
16:11:56:WU00:FS00:0x22: Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:11:56:WU00:FS00:0x22: Branch: HEAD
16:11:56:WU00:FS00:0x22: Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:11:56:WU00:FS00:0x22: Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:11:56:WU00:FS00:0x22: -funroll-loops
16:11:56:WU00:FS00:0x22: Platform: linux2 4.19.76-linuxkit
16:11:56:WU00:FS00:0x22: Bits: 64
16:11:56:WU00:FS00:0x22: Mode: Release
16:11:56:WU00:FS00:0x22:************************************ CBang *************************************
16:11:56:WU00:FS00:0x22: Date: Sep 15 2020
16:11:56:WU00:FS00:0x22: Time: 05:11:04
16:11:56:WU00:FS00:0x22: Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:11:56:WU00:FS00:0x22: Branch: HEAD
16:11:56:WU00:FS00:0x22: Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
16:11:56:WU00:FS00:0x22: Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
16:11:56:WU00:FS00:0x22: -funroll-loops -fPIC
16:11:56:WU00:FS00:0x22: Platform: linux2 4.19.76-linuxkit
16:11:56:WU00:FS00:0x22: Bits: 64
16:11:56:WU00:FS00:0x22: Mode: Release
16:11:56:WU00:FS00:0x22:************************************ System ************************************
16:11:56:WU00:FS00:0x22: CPU: Intel(R) Xeon(R) CPU @ 2.30GHz
16:11:56:WU00:FS00:0x22: CPU ID: GenuineIntel Family 6 Model 63 Stepping 0
16:11:56:WU00:FS00:0x22: CPUs: 2
16:11:56:WU00:FS00:0x22: Memory: 1.74GiB
16:11:56:WU00:FS00:0x22:Free Memory: 617.92MiB
16:11:56:WU00:FS00:0x22: Threads: POSIX_THREADS
16:11:56:WU00:FS00:0x22: OS Version: 5.4
16:11:56:WU00:FS00:0x22:Has Battery: false
16:11:56:WU00:FS00:0x22: On Battery: false
16:11:56:WU00:FS00:0x22: UTC Offset: 0
16:11:56:WU00:FS00:0x22: PID: 28
16:11:56:WU00:FS00:0x22: CWD: /fah/work
16:11:56:WU00:FS00:0x22:************************************ OpenMM ************************************
16:11:56:WU00:FS00:0x22: Revision: 189320d0
16:11:56:WU00:FS00:0x22:********************************************************************************
16:11:56:WU00:FS00:0x22:Project: 17427 (Run 0, Clone 1030, Gen 114)
16:11:56:WU00:FS00:0x22:Unit: 0x00000095cedfaa920000000000000406
16:11:56:WU00:FS00:0x22:Reading tar file core.xml
16:11:56:WU00:FS00:0x22:Reading tar file integrator.xml.bz2
16:11:56:WU00:FS00:0x22:Reading tar file state.xml.bz2
16:11:56:WU00:FS00:0x22:Reading tar file system.xml.bz2
16:11:56:WU00:FS00:0x22:Digital signatures verified
16:11:56:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:11:56:WU00:FS00:0x22:Version 0.0.13
16:11:56:WU00:FS00:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
16:11:56:WU00:FS00:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
16:11:56:WU00:FS00:0x22: XTC frame write interval: 10000 steps (0.8%) [125 total]
16:11:56:WU00:FS00:0x22: Global context and integrator variables write interval: disabled
16:11:56:WU00:FS00:0x22:There are 4 platforms available.
16:11:56:WU00:FS00:0x22:Platform 0: Reference
16:11:56:WU00:FS00:0x22:Platform 1: CPU
16:11:56:WU00:FS00:0x22:Platform 2: OpenCL
16:11:56:WU00:FS00:0x22: opencl-device 0 specified
16:11:56:WU00:FS00:0x22:Platform 3: CUDA
16:11:56:WU00:FS00:0x22: cuda-device 0 specified
16:12:36:WU00:FS00:0x22:Attempting to create CUDA context:
16:12:36:WU00:FS00:0x22: Configuring platform CUDA
16:12:41:Saving configuration to config.xml
16:12:41:<config>
16:12:41: <!-- Folding Slot Configuration -->
16:12:41: <disable-viz v='true'/>
16:12:41:
16:12:41: <!-- GUI -->
16:12:41: <gui-enabled v='false'/>
16:12:41:
16:12:41: <!-- User Information -->
16:12:41: <passkey v='*****'/>
16:12:41: <team v='33'/>
16:12:41: <user v='DonatedFolding'/>
16:12:41:
16:12:41: <!-- Web Server -->
16:12:41: <web-enable v='false'/>
16:12:41:
16:12:41: <!-- Folding Slots -->
16:12:41: <slot id='0' type='GPU'>
16:12:41: <pci-bus v='0'/>
16:12:41: <pci-slot v='4'/>
16:12:41: </slot>
16:12:41: <slot id='1' type='CPU'>
16:12:41: <cpus v='2'/>
16:12:41: </slot>
16:12:41:</config>
16:13:05:WU00:FS00:0x22: Using CUDA and gpu 0