Community driven support forum for Folding@home
https://foldingforum.org/
Vijay Pande, the founder of FAH, is currently trying to make a lot of money for startup companies doing innovative work. I hope that includes anything they find useful from FAH. They need a lot of money from the successes to cover the failures, which are usually about 90% in risky fields that are being newly developed. They will use patents I expect to help them compete and bring products to market. If universities can do it better, that is fine with me. But I haven't seen it yet. (Universities get patents too by the way, and like to make as much money from them as they can.)If open science (for example science done at universities, financed by public money) will help us to have drugs approximately at the price of generics from the beginning, this will be a big advance for health and life of people, for which otherwise treatment will not be available.
I hope that the results from FAH and project moonshot are published for unrestricted use instead of patented. At least that is what the promises look like and what should be self-evident for results based on "donors" electricity bills.JimF wrote:Vijay Pande, the founder of FAH, is currently trying to make a lot of money for startup companies doing innovative work. I hope that includes anything they find useful from FAH. They need a lot of money from the successes to cover the failures, which are usually about 90% in risky fields that are being newly developed. They will use patents I expect to help them compete and bring products to market. If universities can do it better, that is fine with me. But I haven't seen it yet. (Universities get patents too by the way, and like to make as much money from them as they can.)If open science (for example science done at universities, financed by public money) will help us to have drugs approximately at the price of generics from the beginning, this will be a big advance for health and life of people, for which otherwise treatment will not be available.
That could be right, but I don't think so. If a NaNs occurs before reaching a sanity check, it should fail there. Another possibility is that the message which is being issued isn't an accurate description of the problem. Running the error condition locally should quickly isolate that possibility whether in John's lab or ThWuensche'sNeil-B wrote:It could be that the sanity checks/checkpoints are after steps 250 or 501 ... That may be why the errors always show up at that point as until checks are done then the core doesn't know there is an issue?
@ThWuensche: This is great! We now have a test case that makes it very easy to reproduce the issue! Can you post this on the OpenMM issue tracker? Our lead OpenMM developer Peter Eastman can work with you there to try a few more things to debug:With the information in your post I have installed miniconda on one more computer, installed openmm, downloaded the zip-file with tests and run RUN9. First it would break with particle coordinate nan before reporting any iterations, after setting the steps_per_iteration to one and increasing niterations it breaks after "completed 250 steps". That is a first hint, since I verified in the logs (FAH) on that computer that in most cases it also breaks at step 250. This indicates that it might be not a result of a calculation running away, but something systematically linked to step 250. Besides many occurrences of step 250 there are some with step 501. From the FAH logs I assumed that maybe 250 would be a first verification point and that would be the reason for that coincidence, but from running openmm in single iterations leading to the same result (counting up all steps before in the console output) that coincidence raises questions.
The issue is already open: https://github.com/openmm/openmm/issues/2813. As mentioned, if run with precision double it does not break.JohnChodera wrote:
@ThWuensche: This is great! We now have a test case that makes it very easy to reproduce the issue! Can you post this on the OpenMM issue tracker? Our lead OpenMM developer Peter Eastman can work with you there to try a few more things to debug:
http://github.com/openmm/openmm
Please tag me there as @jchodera and I will chime in with more information and the test scripts.
If you can also post more details of your configuration (a FAH science logs header will do!) there, it will help keep the information organized.
Thank you for helping us get to the bottom of this!
~ John Chodera // MSKCC
Code: Select all
*************************** Core22 Folding@home Core ***************************
Core: Core22
Type: 0x22
Version: 0.0.11
Author: Joseph Coffland <[email protected]>
Copyright: 2020 foldingathome.org
Homepage: https://foldingathome.org/
Date: Jun 27 2020
Time: 22:50:00
Revision: cfc2940c5dd1aa80f60daa6e28d4a2a417f74edb
Branch: core22-0.0.11
Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
-funroll-loops
Platform: linux2 4.19.76-linuxkit
Bits: 64
Mode: Release
Maintainers: John Chodera <[email protected]> and Peter Eastman
<[email protected]>
Args: -dir 03 -suffix 01 -version 706 -lifeline 12914 -checkpoint 15
-gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
************************************ libFAH ************************************
Date: Jun 27 2020
Time: 22:11:04
Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
Branch: HEAD
Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
-funroll-loops
Platform: linux2 4.19.76-linuxkit
Bits: 64
Mode: Release
************************************ CBang *************************************
Date: Jun 27 2020
Time: 22:10:11
Revision: f8529962055b0e7bde23e429f5072ff758089dee
Branch: HEAD
Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
-funroll-loops -fPIC
Platform: linux2 4.19.76-linuxkit
Bits: 64
Mode: Release
************************************ System ************************************
CPU: AMD Ryzen 7 3700X 8-Core Processor
CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
CPUs: 16
Memory: 15.59GiB
Free Memory: 11.82GiB
Threads: POSIX_THREADS
OS Version: 5.7
Has Battery: false
On Battery: false
UTC Offset: 2
PID: 12918
CWD: /var/lib/fahclient/work
********************************************************************************
Folding@home GPU Core22 Folding@home Core
Version 0.0.11
[1] compatible platform(s):
-- 0 --
PROFILE = FULL_PROFILE
VERSION = OpenCL 2.0 AMD-APP (3137.0)
NAME = AMD Accelerated Parallel Processing
VENDOR = Advanced Micro Devices, Inc.
(2) device(s) found on platform 0:
-- 0 --
DEVICE_NAME = gfx906+sram-ecc
DEVICE_VENDOR = Advanced Micro Devices, Inc.
DEVICE_VERSION = OpenCL 2.0
-- 1 --
DEVICE_NAME = gfx906+sram-ecc
DEVICE_VENDOR = Advanced Micro Devices, Inc.
DEVICE_VERSION = OpenCL 2.0
[ Entering Init ]
Launch time: 2020-08-24T17:43:14Z
Arguments passed: -dir 03 -suffix 01 -version 706 -lifeline 12914 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
For testState comparison of CPU and GPU, will use:
forceTolerance: 5 kJ/mol/nm
energyTolerance: 10 kJ/mol
[ Leaving Init ]
[ Entering Main ]
Reading core settings...
Total number of steps: 1000000
Checkpoint write interval: 50000 steps (5%) [20 total]
JSON viewer frame write interval: 10000 steps (1%) [100 total]
XTC frame write interval: 250000 steps (25%) [4 total]
Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
[ Initializing Core Contexts ]
Using platform OpenCL
Looking for vendor: amd...found on platformId 0
Setting platform precision to mixed
Setting DisablePmeStream to 1
Checking for integrator.xml
Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
Checking for integrator.xml
Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
Checking for system.xml
Checking for system.xml.gz
Checking for system.xml.bz2
Found system.xml.bz2
Deserializing System...successful.
Found 90551 atoms, 10 forces.
Finding State XML file...
Checking for state.xml
Checking for state.xml.gz
Checking for state.xml.bz2
Found state.xml.bz2
Deserializing State...successful.
Ewald error tolerance in force 7 is 0.00025
Ewald parameters: alpha 2.75697 nx 96 ny 96 nz 96
Integrator Type: N6OpenMM16CustomIntegratorE
Constraint Tolerance: 1e-08
Time Step in PS: 0.004
Using CPU platform for reference calculations.
Performing initial sanity checks before starting work...
Comparing forces and energies between initial State and CPU...
Comparing forces and energies between GPU and CPU...