Project 13424 (Moonshot) very low PPD
Moderators: Site Moderators, FAHC Science Team
Project 13424 (Moonshot) very low PPD
Just an observation, the PPD on these units seems to be wildly lower then I would expect, around half of normal output. Currently running the following units:
GTX 1080Ti - P13424(R147,C32,G0) - 730,000 ppd - 3:05 tpf
RTX 2060 - P13424(R30,C27,G1) - 730,000 ppd - 3.05 tpf
RTX 2070 - P13424(R120,C5,G1) - 800,000 ppd - 2.57 tpf
RTX 2070 Super - P13424(R90,C12,G1) - 950,000 ppd - 2.38 tpf
RTX 2070 Super - P13424(R21,C18,G1) - 1,100,000 ppd - 2:22 tpf
RTX 2070 Super - P13424(R50,C49,G1) - 1,180,000 ppd - 2.16 tpf
Any idea why these are all running so slowly, utilising circa 60% of GPU power? It has literally shaved 6 million ppd off my daily output since I started picking these up this morning.
Once all 11 of my cards pick these up (as they invariably will) I will see my total output halved from around 21-22 million to 10-11 million ppd.
GTX 1080Ti - P13424(R147,C32,G0) - 730,000 ppd - 3:05 tpf
RTX 2060 - P13424(R30,C27,G1) - 730,000 ppd - 3.05 tpf
RTX 2070 - P13424(R120,C5,G1) - 800,000 ppd - 2.57 tpf
RTX 2070 Super - P13424(R90,C12,G1) - 950,000 ppd - 2.38 tpf
RTX 2070 Super - P13424(R21,C18,G1) - 1,100,000 ppd - 2:22 tpf
RTX 2070 Super - P13424(R50,C49,G1) - 1,180,000 ppd - 2.16 tpf
Any idea why these are all running so slowly, utilising circa 60% of GPU power? It has literally shaved 6 million ppd off my daily output since I started picking these up this morning.
Once all 11 of my cards pick these up (as they invariably will) I will see my total output halved from around 21-22 million to 10-11 million ppd.
Re: Project 13424 (Moonshot) very low PPD
Strange.My 1070s are down on 13422 but only by around 20% not 50%.
EDIT: Meant to write 13424
EDIT: Meant to write 13424
Last edited by HaloJones on Tue Aug 25, 2020 8:03 am, edited 1 time in total.
single 1070
Re: Project 13424 (Moonshot) very low PPD
I've got no real issue with 13422, which fortuitously I seem to have gone back to on most of my cards since I posted the OP. I just thought it might benefit John to see my findings in case it's a wider issue.
-
- Site Admin
- Posts: 7937
- Joined: Tue Apr 21, 2009 4:41 pm
- Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2 - Location: W. MA
Re: Project 13424 (Moonshot) very low PPD
My understanding is this project has been pulled for now, problems were encountered with many WUs.
iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Re: Project 13424 (Moonshot) very low PPD
Cheers, hopefully it'll be back in New and improved form before long!
-
- Pande Group Member
- Posts: 467
- Joined: Fri Feb 22, 2013 9:59 pm
Re: Project 13424 (Moonshot) very low PPD
We had an issue in the preparation and quality control with the latest batch, so we've halted 13424-5 and rolled back to Sprint 2. We think we've fixed the issue and will re-launch tomorrow.
Thanks so much for your patience! These projects have produced incredibly useful data, and despite each week being nearly identical to the last, there always seems to be a new unanticipated challenge that sneaks in somehow!
~ John Chodera // MSKCC
Thanks so much for your patience! These projects have produced incredibly useful data, and despite each week being nearly identical to the last, there always seems to be a new unanticipated challenge that sneaks in somehow!
~ John Chodera // MSKCC
-
- Pande Group Member
- Posts: 467
- Joined: Fri Feb 22, 2013 9:59 pm
Re: Project 13424 (Moonshot) very low PPD
Just an update on this: We've identified the issue here: A nearly co-linear bond between old and new atoms connected to a central heavy atom appears to be causing the OpenMM CCMA algorithm some pathological problems. Removing this single constraint speeds up the simulation by multiple times. This issue was likely the source of the significant RUN-to-RUN variation in earlier 134xx Moonshot projects.
We're tracking this issue here, for those that want to follow along:
https://github.com/openmm/openmm/issues/2814
While working on a long-term solution, we will re-prepare the system without constraints and start the sprint running shortly.
Thanks again for your patience!
~ John Chodera // MSKCC
We're tracking this issue here, for those that want to follow along:
https://github.com/openmm/openmm/issues/2814
While working on a long-term solution, we will re-prepare the system without constraints and start the sprint running shortly.
Thanks again for your patience!
~ John Chodera // MSKCC
-
- Posts: 18
- Joined: Sat Aug 01, 2020 10:26 am
- Hardware configuration: 1. AMD Ryzen 9 3900X / 64GB RAM / 2x NVIDIA RTX 2080 Ti / Win10 64 Pro / F@H 7.6.13 / ~6-8M PPD /
2. Intel Core i7 4790K / 32GB RAM / 1x AMD RX 5700 XT / Win10 64 Pro / F@H 7.6.21 / ~1-1.5M PPD / - Location: Munich, Germany
Re: Project 13424 (Moonshot) very low PPD
Hi,
Observations on my side today, 29th August are:
Only project 13424 WUs seem to be currently being assigned to all my GPUs (NVIDIA and AMD) and PPD down by around 25% compared with e.g. WUs from project 13422 / 13423.
Maybe those symptoms are useful.
BR
Observations on my side today, 29th August are:
Only project 13424 WUs seem to be currently being assigned to all my GPUs (NVIDIA and AMD) and PPD down by around 25% compared with e.g. WUs from project 13422 / 13423.
Maybe those symptoms are useful.
BR
1. AMD Ryzen 9 3900X / 64GB RAM / 2x NVIDIA RTX 2080 Ti / Win10 64 Pro / F@H 7.6.13 / ~6-8M PPD /
2. Intel Core i7 4790K / 32GB RAM / 1x AMD RX 5700 XT / Win10 64 Pro / F@H 7.6.21 / ~1-1.5M PPD /
2. Intel Core i7 4790K / 32GB RAM / 1x AMD RX 5700 XT / Win10 64 Pro / F@H 7.6.21 / ~1-1.5M PPD /
-
- Posts: 1996
- Joined: Sun Mar 22, 2020 5:52 pm
- Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21 - Location: UK
Re: Project 13424 (Moonshot) very low PPD
For the most part 13422/13423 PPD were very high (on my kit between 50%/100% depending on WU) to allow for the occasional low one ... now that the issue with low ones has been resolved I'd have expected PPDs on current sprint to be near normal ... I am seeing PPDs maybe 5% to 10% higher (if that) than I might have expected for my kit on this core so actually seem quite reasonable to me.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Re: Project 13424 (Moonshot) very low PPD
Confirming @foldingfanmuc.de - PPD down 25-30% with 13424 units compared to 13422 units. Actually base credit and bonus credit is down - processing time similar per WU.
Re: Project 13424 (Moonshot) very low PPD
13424 is less than 13422 but 13422 was significantly above previous WU for my cards.
single 1070
-
- Site Moderator
- Posts: 6986
- Joined: Wed Dec 23, 2009 9:33 am
- Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB
Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400 - Location: Land Of The Long White Cloud
- Contact:
Re: Project 13424 (Moonshot) very low PPD
Welcome to the F@H Forum koolpep,
To paraphrase what Neil-B said, the Project 13424/13425 will now be using normal base points instead of the "higher base points" which previous Projects in 134XX series were using. The rational was in the current Project 13424/13425, there isn't much variation from RUN-to-RUN thus, there was no need to compensate for that. The previous Projects in 134XX series had a large variation between RUN-to-RUN so to ensure that the average PPD would be similar, a higher base points was temporarily used to compensate for the very low PPD in some RUNs.
To paraphrase what Neil-B said, the Project 13424/13425 will now be using normal base points instead of the "higher base points" which previous Projects in 134XX series were using. The rational was in the current Project 13424/13425, there isn't much variation from RUN-to-RUN thus, there was no need to compensate for that. The previous Projects in 134XX series had a large variation between RUN-to-RUN so to ensure that the average PPD would be similar, a higher base points was temporarily used to compensate for the very low PPD in some RUNs.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time
Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time
Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
-
- Posts: 1996
- Joined: Sun Mar 22, 2020 5:52 pm
- Hardware configuration: 1: 2x Xeon [email protected], 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon [email protected], 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: [email protected], 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21 - Location: UK
Re: Project 13424 (Moonshot) very low PPD
Please see viewtopic.php?f=74&t=36001&start=15#p342090 confirming PantherX comments
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
-
- Pande Group Member
- Posts: 467
- Joined: Fri Feb 22, 2013 9:59 pm
Re: Project 13424 (Moonshot) very low PPD
Just to echo @PantherX above: We've finally fixed the issues that were causing lots of heterogeneity, so we're now using a base credit more consistent with other projects. With previous 134xx projects, you may have been the lucky ones that never hit any slow RUNs---if so, you got extra points! If not, you probably had a more normal amount of PPD. Now that we were able to eliminate that RUN-to-RUN variation, we're back to more or less normal!
~ John Chodera // MSKCC
~ John Chodera // MSKCC
-
- Posts: 78
- Joined: Sun Apr 26, 2020 1:29 pm
Re: Project 13424 (Moonshot) very low PPD
All my systems are down today at 82-83% of what I've been sustaining for several weeks. My goal was 300M pts for August which I'll make, but within hours now. User/Donor abandonment is already high, this could accelerate that rate. I'm still seeing normal levels of power consumption & GPU utilization, both a red flag to me that these WU are consuming resources (unlike some others I whined about) at high levels. Points don't cost anyone anything... except disgruntled & frustrated donors that abandon a project as that's the only metric & compensation they see.