Project 13424 (Moonshot) very low PPD

[Shirty]

Just an observation, the PPD on these units seems to be wildly lower then I would expect, around half of normal output. Currently running the following units:

GTX 1080Ti - P13424(R147,C32,G0) - 730,000 ppd - 3:05 tpf
RTX 2060 - P13424(R30,C27,G1) - 730,000 ppd - 3.05 tpf
RTX 2070 - P13424(R120,C5,G1) - 800,000 ppd - 2.57 tpf
RTX 2070 Super - P13424(R90,C12,G1) - 950,000 ppd - 2.38 tpf
RTX 2070 Super - P13424(R21,C18,G1) - 1,100,000 ppd - 2:22 tpf
RTX 2070 Super - P13424(R50,C49,G1) - 1,180,000 ppd - 2.16 tpf

Any idea why these are all running so slowly, utilising circa 60% of GPU power? It has literally shaved 6 million ppd off my daily output since I started picking these up this morning.

Once all 11 of my cards pick these up (as they invariably will) I will see my total output halved from around 21-22 million to 10-11 million ppd.

[HaloJones]

Strange.My 1070s are down on 13422 but only by around 20% not 50%.

EDIT: Meant to write 13424

[Shirty]

I've got no real issue with 13422, which fortuitously I seem to have gone back to on most of my cards since I posted the OP. I just thought it might benefit John to see my findings in case it's a wider issue.

[Joe_H]

My understanding is this project has been pulled for now, problems were encountered with many WUs.

[Shirty]

Cheers, hopefully it'll be back in New and improved form before long!

[JohnChodera]

We had an issue in the preparation and quality control with the latest batch, so we've halted 13424-5 and rolled back to Sprint 2. We think we've fixed the issue and will re-launch tomorrow.

Thanks so much for your patience! These projects have produced incredibly useful data, and despite each week being nearly identical to the last, there always seems to be a new unanticipated challenge that sneaks in somehow!

~ John Chodera // MSKCC

[JohnChodera]

Just an update on this: We've identified the issue here: A nearly co-linear bond between old and new atoms connected to a central heavy atom appears to be causing the OpenMM CCMA algorithm some pathological problems. Removing this single constraint speeds up the simulation by multiple times. This issue was likely the source of the significant RUN-to-RUN variation in earlier 134xx Moonshot projects.

We're tracking this issue here, for those that want to follow along:
https://github.com/openmm/openmm/issues/2814

While working on a long-term solution, we will re-prepare the system without constraints and start the sprint running shortly.

Thanks again for your patience!

~ John Chodera // MSKCC

[foldingfanmucde]

Hi,
Observations on my side today, 29th August are:
Only project 13424 WUs seem to be currently being assigned to all my GPUs (NVIDIA and AMD) and PPD down by around 25% compared with e.g. WUs from project 13422 / 13423.
Maybe those symptoms are useful.
BR

[Neil-B]

For the most part 13422/13423 PPD were very high (on my kit between 50%/100% depending on WU) to allow for the occasional low one ... now that the issue with low ones has been resolved I'd have expected PPDs on current sprint to be near normal ... I am seeing PPDs maybe 5% to 10% higher (if that) than I might have expected for my kit on this core so actually seem quite reasonable to me.

[koolpep]

Confirming @foldingfanmuc.de - PPD down 25-30% with 13424 units compared to 13422 units. Actually base credit and bonus credit is down - processing time similar per WU.

[HaloJones]

13424 is less than 13422 but 13422 was significantly above previous WU for my cards.

[PantherX]

Welcome to the F@H Forum koolpep,

To paraphrase what Neil-B said, the Project 13424/13425 will now be using normal base points instead of the "higher base points" which previous Projects in 134XX series were using. The rational was in the current Project 13424/13425, there isn't much variation from RUN-to-RUN thus, there was no need to compensate for that. The previous Projects in 134XX series had a large variation between RUN-to-RUN so to ensure that the average PPD would be similar, a higher base points was temporarily used to compensate for the very low PPD in some RUNs.

[Neil-B]

Please see viewtopic.php?f=74&t=36001&start=15#p342090 confirming PantherX comments

[JohnChodera]

Just to echo @PantherX above: We've finally fixed the issues that were causing lots of heterogeneity, so we're now using a base credit more consistent with other projects. With previous 134xx projects, you may have been the lucky ones that never hit any slow RUNs---if so, you got extra points! If not, you probably had a more normal amount of PPD. Now that we were able to eliminate that RUN-to-RUN variation, we're back to more or less normal!

~ John Chodera // MSKCC

[cine.chris]

All my systems are down today at 82-83% of what I've been sustaining for several weeks. My goal was 300M pts for August which I'll make, but within hours now. User/Donor abandonment is already high, this could accelerate that rate. I'm still seeing normal levels of power consumption & GPU utilization, both a red flag to me that these WU are consuming resources (unlike some others I whined about) at high levels. Points don't cost anyone anything... except disgruntled & frustrated donors that abandon a project as that's the only metric & compensation they see.