Project 18251 very low PPD on RTX 2060s

[muziqaz]

Yes, WU preparation as well as checkpoints take a long time to process. Difference is that WU preparation is single threaded, whole checkpoints are multithreaded

[appepi]

Has Fah Donor Stats really lost 2 million donors?

Yes I figure this should be a new topic but I can't find a new topic button to click, so I have done it here. On my own stats I keep track of the donors, and yesterday there were 3,012,048 and today it is a suspiciously round figure of 1,000,000.

Donor Statistics
Z443
Rank 7,265 of 1,000,000

I have earned 672,753,497 points by contributing 4,255 work units. My work unit was last recorded 2 hours ago
Active clients

1 within 50 days
1 within 7 days
My Teams

appepi (243941)

Earned 672,753,497 points by contributing 4,255 work units

[appepi]

WARNING: Just when I thought it might be safe to get back into the general folding pool instead of preferring Cancer, the (TU106) RTX 2060s in Z441, Z442 and Z443 were ALL attacked by a pack of lurking Project 18251 jobs and wasted their time plodding along at 1/3 PPD rates for a total of 36 hours to achieve 1.3M points between them, partly because I paused the job on Z441 that was planning to spend 3 hours in "peak" electricity time or 7 hours in off-peak equivalents.

[BobWilliams757]

It could be worse. I just somehow picked one up even though the preference is set to cancer.

[appepi]

I should have clarified that I still had the preference set to Cancer, so it looks like there's simply not enough work of that kind around to avoid 18251s. [It would be so much simpler if Fah offered a "not-18251" preference or 18251 processing had "not TU106" preference.] Today I awoke to another wasteful overnight run with 18251's running on all 3 devices, and this time paused all 3 immediately. Last weekend I collected two normal 24 hour runs (no "peak" rating period on weekends) yielding the usual PPD ie 6.2M and 6.0 M (3 x RTX 2060) or 2M PPD each. So that is my benchmark and on weekdays I normally get around 3M PPD with 11-hour running x 3. The last two days have been 2M and 1.6M, and tonight will be bad also because the three paused 18251's have to finish before more efficient users of my electricity can get access. Hohum. Meanwhile the transfer of the Zx00 sub-network (Z601, Z602, Z803, Z802, and Z805) from the dining room table where they have spent the last 8+ years to purpose-built accommodation in the attic is going well, so mustn't complain about the small stuff I suppose.

[muziqaz]

If FAH offered non-18251 preference, then every project should have one. That is not possible, and that would definitely lead to cherry picking

[BobWilliams757]

If FAH offered non-18251 preference, then every project should have one. That is not possible, and that would definitely lead to cherry picking

There is a huge difference between cherry picking and avoiding a project that obviously is creating problems for a fairly wide chunk of hardware. It only takes seconds looking at the project returns on LAR to see that it is impacting quite a few Turing and Ampere GPU's. Being that many of them are producing less than 1/2 usual PPD, or down to 1/3 in many cases, it wouldn't be a surprise that some users will probably just dump them.

THAT might be considered cherry picking, but even then it's a stretch.

[appepi]

Well I did suggest an alternative:

It would be so much simpler if ..... or 18251 processing had "not TU106" preference.

If (as BobWilliams757 noted earlier, it is possible to keep 18251 away from the largest GPUs because "it didn't scale well" then it must be possible to exclude a class of GPUs, and this might be extended to more detail, eg TU106, where it also doesn't scale well.

A feature of this kind would be able to deal with other anomalous cases of Project/GPU interaction, as distinct from main effects of Project across GPUs and main effects for GPUs across Projects. And although this might open other possibilities for cherry picking, of course researchers would never contemplate cherry-picking themselves, and to make assurance doubly sure it could be prevented by having criteria - eg less than 50% usual-for-specific-GPU performance. Though really I think that systematically anomalous performance of this kind should always be the subject of investigation in case it is the result of a systematic bug that might affect the science.

[muziqaz]

Researcher had the option, but with long consideration they chose to enable high end.

[C-A]

18251 is about 25-28% slower than 12294/12295 on the Arc B570. It receives 45-50% less PPD.

[BobWilliams757]

18251 is about 25-28% slower than 12294/12295 on the Arc B570. It receives 45-50% less PPD.

Just curious, what OS are you using?

I ask because the earlier runs of the project ran at usual speeds for me, but at some point slowed way down and have run slow ever since. Though I didn't find evidence of any recent Windows updates when I first noticed the behavior, it could be a combo of an update and/or another setting being changed on my system. In my case sometimes hard to track, since my wife uses it as much if not more than I do.

[C-A]

Windows - i didn't start paying attention to 18251 until a few months ago so i don't know if it had different behavior. it's the only project i've worked on with checkpoints longer than 3 seconds. with the 5600G it takes about 26.

[appepi]

Well, it seems that my reward for hauling 100Kg of workstations up two flights of stairs and 3.3 meters of ladder to their new home in the attic over the last few days has been to just make them even more attractive to 18251. Z803/RTX1080 had picked up an 18251 and dealt with it in about 6 hours at around 1.5M PPD - normal PPD - while poor old Z441/RTX2060 and Z443 /2060 are struggling for twice as long with 18251s and getting only 0.6M PPD for their trouble. And all despite being set to prefer Cancer jobs.

with the 5600G it takes about 26.

Sorry, I don't understand this. As far as I know (ie Google) Ryzen 5600G is a CPU with integrated graphics, while 18251 is a GPU project.

they chose to enable high end.

It is s strangely enabled "high end" where https://folding.lar.systems/projects/fo ... file/18251 shows Team Green with only a 4090 (#1, 29.1M PPD) followed by a 1080Ti (#7, 1.8M PPD), TITAN V (#9, 1.7M PPD), TU104 2060 (#10, 1.6M PPD) and 1080 (#13, 1.4M PPD), ... 3060 Mobile (#23, 1.1 M PPD) etc.

By comparison, another Core24 project (Cancer, 18230) has the first 12 ranks dominated by Team Green high end devices from 4090 to 3080.

So whether the Project enables the high end or not, it doesn't look like the NVIDIA high end enables the project?

[C-A]

At every 5% checkpoint on 18251, the CPU goes to 100% utilization for about half a minute before continuing the GPU fold. The 5600G is only about a 60W processor so it's not too bad, but I fear those with older dual Xeons may not fare so well.

The Intel Arc B570 taps out before 1.4M PPD. All other projects received so far for the card do 1.8M-2.5M.

[appepi]

This lead looked promising, so I had a quick look at Geekbench 6 Processor numbers and OpenCL numbers.

The 5600G as a 6 Core CPU gets 1936 single core and 7709 multicore on GB6.

[You don't give an actual Fah PPD figure for 18251 with this processor, which would be useful. LAR has a result for RX 5600 OEM/5600XT/5700/5700XT / Navi 10 [RX 5600 OEM/5600XT/5700/5700XT] of 0.33M PPD on 18251 but I don't know if that is relevant.]

Z803 has two Xeon X5675 (6 cores 12 Threads) which yield 502 single core and 2449 multicore (1CPU) in GB6 processor benchmarks, or actual (Z803) 3557 multicore with 2 CPUs and thus 12C 24T. Z803 just got 1.5M PPD on an 18251.

Z441 and Z442 have a single 6-core Xeon E5-1650 V3 which get 1213 single core and 5524 multi-core. I get 0.6 M PPD on 18251s.

We don't know how many cores Fah allocates to this CPU processing when running Core24 GPU projects, and how the Fah usage of the CPU is related to the basket of GB6 tasks, but historically when I used them for CPU processing in Fah, the relative PPD and relative GB5 or GB6 scores were aligned. Thus if CPU power is relevant to PPD for a Core24 project, my Z440s with more powerful Xeons and GPUs should not be taking about twice as long and getting only 40% of the PPD of the Z800 with less powerful Xeons and GPU - on Project 18251 jobs, ie the opposite of the predicted difference.

In terms of GPU power, GB6 Open CL comparison is:

Radeon graphics on 5600G = 60,326; RTX 1080 = 59,213; RTX 2060 = 76,598. With all the necessary caveats about GB6 OpenCL versus Fah CUDA, it is still a problem that the TU106 2060s are more powerful and get only 40% of 1080 PPD. So the mystery remains.