Folding Forum

mdk777 wrote:

The PPD for bigadv were plain and simple out of line and needed to be adjusted, as VJ said they are in a tough spot and had to make a decision one way or the other.

If you recall, I argued for a reduction of the base points instead of a capping of the upper limit.

The final result to date is not far from what I argued for in regard to a possible solution.

My point is that the process is still flawed.

Rather than assuming

Moreover, with points, there will never be any system which makes everyone happy,

perhaps some thought should be given to changing the process.

Process Quality control 101

You don't attack the variable outcomes, you attack the process that leads to them.

Me thinks you repeated what I said, but I am not sure.

Me thinks you repeated what I said, but I am not sure.

OK, last time.

Change, rebalanced, normalization,whatever you want to call it; may or may not be required. The details are really not important.

What is important is that no rational person can make heads or tails of the process.

If there were a defined process, people could follow it and respond appropriately...

Since there is no defined process, people (donors) are left with just looking at the results of the process and trying to reverse engineer the factors that went into the evaluation process.

Every open source project, from LINUX to Ubuntu to Gimp has predictable process. I am not talking about re-inventing the wheel.

You can look at what is going into the next version... you have the ability to see what to expect....You might not agree with changes, you might not get what you want, but you know what to expect.

Here people are just blind-sided.

At this point someone will come in and post about the disclaimers about trial programs, and Beta projects.

If you are not going to administer, monitor and update: then these disclaimers amount to nothing more than boilerplate and CYA.

Simplified version of what you are saying.

1,479,431 reported folders last 24 hrs according to Free-DC http://stats.free-dc.org/stats.php?page=proj&proj=fah
185,640,308 28-Day Ave: PPD reported by Free-DC http://stats.free-dc.org/stats.php?page=proj&proj=fah
58,626,427 PPD folded by top 100 folders 24hr avg excluding anonymous, PS3 and PDC reported by EOC http://folding.extremeoverclocking.com/ ... p?s=&srt=1
Did the top 100 folders fold approx 1/3 of the total science value. Is this # a true representation of the science done.

So some how Stanford needs to let the general public know how they are coming up with the value of the science itself. Which is most likely determined by the complexity of the protein folding itself and what is required to do the complexity. the urgency of the project and cost to supply the resources needed. And what the future projects look like. Thus helping the folding public determine what there future plans might be.

I really dont understand it, they wanted fast returns and wanted to give a bonus for those that did,people invested or got access to machines that would do it. Now its a reduction or infraction for quick returns. On the outside looking in it makes no sense at all, a few folks complain because joe schmoe is ripping through the WUs and getting a ton of points so now stanford decides we dont need these WUs so quick so, lets piss a bunch of our donors off and see how many we can run off because joe schmoe #2 is complaining his P4 with socket 478 is not getting enough points.

I understand the reasoning for the change but I now have no idea what I should consider for future FAH investment (if anything now).

It's ALWAYS been that way due to technological advances and refined methodologies at Pande Group, and will continue to be so unless the research and technology become static. Is it frustrating at times? Yes, of course it is. Thinking back on the upgrades I made over the past 10+ years to leverage the Folding production trends - Athlon 900 to Athlon 1400, then Hyperthreaded Pentium 4s (Double Gromacs ), then pseudo dual core (Pentium D series), then Nvidia video cards (I missed the original GPU push with ATI), then onto quad core processors, then HT 4-core/8 thread, then the bottom nearly fell out of video cards. What next? I don't know, but I'm not asking Stanford and Pande Group to stand still for us. I upgrade cautiously, but I still upgrade machines. I won't always estimate and execute the optimal solution, but what's new? My comfort zone will continue to be challenged, and I have learned to accept that.

So, when the Bulldozer chips come out and people start complaining that 4p Bulldozers are scoring "too many" points....

If the points are out of proportion to the contribution returned to the science, then I would hope Pande Group would realign the points formula - down or up, whatever is most accurate.

I really dont understand it, they wanted fast returns and wanted to give a bonus for those that did....

The bonus for fast returns has not gone away, period. My -bigadv Folders are producing fewer points now, too. So be it. PG decided my fast returns were valuable, but not as much as originally estimated during the early stages of the big advanced beta initiative.

So some how Stanford needs to let the general public know how they are coming up with the value of the science itself.

Yes, I agree. If that's already out there, I haven't seen it, or did see it but didn't understand.

Simplified version of what you are saying.

No. What you are saying is great and I don't disagree. A unified explanation of the entire point system would be great.

No I am talking about just basic stuff;who,what, where, why, when, how.

From two years ago I get the following regarding when and how:

viewtopic.php?t=10697#p105038

Q: How long will this bonus program last?
A: It depends. We will continually be evaluating this bonus program and may alter or remove it at any time. We anticipate maintaining it at least through the initial cohort of 2681 work units, but everything depends on how the bonus trial goes.

OK, when will it be evaluated? once a month, every 3 months, or just once every 2 years

continually

This would imply that unless donors here otherwise, the project is continuous ?

Who will evaluate it and who will communicate that evaluation(where can I follow it?)

How will it be evaluated?

1. based on the speed of WU returned?
2. based on the server ability to handle increase of WU?
3. based on supply and demand of complex WU?
4. based on Political affect on donors and the moral of the overall project?

All of the above combined? The phase of the moon on off presidential election years?


This is project management 101. You make goals, you outline a plan.....AND YOU ESTABLISH A METHOD OF EVALUATING THE EXECUTION/REALIZATION OF THOSE GOALS.

I see the first part and not the second.

Since the second part is undefined, any change in the Plan of action is by definition arbitrary.

EDIT:

More boilerplate added after the fact:

Q: What happens to the bonus program at the end of this trial?
A: We will evaluate several types of results from the trial: distribution of return times for the WU, fraction of WU requiring reassignment, number of machines assigned by their users to fold these WU, reported problems with the WU and bonus system, and donor feedback posted on the forum. We will also consult with the forum mods and admins and with our beta testers. At that point we will decide whether to continue, revise, or shut down the bonus program. As in any trial, however, we reserve the right to stop it or put it on temporary hold if serious problems are noted.

When does the trial end?
where can I see updates? (official reviews, not just forum chatter)

Apparently all my hypothetical questions have been answered already.

My point is that donors have an expectation of greater direction and continuity, rather than random adjustments every two years without warning.

Napoleon wrote:Updated the spreadsheets to match the code above better.

Late or not, I updated the spreadsheets as well as the example graph once again. This time complete with PPD, instead of points per WU. As seen from the example graph, a 1% cap with a 5% applying threshold (fractions of preferred deadline length) wouldn't have much of an effect on PPD for a more or less hypothetical 8000 base points, kfactor = 27, pref. deadline 100 hours (~4 days) WU. One can experiment with different parameters in the spreadsheet - base points, kfactor, preferred deadline, P (Preset cap %) and T (cap application Threshold %).

Capping would only get applied once your time per WU gets below 5% anyway (5 hours in this example), and for a (near) zero completion time you'd be credited per WU as if you returned the WU in %1 of the preferred deadline (1 hour in this example). So, if you returned 60 / 30 / 15 / 7½ of these hypothetical WUs per hour, the PPD ratio would be about 617MPDD / 306MPPD / 150MPPD / 73MPPD - getting nearly linear PPD curve (2x PPD for ½ TPF) when you approach zero TPF (60 WU/hour = 1 WU/minute = 0.6s per frame). 0.6s TPF on a BigAdv class WU on a home computer any time soon... getting pretty SciFi already.

Please note that the horizontal axis is time per WU, not Time Per Frame.


I still think this "time shift" method applied to the bonus curve is pretty neat idea - something for the not so immediate future, perhaps? Allow me to present my reasoning behind this:

1. Timeshifted curve converges with the original at the Threshold point (points offset is calculated automagically in the spreadsheet & example code)
2. P & T could be associated to individual projects or even individual WUs in the future without too much effort, I think
3. For vast majority of donors, this would not have presented itself as immediate change, assuming P & T paramaters were chosen carefully for each (existing) project
4. The capped bonus curve would be practically identical with the original one up to a certain point, but the "time shift" would offset Moore's law somewhat and prevent the übercomputers of the future from completely devaluing earlier work. SMP and BigAdv WUs of today would not seem to be so much of a joke in a couple of years - points wise - as e.g the uniproc WUs seem to be today
5. It has slightly better potential for communicating the value of quick returns to donors on a project basis. Returning WUs close to Threshold (percentage T of preferred deadline) is a job well done. Returning between T and P still yields decent nonlinear increase in bonus points. Returning them faster than fraction P of deadline is "nice" and a donor would still get more or less linear increase in PPD, but it doesn't change the big picture that much in terms of science which can be extracted from the results

As for drastic changes... FAH is simulating (predicting ?) protein folding. I'm under the impression that simulations/predictions in general can be a bit volatile, it's part of their nature. Just consider other natural elements, like the weather and such. The forecast methods are usually quite good, but sometimes an anomaly occurs and you have more or less unanticipated tsunami/hurricane/whatever at your doorstep. Now, which is better, people debating about applications of Project Management 101 on TV or some other public forum, or people at least trying to do something about the situation at hand, according to Crisis Management 101? Sometimes it's not just the thought that counts. There's pressure to do something, and do so quickly. Hasty actions rarely are the best actions, but sometimes they're necessary in order to prevent a bad situation getting completely out of control.

IMHO, purchasing the latest and greatest hardware in the hopes of a perpetually consistent boost for a highly specific task with moving targets is a bit similar to (re)building a house next to an active volcano. Sure, it may be benign and convenient source of heat for a while, but you shouldn't be too surprised if you eventually get lava in your living room.

Frustrated, maybe ("Oh no, not AGAIN! I just got this darn thing rebuilt!"), but suprised, let alone angry?

PS: Still not sure whether the bonus multiplier formula should be sqrt(k*preferred_deadline/elapsed_time) or sqrt(k*final_deadline/elapsed_time). I used preferred_deadline, but the web based bonus points calculator seems to use final_deadline. Which one is correct? I used preferred_deadline, that just makes more sense to me because you only get base points anyway if elapsed_time > preferred_deadline, right?

Maybe the whole evaluation system as it is needs to be started afresh instead of constantly "adjusting" it,
Compare any of the machines used to benchmark various projects and even at it's release there are donor rig's waiting
that are far more powerful, take that 1-2 years down the track and donor machines are many times faster,
Two possible solutions are:
1. Outsource benchmarking to a "supercomputer" which will remain much faster than any donor machine for the estimated
life of the project, and run all projects through the same machine, then any QRB wouldn't go out of control as it would be
awarded by how many times slower the donor machine is,
2. One that would cost zero $$ but require more administration time would be a "project average" score which would move with donor
performance, as a possibility, all Wu's in a given project have a base point value, then at the end of each week, or even month a
"donor average score" is calculated and any bonus awarded relative to that score....
All I have ever wanted to see is a points system that remains fair and equitable while having a tangible link to the science work actually done,
other than that points are worth nothing but are (or should be) the "fun" part of Folding, Team's, leaderboards, races, challenges etc is all
a game, not a matter of life and death,
The Wu's we all return every day however, for someone, somewhere, someday, just might be.

Jester, I've been thinking down those lines for some time now.

I know that PG wants to keep control internally of the benchmarking process but maybe they should look at having a beta benchmarking team.

Since it's the private sector that is keeping up with technology PG could use those systems to do the benchmarking on instead of antiquated systems.

orion wrote:Jester, I've been thinking down those lines for some time now.

I know that PG wants to keep control internally of the benchmarking process but maybe they should look at having a beta benchmarking team.

Since it's the private sector that is keeping up with technology PG could use those systems to do the benchmarking on instead of antiquated systems.

That's the main reason I suggested a "Supercomputer", something that could run all the projects for benchmarking much faster than any donor rig,
It wouldn't matter if were Bigadv, smp or Gpu, they'd all be run on the one machine and scored as such,
Then any QRB bonus scheme would be worked out as 5 x reference time 10x reference time etc.,
it would involve setting aside the $$ in the budget, but that would be a drop in the ocean compared to the $$ spent by donors to give a " super, supercomputer"
for free.

With BD almost out and Interlagos and whatever Intel has in the pipe line some of us are going to get them and have the next best thing to a super computer anyway, as far as an enthusiast/home PC can get, so PG might as well use those for benchmarking.

Even our 48 core AMD's and the 12 core/24 thread SR2's would be better then what they are using now along with ATI's and Nvidia's best offerings.

There's pressure to do something, and do so quickly. Hasty actions rarely are the best actions, but sometimes they're necessary in order to prevent a bad situation getting completely out of control.

This was immediately a problem from inception. (If you view it as a problem)

Nothing has changed except perception. The fundamentals have been constant.

2 years is a long time to come up with a hasty solution.

EDIT:

PREVIOUS POST showing that it is certainly not a new(volcanic)topic:

Re: -bigadv on x6

Postby mdk777 » Sat Jul 03, 2010 9:57 am
Cinebench 11.5 reports results by core and thread.

If Cinebench 11.5 can tell the difference between cores and threads, I do not see why PG could not.

The initial release of bigadv made very specific and detailed mention that they did not want any OC 4 core machines (8 threads).

However, immediately they allowed them anyway.

This is just another case of the lack of consistency, rigor, fairness, discipline, logic (Whatever word you think is most appropriate to the situation.) of the point system.

Is the intention to morph the -bigadv point system into the -smp point system?

If so great! ; then the objective would be to eliminate the -bigadv entirely and have a continuous (not necessarily linear) bonus system that went from 1 core to 48 cores, (or whatever future cores are available.)

Alternately, if such a point system were impracticable, -bigadv could be opened up to all comers based on required memory and the 80% completion rule.

making arbitrary rulings, and then making exceptions is never going to be the way to encourage "investment". Investment thrives in environments of stability and predictability .

You may say again for the 10,000th time that "Science" is not predicable, or stable. Yet, that has absolutely nothing to do with how you reward your donors.
Just because the science is unpredictable, there is no reason the point system needs to be.

AND from the year before in a thread discussion of the difference in 8 core systems:

Re: How many people have 8 physical cores?

Postby mdk777 » Mon Jul 27, 2009 8:18 am

Q: Can I try running these units on my super-overclocked, liquid-cooled quad-core system?
A: No. In our experience, fast quad-core systems tend to come in over the 4-day timeout and would thus 1) not contribute to the scientific goals of finishing these projects quickly and 2) not receive bonuses. Quad-core systems can make important contributions to standard SMP projects, and we'd encourage you to apply them there.

Transparency and Accountability, the necessary foundation of any great endeavor!

The Bonus system immediately set up hardware based classes.

The only thing that has changed is that two years have now passed, and true 8-48 core systems are now becoming affordable.

So, now, when a great many people have the ability to "take advantage" of the offered bonus, it now becomes obsolete (normalized, adjusted, tweaked whatever...)

Well, that is really just bad management. It has nothing to do with crisis management.

Finally, besides pointing out the inconsistencies what have I contributed to finding a solution?

In my opinion, this part of the project is too important to be left as an afterthought, or a dreaded obligation.
While not central to the research science, it is central to maintaining the donors.

Hence it would be a good investment to have a full time person, a project manager. Someone who is not tasked with maintaining servers, or doing research papers. A person who would be both a points czar, and a liaison between the researchers and the folding donors.

With close to a half million donors, it certainly seem that a professional full time person would be justified.

Re: What is going on with BIGADV?

Postby mdk777 » Wed Oct 27, 2010 1:24 pm

If you were here day to day and saw what goes on, I think you'd have a radically different point of view in terms of how all of this works. We're here to push science forward and aren't trying to "bait and switch" donors (there's nothing to gain from that). We're trying to push the envelope and try new things, but in the end, we're here to push the science and that always takes precedence. The day when science doesn't take precedence is the day that we stray from our mission and betray our donor base, so it's something we take very seriously around here.



but in the end, we're here to push the science and that always takes precedence.

I don't think anyone has any problem with this. However, you are also correct that the donors cannot see the day to day decisions.
They only see the results of those decisions.

As 7im has implied, I read the forums much more regularly than most people (99.99% of the 300K folders)
Yet, I could not predict the status of either the Linux client fix, or the intended future availability of BIGADV WU for Road-runner.
Thank you for your response. While it might not have been the answer he was looking for, it was at least candid.

My conclusion:

For a project of this size a dedicated liaison person would be well worth the investment.
As you eluded, even MODS don't have sufficient access to know the status of various issues.
Educated guesses and conjecture is not really a substitute for reliable inside information.

I know there was a few posts regarding a "help desk".
I had hopes that this was a move in the direction of a dedicated liaison person. (A person who's primary responsibility,if not only responsibility, would be to communicate to donors; not involved directly in research or programing{yet have intimate access}.)

@mdk777
Well, I'm more of a micro-management type, I'm not up to managing FAH for PG.
However, after 18 or so pages of debate, I figured it's time to recheck the topic and recap what seems to be the bare essentials (to me):

Points system is getting ridiculous because...

1. the bonus points per WU curve is uncapped, and
2. Moore's law devalues all earlier work too quickly, and
3. it isn't clear to donors where quick returns are absolutely critical for scientific progress (read: GPU as well as most of uniproc work still is on a linear curve - ½ time per WU gets only 2x PPD)

So, I tried to come up with a fair and reasonable proposal for capping, in order to address at least 1) and 2).
Hopefully also something that could be applied to 3) without a hitch.

IMHO, it isn't rocket science to experiment with the latest spreadsheets I provided and give some feedback on my proposal. Something good in it? Something really bad? Should it be accepted by PG as such? Should it be modified first, and by whom? Should it be rejected altogether?

Apart from a few exceptions, it's been rather quiet in that sense... I guess it's more fun to reheat old issues using the same familiar arguments. I'm not saying there's no merit at all in what mdk777, for example, is saying. Let's just say that things aren't improving quite as fast as we'd like to see, but some steps have been taken already and there seems to be further corrective steps underway. Why not just leave it at that, at least for a while? PG has scientific priorities, we donors have PPD and ease-of-use priorities. I just can't see those ever becoming a perfect match, but I'll settle for close enough. My "close enough" isn't necessarily your "close enough", that goes without saying. But there it is, I said it anyway.

In fairly early stages of this thread - viewtopic.php?f=16&t=18797&start=45#p188200 - PG requested specific suggestions. As opposed to generic ones already seen, time after time.

VijayPande wrote:Could you make some specific suggestions for what you'd like to see us change? If bigadv is getting out of control, we can always stop that program or significantly bring it down, although of course other donors will be very unhappy with that. The challenge here is to find some new plan which can get a broad consensus, although I agree there will likely be no plan which makes everyone happy (hopefully we can make more people happy than now).

Well, I (finally) put forth a suggestion. In the form of a spreadsheet & pseudocode & visual aid, so I daresay I've been pretty specific here. I hope it helps.

@Leonardo Personally I kept quiet since you are one who's giving input to this thread in a concrete manner and one which I agree to be what was asked for ( I agree with your 3 points and your proposed fix ). Though I think you're not doing exactly what I had in mind it comes very close ( see the last post from mine where I talked about the flat line ), so there is no point in commenting on it as I already have done so. The 'fix' however I could not put in the form of a formula so if you didn't understand what I meant I will try to put it in flow diagram/pseudo code.

@mdk777

I hear you but I don't really understand your proposal. If you would like a dedicated person to just deal with points and donors, I don't think that's a good thing since I'm afraid it would work very much in a military top down regime which isn't the way to deal with donors.

In fact, if you replace 'person' with 'entity', this forum ( being that entity ) is exactly meant for acting as communication method between PG and donors. Maybe what you want is already here, but in your view it's not working as it should and if you feel that way that alone makes your point valid. So, I think the next thing is asking for 'specifics' on how you think this would be better handled in the future? Would it be enough if the moderators were able to give more answers ( eg should PG include the forum staff earlier on in the decision making process )?

Edit:

I should accompany that question with another one: do you think the forum staff should be the one's to communicate things or did you already imply that you think it's better to have a single person from PG themselves do this?

PG has scientific priorities, we donors have PPD and ease-of-use priorities. I just can't see those ever becoming a perfect match, but I'll settle for close enough.

Well, that is the point I made in the other thread to which you alluded.

If donors are willing to accept the current status, willing to over-look periodic re normalizations: then there is no need to change anything.

Would it be enough if the moderators were able to give more answers ( eg should PG include the forum staff earlier on in the decision making process )?

Edit:

I should accompany that question with another one: do you think the forum staff should be the one's to communicate things or did you already imply that you think it's better to have a single person from PG themselves do this?

In my opinion, the researchers consider; interacting with donors, administrating the benchmark system, posting on the forum... as necessary evils.
I can understand this.
They use this project as a means to an end. They became involved because of their fascination with bio-physics, research, or computational methods, not PR, group dynamics, and Non-profit leadership and administration.
I agree with Napoleon, that the motivation and focus, let alone time allocation, of these two jobs is incompatible.

Do I think the forum adequately replaces a professional administrator? No.

I do think this position would benefit both ends.

Moderators and forum admins. would be given clearer, and timely answers. While researchers would be given greater time to focus on their primary missions.

do you think the forum staff should be the one's to communicate things or did you already imply that you think it's better to have a single person from PG themselves do this?

The forum staff can only communicate what they are given. A full time person would have the time and focus to provide much greater, much more detailed, and much more timely response.

I guess it's more fun to reheat old issues using the same familiar arguments

No, insanely frustrating.

Hence my suggestion to move away from the specific issue, to the structural flaws, the process bottlenecks that hobble the project.