Please allow smaller WUs on FAHClient

[JimboPalmer]

And to be more extreme, why not making all WUs 10 times smaller as it would bring more flexibility, more folding, less dumping, shorter deadlines and quicker reassignment etc?

This will make the stats database 10 times larger going forward, and it will make recovery of the stats at least 10 times slower. (My guess would be 30 times slower, but the is just SWAG)

This will make both the official stats and 3rd party stats mush slower and more expensive.

[bruce]

It might bring more flexibility but it will NOT create more science. If anything, I suspect it would INCREASE the number of WUs that would be dumped because completing a particular WU would be less important to those who care about their PPD.

[MeeLee]

With hardware evolving, and proteins getting more and more complex, I don't think that WUs will be decreased in size by much over what's already here.

The smallest ones currently running, take about 1 hour to finish with an RTX 2060, and I'm ok with that size being considered as a 'small' unit.
1 hour units, are easy to plan life around, and it's those that I'd rather see run on my portable device.
As cards get faster this 1 hour unit size, will be done in even less time.

@ Bruce: It would also take a lot of small (1 hour) WUs to dump, to compete with a single larger (say, 4-8hours) WU.
So, while more WUs will be dumped, I presume less overall data will be dumped.
And if 2 or 3 smaller WUs are dumped, the remaining 4 to 5 parts that would have made the larger WU, can much quicker be reassigned to other people (parallel processed), as with larger WUs the server has to wait for the WU to be fully processed (Serial processing).
Which is why you don't really want to give very large WUs to just anyone out there.

I think you'd have to outweigh what is more important, and prevalent,
Who will be most satisfied?
People occasionally (or frequently) dumping WUs for score (they don't get any score for a dumped WU, and I think they're the fewer of them), or the flexibility for people who are folding on and off (the majority of them)?
Besides, I wouldn't know who will make the time to dump a lot of WUs for points, especially if they've been folding for weeks or months on end.

[bruce]

And if 2 or 3 smaller WUs are dumped, the remaining 4 to 5 parts that would have made the larger WU, can much quicker be reassigned to other people (parallel processed), as with larger WUs the server has to wait for the WU to be fully processed (Serial processing).
Which is why you don't really want to give very large WUs to just anyone out there.

WUs will not be variable sized. If one WU is dumped, it will be reassigned and completed before the next WUs are ready for distribution and they'll still be small. Each project has a single size, and if you are ever given the opportunity to choose, you will effectively be choosing from different lists of projects. Just like projects that are currently classified as beta or advanced, you will be expressing a PREFERENCE and if there are no projects matching your choice, you'll be given one which doesn't match your choice.

[ifolder]

Each project has a single size, and if you are ever given the opportunity to choose, you will effectively be choosing from different lists of projects.

You said that each scientist sets the size of the WUs according to some criteria. This means that the size of the WU can quite freely be chosen.
So by asking all projects to set small WUs that all bring the same amount of points, you immediately solve the problem of people dumping for points and you can shorten all the periods (expiry, reassign...).

I still think it could increase science because of the possibility to only fold during the night in summer rather than not fold at all for a few months. Also gamers could then use their gaming GPUs to fold because they could imagine to launch a short (<1-hour) WU from time to time when when they don't play but won't accept to have their GPUs occupied for hours...

With hardware evolving, and proteins getting more and more complex, I don't think that WUs will be decreased in size by much over what's already here.

If my understanding is good it is the number of steps that makes the size (in time) of the WU. So if you have a much more complex protein, you just reduce the number of steps and you get a shorter WU.

This will make the stats database 10 times larger going forward, and it will make recovery of the stats at least 10 times slower. (My guess would be 30 times slower, but the is just SWAG)
This will make both the official stats and 3rd party stats mush slower and more expensive.

Are there other 3rd party accessible stats than: https://stats.foldingathome.org/api ? Because on those ones besides points you only get the number of WUs per donor and no detail about each WU itself. So the number of WUs doesn't seem to impact the size of the stats.

[MeeLee]

They can decrease the size, yes, but like JimboPalmer wrote, it would add to network bandwidth, and server cost to run smaller units.

Which is why I think the current smallest (1 hour) WUs, are about as small as you'd get small WUs at.
Unfortunately, for older cards, this 1 hour WU, takes longer.
The GTX 1050 ti would do this at roughly double the time an RTX 2060 does the job at (my guesstimate), and a GT 1030 does it at ~4 hours.
But Pascal cards aren't the future. They'll eventually phase out (already are phasing out) and it's not where F@H is going.
Even Turing cards will eventually phase out, so if we're to set a standard, I think it has to be set for modern hardware today, with eyes on the future.
Give or take 2-3 years from now and a new set of GPUs will come out, and do this small '1 hour' WUs in 30-ish minutes of time.
That's not bad at all!

[ifolder]

They can decrease the size, yes, but like JimboPalmer wrote, it would add to network bandwidth, and server cost to run smaller units.

I would like to see figures that prove these assumptions. Moreover like GPUs, server performance evolve with time.

Which is why I think the current smallest (1 hour) WUs, are about as small as you'd get small WUs at.
Unfortunately, for older cards, this 1 hour WU, takes longer.
The GTX 1050 ti would do this at roughly double the time an RTX 2060 does the job at (my guesstimate), and a GT 1030 does it at ~4 hours.
But Pascal cards aren't the future. They'll eventually phase out (already are phasing out) and it's not where F@H is going.
Even Turing cards will eventually phase out, so if we're to set a standard, I think it has to be set for modern hardware today, with eyes on the future.
Give or take 2-3 years from now and a new set of GPUs will come out, and do this small '1 hour' WUs in 30-ish minutes of time.
That's not bad at all!

Sure, the main idea is "Make smaller WUs for flexibility and all the reasons previously mentioned" and make the meaning of "small" evolve in time with the evolution of GPU power.

[JimboPalmer]

They can decrease the size, yes, but like JimboPalmer wrote, it would add to network bandwidth, and server cost to run smaller units.

I would like to see figures that prove these assumptions. Moreover like GPUs, server performance evolve with time.

From Wikipedia, on Sorting:

"Sorting algorithms are often classified by:

Computational complexity (worst, average and best behavior) in terms of the size of the list (n). For typical serial sorting algorithms good behavior is O(n log n), with parallel sort in O(log2 n), and bad behavior is O(n2). Ideal behavior for a serial sort is O(n), but this is not possible in the average case. Comparison-based sorting algorithms need at least Ω(n log n) comparisons for most inputs.

Memory usage (and use of other computer resources). In particular, some sorting algorithms are "in-place". Strictly, an in-place sort needs only O(1) memory beyond the items being sorted; sometimes O(log(n)) additional memory is considered "in-place"." - https://en.wikipedia.org/wiki/Sorting_a ... algorithms

So a sort with 10 times the new data to be inserted will take 10 * log10 as long on average, worst case 100 times as long.

The memory needed to sort 10 times the data is 1 times the memory at best, with 10 * log 10 being worst case.

[bruce]

If my understanding is good it is the number of steps that makes the size (in time) of the WU. So if you have a much more complex protein, you just reduce the number of steps and you get a shorter WU.

FALSE. That's only one of the factors involved.

Some projects are short because they have fewer atoms and others a can be short because they simulate a shorter number of steps. While the goal is to make all WUs earn the same PPD, that's not possible depending on what GPU you're running. Some GPUs are more efficient on proteins with larger/smaller numbers of atoms than other GPUs. To compensate, the number of steps or the points can be adjusted, but the same adjustment doesn't affect all GPUs equally.

If my GPU is twice as fast as yours on project X, it will not be twice as fast on project Y unless those projects have the same internal atomic complexity numbers of atoms.

[bruce]

With hardware evolving, and proteins getting more and more complex, I don't think that WUs will be decreased in size by much over what's already here.

The percentage of large proteins is gradually increasing, but there are still studies needing to be done on smaller proteins.

... if 2 or 3 smaller WUs are dumped, the remaining 4 to 5 parts that would have made the larger WU, can much quicker be reassigned to other people (parallel processed), as with larger WUs the server has to wait for the WU to be fully processed.
Which is why you don't really want to give very large WUs to just anyone out there.
Each project is assigned randomly. Currently you may get a large or a small protein and you do not control the selection. Moreover, if you dump a small protein, your next assignment might from a small project or a large project -- without a bias. Those 4 or 5 parts will still be randomly assigned.

The servers do make an effort to prevent WUs from being dumped and to make sure that projects belonging to each scientist get their share of assignments. Speaking from the perspective of a project owner, if I create a project with small WUs, I should be entitled to an equal share of donors working on it as if my project were created as large WUs.

[ifolder]

They can decrease the size, yes, but like JimboPalmer wrote, it would add to network bandwidth, and server cost to run smaller units.

I would like to see figures that prove these assumptions. Moreover like GPUs, server performance evolve with time.

From Wikipedia, on Sorting:
"Sorting algorithms are often classified by:
Computational complexity (worst, average and best behavior) in terms of the size of the list (n). For typical serial sorting algorithms good behavior is O(n log n), with parallel sort in O(log2 n), and bad behavior is O(n2). Ideal behavior for a serial sort is O(n), but this is not possible in the average case. Comparison-based sorting algorithms need at least Ω(n log n) comparisons for most inputs.
Memory usage (and use of other computer resources). In particular, some sorting algorithms are "in-place". Strictly, an in-place sort needs only O(1) memory beyond the items being sorted; sometimes O(log(n)) additional memory is considered "in-place"." - https://en.wikipedia.org/wiki/Sorting_a ... algorithms
So a sort with 10 times the new data to be inserted will take 10 * log10 as long on average, worst case 100 times as long.
The memory needed to sort 10 times the data is 1 times the memory at best, with 10 * log 10 being worst case.

Sorry but I think your demonstration isn't pertinent. The sorting is indeed performed on the users not on the WUs. With 10 times more WUs you'll still sort the same number of users. Only the values of the sorting keys (number of WUs) are going to be 10 times larger. Unless of course if there is some other API than the official one I mentioned that gives statistics up to the WU level???

[ifolder]

With hardware evolving, and proteins getting more and more complex, I don't think that WUs will be decreased in size by much over what's already here.

The percentage of large proteins is gradually increasing, but there are still studies needing to be done on smaller proteins.

... if 2 or 3 smaller WUs are dumped, the remaining 4 to 5 parts that would have made the larger WU, can much quicker be reassigned to other people (parallel processed), as with larger WUs the server has to wait for the WU to be fully processed.
Which is why you don't really want to give very large WUs to just anyone out there.
Each project is assigned randomly. Currently you may get a large or a small protein and you do not control the selection. Moreover, if you dump a small protein, your next assignment might from a small project or a large project -- without a bias. Those 4 or 5 parts will still be randomly assigned.

The servers do make an effort to prevent WUs from being dumped and to make sure that projects belonging to each scientist get their share of assignments. Speaking from the perspective of a project owner, if I create a project with small WUs, I should be entitled to an equal share of donors working on it as if my project were created as large WUs.

Bruce, is there a paper that describes in detail how all those things work?

[bruce]

Not that I know of, though there are a number of one-sentence comments in various papers which describe a single aspect of a complex process.

[Theodore]

It'll take a while for software to be optimized, to run multiple smaller atom simulations in one GPU, to keep it running close to 100%

[bruce]

I am not aware of any plans to optimize CPU WU to run muiliple WUs concurrently, but you can do it manually by constructing muliple slots. Similarly, I am not aware of any plans to allocate N shaders to one WU and (100%-N) shaders to an independent GPU project and manage downloading/uploading them independently -- especially since N is not a fixed number. If it were possible to do it manually AND if you could choose which projects you wanted to run, you'd have a difficult time managing every download -- and software would also have a difficult time doing the same job.

Running one project is ALWAYS going to be faster than running multiple WUs on the same piece of hardware so FAH is fundamentally opposed to that concept.