Suggestion: download new WU before current one finishes

[Rum@NoV]

I've did a little search on this subject, but I don't think it has been asked before.

My suggestion to improve the fah-client: download a new WU before the current one finishes.

Why? Uploading a SMP WU can take some time, depending on the upload speed of your internet connection. During the upload, the CPU/GPU is idling (which is a thing every folder hates I guess )

Some numbers (based on 24h/24h folding):
- A completed P2671 mesures 49203991 bytes (= 48050 kilobyte or 46,92 megabyte)
- Upload time on my connection (20Mbit down / 1Mbit up) = 48050 / 128 kilobyte (128 = optimal situation, my average upload speed is 106 kilobyte!) = 375,39 seconds (6 minutes 15 seconds) upload time.
- If a CPU completes two of these WU/day the upload and idle time would be at least 12 minutes 30 a day. Per year the upload and thus idle time would be 74 hours 31 minutes and 15 seconds, which is more then 3 days!

To optimize the efficiency (more work completed with the same hardware!!) of the client I suggest the client starts downloading a new WU, when for example the current WU is completed for 95%. The client then continues working and uploads the finished WU in the same time.

What do you guys/girls think, is this possible? Is there a chance this enhancement might be included in the next client update?

[Zagen30]

Maybe there hasn't been much discussion of that topic on this edition of the forums, but I know for a fact that back in the good old days (way before I started, mind you) there were lengthy debates over the subject of queueing work units. The answer has always been no, and it's not going to change. If we get anything resembling queueing, it'd be downloading a new WU when the client goes to upload a completed WU, but I'm not sure if that's even being considered. The Pande Group doesn't want to risk losing multiple WUs to a computer that suddenly stops folding for whatever reason, as losing one sets back the project a fair amount as it is.

[toTOW]

This has been suggested already, and explained why that will probably never happen : search on the forum similar topics.

[Pick2]

The layman's version is:
Each finished WU generates the next WU in the chain. If everyone downloaded a new WU before finishing the current WU , We would need twice as many WU chains available. Lots of people would have to wait for their next WU , negating any gains. Doubling the number of WU chains running would make each series take twice as long to complete.
That's My story , and I'm not going to argue the point.

[codysluder]

. . .
That's My story , and I'm not going to argue the point.

An excellent summary.

From the PG perspective, your suggestion would SLOW THINGS DOWN. They count the time from the moment a WU is downloaded until it is uploaded, and you're extending that time.

Take a look at the experemental bonus program that's currently being tested. If you were participating in that program, your suggestion would reduce the bonus, which is another way that the Pande Group is telling us that in many cases the scientific value of our contributions may not maximized simply by maximizing CPU use.

[jrweiss]

There is merit in one piece of Rum's suggestion: When the FINISHED_UNIT message is sent and the client starts the upload/download process, the client could download the new unit BEFORE uploading the finished unit. Delays waiting for the next WU will be reduced, and the new WU can be crunching away while the client waits for the results server to respond.

In many cases a new WU is downloaded while the client is waiting for the upload to be accepted, so the concept works now. The only change would be to make it a default behavior instead of a contingency behavior.

[Rum@NoV]

Thanks for the replies.

I understand that it is important for the PG to receive the results asap and my suggestion would slow things down if one isn't folding 24/24.

But what jrweiss suggests isn't that possible? Instead of uploading the finished WU, the client downloads a new WU first (after receiving the FINISHED_UNIT message) and then immediately uploads the finished result. Downloading a new WU generally goes a lot faster then uploading one so we could win a lot of time and PG will have their results as fast as they do now.

[GTron]

Okay, maybe I spent WAY too much time staring at a tiny datascope screen in a previous decade. However:

We are talking about SMP (as in symmetric MULTIprocessing) clients communicating with the Stanford servers over what is normally a FULL DUPLEX comm link. Why should either the upload or download need to be sequenced after the other? They could happen CONCURRENTLY. Provided, of course, the clients and servers were coded accordingly and that is the real issue. While I would like to see that feature in a future version of the software, it will not stop me from folding now.

Just my $.02

Greg, Folding On

[PennyPincherP]

I have 256Kbs upload speed on DSL. It takes 45 mins to upload -bigadv results according to my logs. Download speed is 3Mbps and takes 2 mins 15secs. According to the logs, nothing is done in parallel. That's fine for the uniprocessor clients but isn't very efficient for smp, especially for huge WU's. Assuming the upload could be done in parallel with the core, downloading the next WU and reinitiating the core before the upload makes sense. The upload process would of course would need a higher priority than the core. Error handling and queue maintenance would be a lot trickier as well. A 2 minute delay in getting results after 3 days is acceptable if it means the getting 45 more minutes of computation.

[whynot]

As you already know tear has some success with this issue. While I highly appreciate his success, I would somewhat argue with his methodology. However since I don't have that issue (my upload is ~x100KB each, sometime less) I won't comment.

[7im]

I read in vijay's blog where they just got $100K for system upgrades, and should help with some issues. But at times, in the past, the number of connections to Stanford was limited, and servers were over loaded. When that happens, they would not be able to handle twice the connections to do simultaneous uploads and downloads. And when a choice needs to be made, they'd prefer to get the completed data first.

I'm not saying that's right or wrong, just how it is... or was back in the day of dialup, when uploads and downloads were almost the same speed. And a lot of what we would still like changed today (perpetual suggestions like this) are legacy hold-overs from days gone by.

This type of concurrent uploads/downloads would take a serious rewrite of the client and server codes. And major over-hauls run the risk of being very disruptive to the project. For example, look how quickly people post when a single server goes down, or a single work unit fails to upload. Now imagine if a client / server rewrite had a small glitch that prevented 100,000 users from getting new work for a few days until it was fixed. Not pretty (examples of this have been seen in the GPU and PS3 clients in the past, so not unheard of...).

I'm not saying they shouldn't improve the process, but I am saying it's a very risky change for not much reward. Some would say the available development time they have is better spent on making a better SMP(2) client for Windows, and optimizing the GPU clients for the latest hardware. And then work in changes like this gradually over time.

[k1wi]

My thinking is along the same lines as Penny Pincher;

There are two problems,
1. limited concurrent connections to the server,
2. Asymmetric connections DL/UL ~10:1, exacerbated by download files ~4MB and upload ~25MB

Why not just change the code to swap when the download and upload occurs and end up with the download occurring immediately before the upload begins? That way the same number of connections are being made, and there is less than a ~3 minute difference in reporting back, but the donor's CPU is being utilised better?

[PennyPincherP]

If concurrent connections is the problem. Maybe Stanford could look into considering offsite hosting of their result servers where they have higher capacity. The aggregated results could then can be uploaded with high speed bursts back to Stanford as appropriate. Maybe you could get some site to donate this service or do it at nominal cost. If it's a question of hardware, I have at least a dozen 8 core boxes that I'm not using that I could donate.

[7im]

Concurrency would have been a problem in the past, but I don't know what the current state is after all the upgrades they've done, and are still doing.

As for just swapping the order of the download/upload, that goes back to what I said about making changes. No change is a small change where reprogramming 400,000 clients, and 25+ servers is concerned.

[jrweiss]

As for just swapping the order of the download/upload, that goes back to what I said about making changes. No change is a small change where reprogramming 400,000 clients, and 25+ servers is concerned.

I have to disagree here.

They would only be reprogramming a dozen or so clients, and I suspect the source code logic in that segment of the client is identical for all of them. The number of servers is totally irrelevant. All it takes is to move the initial download connection to just prior to the initial upload connection. All the remaining logic remains intact, including the timing for waits between subsequent attempts of either action.

As it is, an unsuccessful upload attempt can delay the download of the next WU for hours, and the start of the next WU in the server queue is not at all affected by the WU results awaiting upload. We know this because there will eventually be a download attempt even if the upload does not complete.