F&H client software lags behind

[jaak ennuste]

Its more than 5 PetaFLOPS today combined power of F&H community. Serious thing! Community has given huge resource. I would feel happier, when F&H developers now give more their donation to enroll following things:

a. Final versions of GPU and SMP clients, installable by not it-graduated. If everyone can install it like "usual windows application", it would open more doors..

b. A2 fahcore for Windows. Currenly majority of Win people pay for energy, getting twice less PPD than Linux users. Lots of power unharnessed there.

c. Instead of console and systray clients to move on 1 version -- get rid of command-line console version. Console clients are really meant for command-line people, may-be 1-2% of PC users.

All these improvements will help to grow voluntary community and make life easier for voluntarees. I have been hearing lot of feedback, that F&H clients is too complex to install compared to other voluntary GRID projects. I'd like to see F&H to rejuvenate and develop together with other software. It's all about client software, not about huge knowledge and success in protein base research.

Jaak, http://estoniadonates.wordpress.com/

[toTOW]

Why use a GUI when a command line can do it ?

Proud to be geek (since MSDOS ... I miss my command line) !

[Ivoshiee]

If anything then the GUI client should be terminated. All effort (SMP, GPU, ...) should be put into unifying everything into one single console client and just provide GUI/tray/SS wrapper for it.

[MtM]

Its more than 5 PetaFLOPS today combined power of F&H community. Serious thing! Community has given huge resource. I would feel happier, when F&H developers now give more their donation to enroll following things:

a. Final versions of GPU and SMP clients, installable by not it-graduated. If everyone can install it like "usual windows application", it would open more doors..

b. A2 fahcore for Windows. Currenly majority of Win people pay for energy, getting twice less PPD than Linux users. Lots of power unharnessed there.

c. Instead of console and systray clients to move on 1 version -- get rid of command-line console version. Console clients are really meant for command-line people, may-be 1-2% of PC users.

All these improvements will help to grow voluntary community and make life easier for voluntarees. I have been hearing lot of feedback, that F&H clients is too complex to install compared to other voluntary GRID projects. I'd like to see F&H to rejuvenate and develop together with other software. It's all about client software, not about huge knowledge and success in protein base research.

Jaak, http://estoniadonates.wordpress.com/

Did you search the forums for simular discussions? I know I been involved in allot of them, and that's not even counting the one's which got pruned because they got out of hand because people couldn't discuss a point without resorting to personal attacks.

a> Do you see the threads about issues with clients at this time? Do you really think they are ready for this?

b> Don't you think if it was possible to have it released already they would just sit on it? And besides, linux clients do not only run a2 projects, they get a1 projects as well where the diffence between windows and linux is only a few %.

c> Console have added usage, running as service being a major one.

I love the enthousiasm, and I'm not the one who should say, think before you post, as noted by some of the wrong advices/conclusions I offerd here on the forums but in this case I will make an exception and say: you didn't think this through/didn't use the search function.

Jaak, I worked on a multi client installer before, and I asked the same things as I thought it wouldn't be hard to do. Fact is, it is not, fact is also, if you give me a few days, I can release something which should covers point a and c somewhat. But, the issue would be, when I do, I would get swamped with requests about support for issues which are not mine to solve. I can set up clients based on things in the faqs, I can not change the behaviour of clients themselfs. I quit working on this, but two days ago I got the project out again ( mainly because my intrest got spiked again due to something I am not allowed to talk about in this section but beta testers probably know what I mean and I think it won't take long before it will go public ).

Installing a client automaticly isn't hard, it's trying to add everything together which made it hard. Statistics gathering, monitoring and controlling clients. Everything, on it's own, I got working. It's not combining allot of modules/classes and forms into one program which is proofing an issue as I didn't design each part to be modular at first. It's taking time, yesterday I spend 5 hours combining classes and fixing errors which where caused by broken relationships since I janked them from 4 diffrent programs. I got +-500 errors fixed and a multitude to go, and then I still got to link everything together again

I pulled a thread earlier about an installer because I asked PG to change the mpich smp installer, which didn't get honord and I felt dissapointed. But in the end, the most compelling issue to not do what you want as a 3rd party dev is because I also feel like I'm opening a can of worms. People might expect things to work out of the box, something I can never guarantee as I aimed to install/monitor/manage smp and gpu2 clients which are still in development. And, I know for a fact, that creating this interface is nothing compared to creating the actual computing cores themself so I feel like the saying you hear allot around here is right especially on this subject: when we're ready you will know.

I got the same hopes and whishes you do, believe me about that, and I expect everyone here to feel the same. But what can we do about it?

Edit:

If anything then the GUI client should be terminated. All effort (SMP, GPU, ...) should be put into unifying everything into one single console client and just provide GUI/tray/SS wrapper for it.

http://img26.imageshack.us/slideshow/pl ... 674cu.smil

It's old images from a pulled thread but as said, I both got more then enough time to try, and I just can't leave it alone.. even if I know there are con's to it.

[John Naylor]

If anything then the GUI client should be terminated. All effort (SMP, GPU, ...) should be put into unifying everything into one single console client and just provide GUI/tray/SS wrapper for it.

I agree... but isn't that essentially how the GUI clients work at the moment? (although they are all seperate clients at the moment)

[MtM]

If anything then the GUI client should be terminated. All effort (SMP, GPU, ...) should be put into unifying everything into one single console client and just provide GUI/tray/SS wrapper for it.

I agree... assuming you mean console clients for all client types then a common GUI app that can be downloaded seperately that works with all clients

John, allot of the things needed I got working ( though messy ). What I need is someone who can coordinate everything, I'm really chaotic and disorganised. But I got classes which are derived from the queue.dat wiki documentation and multiple talks with UF and smoking2000, this class is finished ( only not sure if I want to place filesystemwatchers inside the queue parsing class and events about new frames/project end or to keep those in client control ( a class which is also working 100%, works with console clients both gpu and smp and can run/control them from the systemtray ). I got a class which installs smp clients without user input, but it's not that clean, I got classes which query hardware which isn't that hard though, I can test for cuda devices now as well but I can't do the same with cal/brook as a> I don't have a gpu2 ati card and b> the sdk isn't aimed at vs08, I got a class which parses log files for eue statistics ( not entirely done ), I got classes parsing project info needed to monitor project's and predict ppd but it stores info in xml form and needs to be changed to serialised structure so I can save them as binary files. I can set cpu affinities for multiple clients through a GUI ( no I didn't use boab's program, I fixed my own error why my previous attempt did not work ( and, unlike him I am unable to find matching cores with intel mcm quads ). I can read qeueu entries in a wingui, with help I could add qfix like features in a wingui. I wanted to be able to control multiple of my managers over a network, being able to control networked clients directly. Maybe even creating a coorporate environment management center where it's easier to distribute clients to work stations and as noted in the Salem college thread, collect their work files into a central place so they will not get wiped with ghosting.

Allot of things which I think can be done, but it's so much work I never finished it and guess I never really will on my own.

I asked allot of people for help, but no one really had the time, or maybe did believe I had groundwork done for almost everything I mentioned

[MtM]

Sorry for the negative impression I might have given with my previous post, it's a personal issue in how thing affect me and not always a proper representation of a situation.

To make up for it, or atleast try to , I'm working on the project again, I got uniprocessor support added partly ( need to add it the xml definitions containing download/guide/faq links and add the appropiate class to my client control class ) and I dropped some things I was working on which don't really have added value ( as in, graphical forms for custom balloon tips ) so I can work on things which do matter. I got every error removed but only got 50% of the relinking done. I am focussing on the installer first, and this means the uniprocessor client, and then I need to relink the installer to the tray control ( they where seperate solutions/binaries which I''m now combining in one ).

There are still allot of things I can not do, or which would just take to much time for me to do, but I can provide an win installer and tray application which should work with uni/smp/gpu2 clients besides client specific issues which are not mentioned in the faq's. I made some pages in the past with sourcecode but those are all old and have code which has been fixed and updated allot of times so I will need to remake a proper project page. I will do this when I got the installer/manager finished in the form it was before, just relinked and cleaned out. After that, I hope people will step in and fix errors/add functionality where they see fit as it will be open sourced. I'm not sure about the license type I just would like to be rememberd for the time and effort I did put into this, but I'm very much ok with anyone taking the code and changing anything they see fit and releasing it under a diffrent name. In the end, it's the cause which is the focus, not personal 'glory', speaking of which, I'm sure programmers who take a look at the code will have allot of things they would like to slap me around the head with ( I'm not a programmer by profession, just hobbiest ). But, that should also point out it took me allot of effort to come as far as I have

I think it's been 5 years or so since I first posted some ideas and very very very bad programs with the same idiology on FM, which all where so bug ridden ( and made in vb6 which is allot more limited then .net ) I got thanked for the idea's but not for how I worked it out

This will be better, I promise

[MtM]

Finished adding cpu uniprocessor to all classes tonight, took me longer then expected

I'm unsure which client I would like though, I am using console's for each client since they add service capabilites which I like. But the systray clients offer a pause method the console's don't ( I think since they use the same core's, it's an interface limitation of the console and it can be changed but as with allot of things which can be changed, it probably will not happen or atleast not soon ).

If anyone wonders why it takes so long to change a few forms and link in a new client type, the whole project is 640kb of code files ( no designers and no resx files counted ) and only partially modular, most classes expect other classes to have fixed functions/interactions, I didn't use inheritance allot and so everything is hard linked

[MtM]

[Eveofwar]

Looks very nice so far, keep up the good work

If you need beta testing, let me know.

[MtM]

Not at the moment but thanks. Still need to edit some gui changes in the tray form so affinity and priority is easier managed. Atm it's per client with no way to get a complete overview.



That's an ide shot, can't load the form atm as I don't have configuration files ( which are created by the installer, I started my clients again the old fashioned way ). The worker combobox get's filled with fahcore_xx.exe references ( so one for a cpu client and gpu client, or four with smp clients, priority should speak for itself and affinity is a flowlayout panel which get's multiple checkboxes with cpu core's depending on hardware ( though I'm space limited this way to 8 core's and I want that doubled just in case ). I need to add a global overview page for affinities and priorities where they can be managed all at the same time, in this view only the selected client can be alterd. Then, I can add some other process info such as total cpu time ect ( say, the same info as taskman will show about a process ).

Besides that the tray form is working in regards to client config/management ( though I want to add a scheduler now I added uniprocessor client support ) but I got some classes which track eue rate's and gather data from them and I want to add that to the main view as well. Allot of work

Edit: btw you shouldn't quote a post with images when you're the next person to post in a thread. It's not needed and takes up allot of screen

[John Naylor]

This looks very good indeed! I'll betatest it if you want another tester

[MtM]

Adding an overview page for affinty and priority right now, gf and kid playing outside so no distractions

I also made some notes about other things among which are

• Since with nvidia I now enum cuda devices I can cross check that with wmi to see if sli is enabeld ( public nvapi does not allow this sadly ) and if so prompt the user to disable sli manually ( again, public nvapi does not allow sli topografy enumeration and alteration ).
• Seperate wmi query application so I can debug errors with Ati cards being recognised, as I cant get brook+/cal working I use device name comparisons and I expect this to be troublesome.
• I will use cpu console not tray applications even when I feel it's ashame to loose the built in pause function I can't loose the feeling service mode is more important.
• I'll add a scheduler which will start clients at a predetermined time, but I will not stop clients which are processing wu's ( I will use -oneunit for each wu and restart the client if the end time has not been passed, or and I need some input on this, when I can assume based on previous project durations it will complete a next wu in time which leads to the problem that you can not predict the wu you will get ).
• The monitoring page will be smaller, and I will not add functions to monitor clients which aren't installed by the installer module. I can not compete with monitoring versus a whole community working on fahmon updates for one, and secondly .net seems to read vm's properly with it's built in functions but the queue monitoring and parsing functions I'm using do not support time outs so even when I built in some manual checks to disregard network clients which don't seem to be accesible while already parsing a queue of it's associated log files I am fairly certain I'll get in trouble there ).
• Left Clicking on a clients icon in the systemtray now either makes the console visible or it shows a window with the log of that client, I'll add monitoring and reporting information to this window so I can make the summary window with all clients smaller. Right click context menu will likely stay the same and show start/stop/configure/show work files.
• I want to add checkpoint creation and restore functions, creating archive files for each clients last completed step ( untill it's completed succesfully after which they will be deleted ).
• I have already partially completed classes which check with the queue information about upload status and confirm this with the serverstatus page only if a client reports to have problems uploading. The built in client's method of trying uploads on timed intervals is there with a reason, but I think it might be a good feature to be notified when a wu which is stuck *should* be able to upload so one can try a -sendall or -send x command from the tray menu/management view.
• I'll talk with smoking2000 about qd and qfix, as my class can already read the queue file succesfully I would like to add qfix like features ( I am assuming all it does it checks for the upload status bit and if the wuresults_0x.dat is still in the work folder and if so only changes that bit which will make the client realise the wu has not been send succesfully but I'm not sure ).
• An added reason to not monitor network clients is that with current setup monitoring is real time, based on filesystemwatchers, and as such I can harvest information about core status messages allot quicker then with timed intervals since I don't have to parse allot of lines from the logs. This reduces cpu time compared to timed intervals by 40%. On the other hand fahwatch being used on XS at the time did not use allot of cycles with timed intervals and it used the old method of parsing ( and still had lower cpu usage monitoring clients then other popular monitoring applications ( it didn't display eta/ppd info out off choiche not because the info couldn't already be gotten from the gatherd info ).
• Speaking off ppd measurement, I will add an 'actual' ppd field which will better show ppd as should be expected on the official stats updates. To do so, I'll use the queue entries information from succesfully completed wu's which consists of download time and upload succes ( which is only available with real time monitoring hence again, no network clients ). This ppd should be more accurate then even fahmon's effective rate afaik, though it will show lower ppd when EUE's occur as I can not estimate the partial credit awarded ( if there is a formula for this, I would like to hear it? ).
• My project browser atm links to url's with the actual description, but I noticed there is a download link for all project descriptions so in the future I will download that file when a project is running for which the description is not known just as it downloads new psummary information with unknown projects.

The longer that list get's the more I look up to doing it but I'll just continue one step at a time trying to not get distracted as much.