monitoring multiple headless linux clients

[fold_all_the_things]

Hello,
I'm deploying the client on several (at least 10 - 15 machines each with 3x nvidia 2080s) ubuntu 18.04 servers with no GUI. I have it working just fine, machines complete their work units and all that. The problem is it is a huge pain to monitor so many machines. At the moment I just open up a firefox tab for each server on port 7396 via my main computer (it's a windows box) but that's just ridiculous.

Is there a better way to monitor so many machines?

I have tried to use the fahcontrol application on Windows but no luck. I suspect I may be missing something in the configuration of either the servers, the control app or both.

If I add one of the servers to fahcontrol it remains stuck indefinitely in "updating" status with everything greyed out.



here is the configuration file, identical on every folding client:


I suspect it may be the password field on fahcontrol when adding a client, which I am leaving blank since it is not configured on the clients. Does fahcontrol not work if there is no password?


Any tips would be appreciated. If there is any config changes to do on the clients please provide the relevant xml file modifications since I can not edit the settings any other way that I am aware of on a headless server.

Thanks,

[Jesse_V]

I think there are a couple problems. The local FAHClient instance is not configured to allow connections from localhost, which is why the FAHControl front-end can't connect. To set up remote access for the headless clients, add the following lines to /etc/fahclient/config.xml to each headless client so that their configuration looks something like this:

<config>
<allow v='127.0.0.1 10.1.0.0/16'/>
<password v='MyPasswordHere'/>

<!-- all other default other configurations listed below -->
</config>

Then restart FAHClient with "sudo service FAHClient restart". Then in FAHControl on your main system, click the "Add" button in the bottom left corner, type in the IP address of a headless client and the password that you put in the configuration file, and it should connect to the client, assuming that there's no firewall blocks of course.

[fold_all_the_things]

That did it, thanks a lot.