Ad-hoc multi-server management with pdsh and friends

2 points by lcall 7 hours ago

Back when I was occasionally managing many servers, I would have to run the same adhoc command on all of them and see which ones succeeded identically to the others, and which ones failed, for further follow-up. These commands were really useful (assuming ssh keys are set up already, for logins). (I didn't know about these until a coworker told me, so I thought I'd pass it on.):

To send the same command to multiple servers, use pdsh: https://linux.die.net/man/1/pdsh

To collect all the results and show which ones are the same or different, use dshbak (i.e., "pdsh <parameters including servers>|dshbak"): https://linux.die.net/man/1/dshbak

Similar things, sometimes more convenient but less efficient for a large number of servers, are to use the konsole terminal program and link multiple window tabs together so the same typed command goes to all, and quickly view the results across the tabs; or to use tmux and send the same commands to multiple windows (possible useful "man tmux" page terms: link-window, pipe-pane, related things to those, activity, focus, hooks, control mode).

And others that I haven't used but which also look possibly interesting for platforms where pdsh and dshbak might not be available (like OpenBSD at least):

- https://github.com/duncs/clusterssh/wiki (available on OpenBSD as a package)

- https://www.gnu.org/software/parallel/ (also available as a package on OpenBSD 7.6: named "parallel-20221122"; might relate to "pdksh")

Maye things like Puppet or CFEngine have similar capabilities where those are in place; I haven't used them enough to know.