PSSH for Parallel Tasks
Have you been caught in the situation where a project you thought would be simple and low maintenance turned out to be popular and now you are faced with the problem of logging into servers and doing repeated tasks because you haven’t had enough ‘free’ time to automate some of your processes? A quick bandaid could be
pssh (Parallel SSH).
PSSH Quick Intro
Pssh lets you log into multiple servers at the same time and run commands. Since pssh actually SSH-es to your servers, you need to have SSH working. Although it supports using passwords to log into servers, I would highly recommend using SSH keys.
A couple of things to be mindful of. Out of the box, pssh does not work with SSH keys that have passphrases, but the code can be easily patched to fix that. You can find the bug report and fix here. If you do not want to tinker with the code then it would be better to stick to ssh keys without a passphrase. Also, as mentioned earlier, it is possible to pass a password when logging into a server but it is kind of an issue passing a password later if you need privilege escalation. The work around could be logging into the server as root ( with an SSH key) or using a user that has been configured to have no SUDO password. These methods have security implications!! Use a method that fits your security requirements. The choice is yours. Now that we have that disclaimer out of the way lets proceed.
Installing on Debian/Ubuntu is just:
sudo apt install pssh
on MacOS with brew is:
brew install pssh
I would advise having a look at all the options available (
pssh --help) as some of the options might be useful if you have some very complex scenarios. Some useful once I have found are:
-i: To print out the output and error for the server.
-h: File containing the list of hosts to connect to.
-t: Timeout in seconds
-p: Maximum number of parallel threads
-x: Extra command line arguments
-O: Options from SSH.
-I: Read from standard input and send as input to SSH.
-A: Ask for a password
Create an ordinary text file with the list of hosts you want pssh to connect to. Comments ( lines starting with #) and empty lines are allowed. The server entry format is
[user@]host[:port] . Just a reminder, if the user running the pssh command is the same user you want to run commands on the server then you can omit
user in the server entry. If the SSH port to connect to the server is 22 then port can be omitted as well. Also,
host can of course be a hostname or an IP address. An example host file would look like
# Kamailio servers 192.168.86.110 email@example.com # Media proxies firstname.lastname@example.org 192.168.100.10
To segregate servers you would need to have multiple host files. So for example you could have one host file with all servers and then host files for server types, e.g. SIP servers, media servers, web servers etc.
Example 1: Checking the date
pssh -i -h hosts.txt 'date'
 02:30:18 [SUCCESS] 192.168.86.110 Sat Feb 27 12:30:18 GMT 2021  02:30:18 [SUCCESS] email@example.com Sat Feb 27 12:30:18 GMT 2021
Example 2: Checking a server behind a virtual IP address
SSH keeps a record of hostnames and their public keys. When you have 2 servers behind the same hostname, SSH might suspect a possible man in the middle attack because of the different keys. To bypass this check, we use the SSH option
Now for example lets reload the dispatcher table on multiple Kamailio servers:
pssh -i -p 2 -h hosts.txt -O StrictHostKeyChecking=no 'sudo kamctl dispatcher reload'
Example 3: Running a bash script on the servers
To run a more complex script you can first transfer it with
pscp (Parallel SCP) which comes with pssh. The steps would be:
pscp -h hosts.txt script.sh /home/ivan/
Make executable and run:
pssh -i -p 2 -h hosts.txt -O StrictHostKeyChecking=no 'sudo chmod +x script.sh && ./script.sh'
Pssh is useful not only for quick checks and fixes but also to pull data from servers especially if you do not have some service monitoring in place. For example quick checks on free memory, call volume, restarting services etc. It is possible to go steps further and create aliases for some pssh commands. But if you notice your setup is getting clunky and that you are over using pssh then maybe the ‘bandaid’ is no longer the tool you need and you finally need to make time to automate the process. Consider DevOps tools like Ansible or Chef.