Create an army of Salt minions on DigitalOcean
This guide is for two audiences:
- You want to learn Salt.
- You plan on having a few DigitalOcean droplets and want an easy way to manage them.
We're going to combine the simplicity of Salt with DigitalOcean's snapshot and image feature. For great justice.
Sooner or later you're going to have a bunch of DigitalOcean droplets doing excitingly different things.
Wouldn't it be cool if you could run
sudo apt-get dist-upgrade -y on all of them in one shot?
Yes it would. Let's learn config management.
Why not salt-cloud?
You can do this with another tool called salt-cloud. It uses the DigitalOcean API to provision droplets. It runs a complex script called salt-bootstrap on all the new machines.
We're not going to do that.
Let's be clever with our own template, because:
- It's pretty simple. Salt-cloud is overkill for a small number of servers.
- You will have hands-on control over what's happening.
- You will learn about Salt and Images! Learning is fun.
Here's the plan
- Create a master server. You could use an existing machine to save a few beer tokens.
- Create a minion - thank you Salt developers for using this word - and test it.
- Tweak and snapshot the minion as a template for future droplets.
- Execute a complex victory dance.
The Salt chain of command
You need a single Salt master - a package called
salt-master - in order to issue imperious commands to the unwashed, quivering minions.
Your minions have a client installed -
salt-minion - and are told where the master is. The minions ask for permission to join up, the master says yes, and off we go.
You can install a minion client on the master as well, if you want to be perverse. Implications of doing this:
- The master server can be managed along with everyone else. This is handy.
- You run a small risk of screwing the master when you accidentally send a
sudo rm -rf /*to all minions. Hint: Don't do this.
1. Create the master
Create a new Ubuntu droplet called
configmaster and log in as root. At create time, specify an SSH key because we are adults.
Let's set up a new user and give them sudo privileges:
# adduser salt # # (set a password and default all the other questions) # visudo
Make the sudoers file look like this and save it out:
# User privilege specification root ALL=(ALL:ALL) ALL salt ALL=(ALL:ALL) ALL
That's our dirty root-werk done. Log out and SSH back in as
salt using your new password.
Install the good stuff
As user salt:
$ sudo echo "sudo is working, I feel like an internet hero" $ sudo add-apt-repository -y ppa:saltstack/salt $ sudo apt-get update $ sudo apt-get install -y salt-master $ service --status-all 2>&1 | grep salt [ + ] salt-master $ sudo salt-key -L Accepted Keys: Unaccepted Keys: Rejected Keys:
Your master is now done. Told you it was simple.
What have I created?
Minions ask for permission to join the network by sending over a key. The
salt-key -L command means '(L)ist all the minions currently connected'.
- Minions who are connected but remain patiently waiting for a decision are listed as Unaccepted.
- Notice that a master can choose to Reject a minion that has the wrong sort of haircut.
For debug purposes, the salt-master log is at e.g.
cat /var/log/salt/master. With default settings, the log will be empty unless something is broken.
By default the Salt master listens on ports 4505 and 4506 on all interfaces (0.0.0.0).
2. Set up a minion
We're going to do the absolute minimum to bootstrap this new server. Once it's hooked up, all work can be done remotely from the master.
Create a new droplet called
salt01. Log in as root and:
# add-apt-repository -y ppa:saltstack/salt # apt-get update # apt-get install -y salt-minion # service --status-all 2>&1 | grep salt
There's one item of config business. We need to set the location of the master (an IP address for the moment). Edit the file as below and set it:
# vi /etc/salt/minion # # set the appropriate line to read 'master: <your configmaster IP>' # service salt-minion restart
You can understand what the minion is doing by watching the log:
# cat /var/log/salt/minion
Have a look now. You can see that at first, the minion couldn't find a master at the default hostname
salt. Once this was fixed and restarted, we see
[salt.crypt ][ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
This means the minion is connected but the master hasn't yet Accepted the key.
I for one welcome our new minion
Back on the master (as user
salt), you can see a minion wants to register:
$ sudo salt-key -L Accepted Keys: Unaccepted Keys: salt01 Rejected Keys:
So let's be a nice doorman and let them in:
$ sudo salt-key -y -a salt01 The following keys are going to be accepted: Unaccepted Keys: salt01 Key for minion salt01 accepted. $ sudo salt-key -L Accepted Keys: salt01 Unaccepted Keys: Rejected Keys:
salt-key -A will accept all keys currently waiting for a decision.
Test your powers
Minions can execute functions from execution modules that are defined by Salt. They have names like
The full module list is pretty exhaustive. Naturally you can add your own stuff.
On the master:
$ # You can address a specific minion: $ sudo salt salt01 test.ping salt01: True # or address all minions, to e.g. dump their IPs: $ sudo salt '*' network.ip_addrs salt01: - 123.456.78.123
You can get away with not escaping the
* asterisk but it's probably not a great habit.
cmd.run function allows arbitrary shell commands:
$ sudo salt '*' cmd.run 'uname -a' salt01: Linux salt01 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
So we've created a minion. Do we want to do this crap every time we add a new one? Nope. Let's create a DigitalOcean image.
3. Make a cookie cutter
Now we understand what's going on, we're going to make a template out of
salt01. This requires a spot of minor surgery to remove its identity.
I am not a number, I am a free man
salt-minion service starts up, it generates an identity based on the hostname. This
minion_id, along with a new key are cached to disk.
We need to remove that for our template - credit to this post for the method. On your
# service salt-minion stop # rm /etc/salt/pki/minion/minion_master.pub # rm /etc/salt/pki/minion/minion.* # cat /dev/null >/etc/salt/minion_id
If we didn't do this, new machines created from template would all have a cached identity of
salt01. We would see this on startup:
# salt-minion -l debug [DEBUG ] Reading configuration from /etc/salt/minion [INFO ] Using cached minion ID from /etc/salt/minion_id: salt01 ...
Oh shit! Identity theft. An army of clones worked for The Empire, but not for us.
Don't confuse the master
Since we just blew away salt01's identity, we need to tell the master not to expect that key any more:
$ sudo salt-key -D -y
-D means '(D)elete all minion keys'. The salty D.
Make an image!
DigitalOcean lets you take a snapshot, which creates an image. Images can be used to create new droplets just like the old one, except for a brand new hostname.
First we need to power off
And in your DigitalOcean web console, snapshot it. Call the image something like 'Salt minion template'.
When the snapshot finishes, create a new droplet
salt02 from the image.
When salt02 starts up everything will happen by magic:
- It already has the
salt-minionpackage installed and config ready to go! Great.
- It will generate its new identity.
- It will connect to the master.
Get up salt01, you're fine
After the snapshot,
salt01 will automatically be powered on, and will rejoin exactly as above using its own hostname.
4. With two minions I am unstoppable
We're all done. We have two fresh minions asking to join the master, so let them in:
$ sudo salt-key -L Accepted Keys: Unaccepted Keys: salt01 salt02 Rejected Keys: $ sudo salt-key -A -y The following keys are going to be accepted: Unaccepted Keys: salt01 salt02 Key for minion salt02 accepted. Key for minion salt02 accepted. $ sudo salt '*' cmd.run 'uname -a; uptime' salt02: Linux salt02 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux 18:37:07 up 5 min, 1 user, load average: 0.01, 0.03, 0.02 salt01: Linux salt01 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux 18:37:07 up 8 min, 1 user, load average: 0.00, 0.01, 0.01
Whenever you need a new droplet, use your custom image and watch it automatically come under control of your config management network.
You're free to update the image and make a new snapshot whenever you feel like it - just remember to clean the identity beforehand.
But you have not mentioned *splutter*
DigitalOcean takes care of the hostname and MAC address on new servers. You can confirm this with e.g.
$ sudo salt '*' network.hw_addr eth0 salt02: 04:06:28:41:19:01 salt01: 04:04:29:22:fe:11
Don't forget security! You'll want to take basic security steps like hardening your SSH settings, maybe install
ufw and all that good stuff.
The good news: you can set it up once in Salt and use the master to push to all new servers...