SaltStack and GPG encryption

Why use encryption in Salt Pillar?

Salt pillar can use GPG encryption so you can store your secrets encrypted at rest in pillar. Those encrypted secrets can then be called by other salt states and decrypted using the key pairs created with GPG(2) tools. Only the minions assigned to that pillar data will have access to decrypt the secrets you create. Further, the encrypted data can be placed in the same YAML file as encrypted secrets, to reduce unnecessary complexity or pillar sprawl.

RNG tools on RHEL and Why?

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-encryption#sec-Creating_GPG_Keys

On the server that you want to create the GPG keys, you will want to install rng-tools since VM’s may have issues generating the keys. The above referenced RH article explains why.

Here’s how:

[root@server10 salt]# yum install rng-tools
[root@server10 salt]# yum install rng-tools
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
epel/x86_64/metalink                                                                       |  15 kB  00:00:00
epel                                                                                       | 4.7 kB  00:00:00
ius                                                                                        | 2.3 kB  00:00:00
pgdg96                                                                                     | 4.1 kB  00:00:00
rhel-7-server-extras-rpms                                                                  | 2.0 kB  00:00:00
rhel-7-server-rpms                                                                         | 2.0 kB  00:00:00
rhel-7-server-satellite-tools-6.2-rpms                                                     | 2.1 kB  00:00:00
rhel-server-rhscl-7-rpms                                                                   | 2.0 kB  00:00:00
salt-latest                                                                                | 2.9 kB  00:00:00
(1/4): epel/x86_64/updateinfo                                                              | 874 kB  00:00:01
(2/4): ius/x86_64/primary_db                                                               | 211 kB  00:00:00
(3/4): pgdg96/7Server/x86_64/primary_db                                                    | 185 kB  00:00:00
(4/4): epel/x86_64/primary_db                                                              | 6.2 MB  00:00:01

Resolving Dependencies
--> Running transaction check
---> Package rng-tools.x86_64 0:5-11.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved
==================================================================================================================

Package                  Arch                  Version                   Repository                         Size

==================================================================================================================
Installing:
rng-tools                x86_64                5-11.el7                  rhel-7-server-rpms                 35 k
Transaction Summary
==================================================================================================================

Install  1 Package

Total download size: 35 k
Installed size: 68 k
Is this ok [y/d/N]: y
Downloading packages:
rng-tools-5-11.el7.x86_64.rpm                                                              |  35 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : rng-tools-5-11.el7.x86_64                                                                      1/1
Uploading Package Profile
  Verifying  : rng-tools-5-11.el7.x86_64                                                                      1/
 
Installed:
  rng-tools.x86_64 0:5-11.el7
 
Complete!

Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager

Here’s what you can run to generate enough random whatevers to let the VM generate the keys

[root@server10 salt]# rngd -r /dev/urandom

GPG-Agent and setting the homdir

Ok, now we have to set the homedir for the gpg agent daemon. If you want to use a non-standard directory to put your keys that you will be creating, you will want to do this next step. Otherwise, the gpg agent gets mad and won’t connect during the key creation, and it will all fail. So run this to set the homdir in the daemon for gpg-agent.

Note: I ran this from the directory I wanted to use as my directory to put the keys I was creating (/etc/salt/gpgkeys)

[root@server10 gpgkeys]# gpg-agent --homedir="$(pwd)" --daemon
gpg-agent[16394]: directory `/etc/salt/gpgkeys/private-keys-v1.d' created
GPG_AGENT_INFO=/etc/salt/gpgkeys/S.gpg-agent:16395:1; export GPG_AGENT_INFO;
gpg-agent[16395]: gpg-agent (GnuPG) 2.0.22 started
[root@ifop0510 gpgkeys]#

Create the key pair

[root@server10 gpgkeys]#
[root@server10 gpgkeys]# gpg2 -v --gen-key --homedir /etc/salt/gpgkeys
gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 2y
Key expires at Wed 22 Jan 2020 01:39:23 PM CST
Is this correct? (y/N) y
 
GnuPG needs to construct a user ID to identify your key.
 
Real name: SaltMaster
Email address:
Comment:
You selected this USER-ID:
    "SaltMaster"
 
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
 
You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway.  You can change your passphrase at any time,
using this program with the option "--edit-key".
 
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: writing self signature
gpg: RSA/SHA1 signature from: "BB78ED27 [?]"
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: writing key binding signature
gpg: RSA/SHA1 signature from: "BB78ED27 [?]"
gpg: writing key binding signature
gpg: RSA/SHA1 signature from: "BB78ED27 [?]"
gpg: writing public key to `/etc/salt/gpgkeys/pubring.gpg'
gpg: writing secret key to `/etc/salt/gpgkeys/secring.gpg'
gpg: /etc/salt/gpgkeys/trustdb.gpg: trustdb created
gpg: using PGP trust model
gpg: key BB78ED27 marked as ultimately trusted
public and secret key created and signed.
 
gpg: checking the trustdb
gpg: 1 keys cached (2 signatures)
gpg: 1 keys processed (0 validity counts cleared)
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2020-01-22
pub   4096R/BB78ED27 2018-01-22 [expires: 2020-01-22]
      Key fingerprint = 6F40 FFB7 075F 7770 F75C  6E86 D3ED F536 BB78 ED27
uid                  SaltMaster
sub   4096R/1278D7EB 2018-01-22 [expires: 2020-01-22]
[root@server10 gpgkeys]#

Since this GPG key pair is going to be used for pillar to decrypt secrets you put in there, it has to be set without a password, so that the decryption can be automated during the rendering of the pillar file.

If you were to use this for something personal, like email, you would want to set a password, use a real name, and a real email address. This example is just a generic key pair used for a POC, so I didn’t need to do all that.

Export the key pair and import it on a server/workstation

Create the private key

gpg --homedir /etc/salt/gpgkeys --export-secret-keys --armor > /etc/salt/gpgkeys/salt-master.key

Create the public key

gpg --homedir /etc/salt/gpgkeys --export --armor > /etc/salt/gpgkeys/salt-master.pub

Finally, import the public key on the server/workstation that wants to use it

[root@server10 gpgkeys]# gpg --import /etc/salt/gpgkeys/salt-master.pub
gpg: key BB78ED27: public key "SaltMaster" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)


Encrypt the things

 

You can encrypt things by echoing or cat files into the gpg commands. These can be flat files or strings. In this example I encrypt a “string” that I intend to use as a hypothetical password example.

echo -n "my-super-secret-password" | gpg --armor --encrypt -r SaltMaster

[root@server10 gpgkeys]# echo -n "my-super-secret-password" | gpg --armor --encrypt -r SaltMaster
gpg: 1278D7EB: There is no assurance this key belongs to the named user

pub  4096R/1278D7EB 2018-01-22 SaltMaster
Primary key fingerprint: 6F40 FFB7 075F 7770 F75C  6E86 D3ED F536 BB78 ED27
Subkey fingerprint: 467C 463F 9D7E A783 F872  9AD2 CD23 421B 1278 D7EB

It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.

Use this key anyway? (y/N) y
-----BEGIN PGP MESSAGE-----
Version: GnuPG v2.0.22 (GNU/Linux)

hQIMA80jQhsSeNfrAQ//ZndPvtqd3vJgk94FdZ+wGb1NQlKNy+q1bwXqcWRAo0fq
ZM13ZpJKTts4DdVafxAXuSTNkxmSXyJxWZx9XBlTkO6eFmXvZ2x4gxkOlLVcBkwN
gCR0Tj9WKHSoEQ7Qnx3hsj858Tu53tv261TYuAgDv9D3FeH9LNACMwzoR3/oEzlw
xUl/auhBBHiMdxMokIyWW61cgzugWNm3vzPhVhLCOmre66Zqv/z0Wa9Zyy7m/rmI
kBh77BFpQzecKgJdPuLW+W9Ah9DEya/R9NJTHTQCujvEZ3rzWalG14uUXpr5RoC6
B3cG8pxyRUMLOW+wy/HUVbp999NjLLrlN57bn0trMm+NYexzYPuv7ANAeP6k1jnA
6Fv9v8S29GTMMKP7Kdvvs6WTQy0bWs8pa5xXiVUxcy3HDBNQ0FBjR3lIrRTjGddm
eeZopcMimoNznIn7Y4oqH5DHJ/Xkk5CL0zqDrpeIkNUBM1AcyP4Pi0lfziFINsc2
8dmCUCXXQ6vwuhqM0ph791vI6G9roDl+aRPkk+v+M49jZaogZViNZuhvJlZ2ScCr
yi6prSxoBsWtWC0MJWv1cT5vRmRgsLMn3Hmpu4aqsjuVCH2A7ICDOBkL0rBgscN3
+cfpAx9yy8oe4K/YTexDMM87tC30va54rJW7SPloyb8zMZE4Jkn5+5P8xpaan83S
UwFGgIr87JNW9zgRw1OJHbca0mGuPnmnOXLWzOCoRFt1PFGHjYFWezOAnvWy6ZUy
4sWG9sVE0zzBdhIUu30TUKUEcoRnL/p3byx899rDDeoK6CIR
=Ki8m
-----END PGP MESSAGE-----

Using GPG for secrets in Salt

To use these secrets encrypted by GPG in Salt Pillars we can copy the “message” generated into the pillar SLS file. Before we can do that though, we need to tell Salt which minions will have access to the pillar we intend to create. This is accomplished with a YAML “top file” placed in the pillars directory.

Pillar file system: /srv/pillar
Top file: top.sls (assigns pillar data to minions)

[root@server10 pillar]# cat top.sls

base:
  '*':
    - secrets

Create a pillar (we will use secrets.sls) where we will put our data to be shared with minions. We will use a statement at the top of the pillar SLS file to tell the rendering system that there is going to be GPG encrypted data in this state. We can now use both encrypted, and unencrypted data in the same YAML file! Yay!

NOTE: since this is a YAML file, make sure to respect the proper spacing requirements when copy/pasting the PGP message in. If the spaces aren’t right the YAML syntax will be off and it won’t work.

[root@server10 pillar]# vi secrets.sls
#!yaml|gpg

password1: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v2.0.22 (GNU/Linux)

hQIMA80jQhsSeNfrAQ//ZndPvtqd3vJgk94FdZ+wGb1NQlKNy+q1bwXqcWRAo0fq
ZM13ZpJKTts4DdVafxAXuSTNkxmSXyJxWZx9XBlTkO6eFmXvZ2x4gxkOlLVcBkwN
gCR0Tj9WKHSoEQ7Qnx3hsj858Tu53tv261TYuAgDv9D3FeH9LNACMwzoR3/oEzlw
xUl/auhBBHiMdxMokIyWW61cgzugWNm3vzPhVhLCOmre66Zqv/z0Wa9Zyy7m/rmInotencryptedsecret:
  datainpillar: something not encrypted you want in pillar

[root@server10 pillar]#

Refresh pillar data on each minion (this will update them so they can see what we just created).

salt '*' saltutil.refresh_pillar

View pillar data on the minions you targeted in the top.sls file

salt '*' pillar.items

Salt’s rendering system will render data in the order we specify. So because we set #! yaml|gpg at the top of the pillar SLS file (secrets.sls) it has rendered the YAML data first. After that is rendered, it then renders the next specified data type (in this case GPG, but it could be Jinja, JSON, or even Python).

You can ask Salt for specific values from the key value pairs in your pillar (secrets.sls) file. Since we have the key imported and the minions we are targeting have been assigned this pillar in the top file we created earlier, they will be able to decrypt the PGP message we put in our pillar file (secrets.sls).

[root@server10 pillar]# salt '*' pillar.item password1

Now that you have your encrypted and unencrypted data in pillar and the pillar file is targeted to the minions you want to have access to it only (I used globbing in the example just for simplicity but we have the full power of salt targeting here if needed), you can use it in other Salt states (SLS files).

Example:

sync directory using lftp:
cmd.run:
- name: lftp -c "open -u {{ pillar['ftpusername'] }},{{ pillar['password1'] }}
-p 22 sftp://example.com;mirror -c -R /local /remote"

You can have multiple pillar files on the salt master (which almost every implementation does). They can even be externally hosted, like in an Enterprise setup they might be hosted on a separate VM. You can also use Git as a pillar if you want, for easier version control. It all depends on your use case and needs.

Pillar values can also be passed on the command line if needed:

salt '*' state.apply ftpsync pillar='{"ftpusername": "test", "ftppassword": "0ydyfww3giq8"}'

External References

https://docs.saltstack.com/en/getstarted/config/pillar.html
https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html
http://www.clausconrad.com/blog/using-the-gpg-renderer-to-protect-salt-pillar-items

Hello world!

Welcome to my first post! I have no idea what I’m doing but that’s why it’s fun.

Code block testing example


[root@server10 salt]# yum install rng-tools
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
epel/x86_64/metalink | 15 kB 00:00:00
epel | 4.7 kB 00:00:00
ius | 2.3 kB 00:00:00
pgdg96 | 4.1 kB 00:00:00