Betelgeuse beetlejuice beetlejuice beetlejuice

That joke about AES?

Yeah, so ….

Please re-refer to this.

Also, there’s probably some kale joke in here.

Using Ansible Vault and GPG to secure critical infrastructure in public Github repositories

Not that this hasn’t been extensively covered elsewhere on the internet, but since I went through all the trouble to write it up, here’s my take on it.

The Pledge

I’m going to use TLS through Nginx as my example system, even though that really doesn’t matter since this is all about getting the key to the remote server, but in case I need a concrete example, there you go.

So, I’ve got my key:

-----BEGIN RSA PRIVATE KEY-----
H9COukD5Z1uKptB4D61xSApWCxh0i3vzQOyLe4rYGIHKpHv4rapTNqkdDieWeyfTb
VejsBEl7XMkgtItVl39Zr6lDxLWrmjQyX7Z6qsRn0HVci7nTMGWiIpwnfyqEOHPOz
oIoaTQD1j48ukcH72Snc7NsAbDPmCQ3rt30r1nsSkfTXskPK01IqAlHoLRvqvyxGt
BDsKguI5bxWsctol36WmPTy8kIdI2IBNadFfOIo71h69Z4zqpoXCAHCCtGZDDk057
cPHZ50ZAkyo6Mx7GqvxlowGBtU4Hbuz4IaSruhGUrJYOzdXIKGzn2Ca7Pvl8Ote43
LCR3rPGA2MOaCNovTMJ1ERKOJwtVto4V7qyIe6c3upQyBWeLRPKjap4lq14gw3zdX
5U6Ay1wzm3ZdUH7St4DiJFn4EXDHPR9sB3wXEhrAgJgtif02gUeDOufxohOxWjlna
4enaSlP2sdLquYxljE5UkwLUY0w0ligq9DgtzZgmsnTiSvAFtnnOBhzhIHchfv6oL
AY2YXmPcBo9F5fzzUZHsSJX8k5BCy2idV82wTJIFsz5c8sKPh4aiZCGaKwi6MYrHq
cDVrtgCO3tEiharWQZpUdKmbZWaBNHwOSdHcCa9xVsvxzwJUONKZhKNZUSPgXwOjg
InWV4hrluaftHJWx39NsvJcbC6qc76Bnc28eWcAKA0T6p1HlNLW4WXprYsscjBTNg
ifGU9ln8NgHVjQX79e0s1ZVgoElrrizeNduePYGAyIGFp0SYEFms42TuqCnFvpUYx
t2iLcTpqMy8LQWbw5fMRweQVpxCROJn00Mvi9MTjKfLkJBO1E3Vl8ezY3ikWreMIE
XHGGVer73Zi6UvfXuAgNUtCMXsEH0YnDkYUOQP0s5I3lubSjgrhQwbJIyIPv8HFwT
9QiFj3OMhYZoXodsoa9d5JyR8NNB0ygIkUQ05GskyVe5Q8x7DnmmMvIWsHxmVacwY
gS663RO5yzjrdSS3IZwy1Iz0KzWweXO4fKaogAjpxbqJPmO1l1Hct95rSwfl5xjCy
hny1lSltN7wv2ZIpLbx4weWEe2j8er8C9YsebEPHhL7aXsOCf9oFU5Hn6t1VDNzKa
UdbcVjr1vHOBHxVpHbWpj1Hzgk5AggQJ3HAIZjPQ2chW9TDQTeksCNTgsUxU0r1ns
iLaAhoGe84FXenKJkEC6KfvEtWlLVbuv3CZFeWcXuQEUhzbBNAs9uLUpdAeDSE4Xk
4esvSopZr8nmp2dxynapxrkshV6dYVxZuOsi22UwaAwUV65CCCMXPI7rfODzKzw1X
1jMcF37LEUG2YbJgxYkvu15VD4B3FRbi5mthMTenXOTsCo2xFKozdRm0J6zM2RZEL
ByRwWCM65pXWvPzFFCwejnogDWeg18yGLQxTEu5Jcv0Wr7rOkV52tSvaG8GnFg8Jt
SQ1tFvbHjyFbPaNsYr1L2CQHTXwwnFj57IDUzukB3zspAh5DuNDR8OWFTbQlwZ59D
THqg2gp81U6uZi0pSJB1rl6Voe3y5n1LUF742ZogveLiOU5X7Yo6txvTKrTw0SApo
8vwsfE9e56AdOLtt2diVsAeMsBpFoIUz7OfE1amuyoGDL5UpjDK9oNVU5
-----END RSA PRIVATE KEY-----

(Aside: thx to mihigh’s comment at https://gist.github.com/earthgecko/3089509, that fake key was generated, more or less, with cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 65 | head -n 25.

In the end, the goal is to have that key deployed to /etc/ssl/private or wherever your heart desires entirely through your publicly available ansible playbook with no worries that someone cloning your playbooks repository is going to be able to spoof your secure domain. So here goes:

Prerequisites:

Ansible, obviously. GPG. A GPG-generated PKI identity for yourself. The GPG generated public key for anyone else that should have access to the above TLS key. A basic understanding of TLS, and a basic understanding of how everything is broken. A more thorough understanding of how following any of these instructions and finding out that AES is broken and you stuck the private key for Bank of America up on a public github repository and now everyone is Bank of America is entirely your own problem.

The Turn

## roles/secure_tls, or whatever you want to call it. You’ll have three-ish things in here:

  • files/fake.crt and/or the intermediate certificate and/or a -fullchain certificate. Stop here if “intermediate certificate” was confusing, and see Prerequisites.
  • tasks/main.yml
  • vars/main.yml

Obviously, the public certificate can just be a file, and be stuck on the remote host with a straight - copy: src=fake.crt dest=/usr/share/ca-certificates/fake.crt owner=root group=root mode=0644

The vars is where the trickiness starts. Assuming you already have an empty vars/main.yml, then ansible-vault encrypt vars/main.yml; ansible-vault edit vars/main.yml, otherwise just ansible-vault create vars/main.yml. Save the password you use for the vault somewhere for now, we’ll get back to that.

That file, when being edited, should look like this:

---
secure_tls__fake_key: |
  -----BEGIN RSA PRIVATE KEY-----
  H9COukD5Z1uKptB4D61xSApWCxh0i3vzQOyLe4rYGIHKpHv4rapTNqkdDieWeyfTb
  VejsBEl7XMkgtItVl39Zr6lDxLWrmjQyX7Z6qsRn0HVci7nTMGWiIpwnfyqEOHPOz
  oIoaTQD1j48ukcH72Snc7NsAbDPmCQ3rt30r1nsSkfTXskPK01IqAlHoLRvqvyxGt
  BDsKguI5bxWsctol36WmPTy8kIdI2IBNadFfOIo71h69Z4zqpoXCAHCCtGZDDk057
  cPHZ50ZAkyo6Mx7GqvxlowGBtU4Hbuz4IaSruhGUrJYOzdXIKGzn2Ca7Pvl8Ote43
  LCR3rPGA2MOaCNovTMJ1ERKOJwtVto4V7qyIe6c3upQyBWeLRPKjap4lq14gw3zdX
  5U6Ay1wzm3ZdUH7St4DiJFn4EXDHPR9sB3wXEhrAgJgtif02gUeDOufxohOxWjlna
  4enaSlP2sdLquYxljE5UkwLUY0w0ligq9DgtzZgmsnTiSvAFtnnOBhzhIHchfv6oL
  AY2YXmPcBo9F5fzzUZHsSJX8k5BCy2idV82wTJIFsz5c8sKPh4aiZCGaKwi6MYrHq
  cDVrtgCO3tEiharWQZpUdKmbZWaBNHwOSdHcCa9xVsvxzwJUONKZhKNZUSPgXwOjg
  InWV4hrluaftHJWx39NsvJcbC6qc76Bnc28eWcAKA0T6p1HlNLW4WXprYsscjBTNg
  ifGU9ln8NgHVjQX79e0s1ZVgoElrrizeNduePYGAyIGFp0SYEFms42TuqCnFvpUYx
  t2iLcTpqMy8LQWbw5fMRweQVpxCROJn00Mvi9MTjKfLkJBO1E3Vl8ezY3ikWreMIE
  XHGGVer73Zi6UvfXuAgNUtCMXsEH0YnDkYUOQP0s5I3lubSjgrhQwbJIyIPv8HFwT
  9QiFj3OMhYZoXodsoa9d5JyR8NNB0ygIkUQ05GskyVe5Q8x7DnmmMvIWsHxmVacwY
  gS663RO5yzjrdSS3IZwy1Iz0KzWweXO4fKaogAjpxbqJPmO1l1Hct95rSwfl5xjCy
  hny1lSltN7wv2ZIpLbx4weWEe2j8er8C9YsebEPHhL7aXsOCf9oFU5Hn6t1VDNzKa
  UdbcVjr1vHOBHxVpHbWpj1Hzgk5AggQJ3HAIZjPQ2chW9TDQTeksCNTgsUxU0r1ns
  iLaAhoGe84FXenKJkEC6KfvEtWlLVbuv3CZFeWcXuQEUhzbBNAs9uLUpdAeDSE4Xk
  4esvSopZr8nmp2dxynapxrkshV6dYVxZuOsi22UwaAwUV65CCCMXPI7rfODzKzw1X
  1jMcF37LEUG2YbJgxYkvu15VD4B3FRbi5mthMTenXOTsCo2xFKozdRm0J6zM2RZEL
  ByRwWCM65pXWvPzFFCwejnogDWeg18yGLQxTEu5Jcv0Wr7rOkV52tSvaG8GnFg8Jt
  SQ1tFvbHjyFbPaNsYr1L2CQHTXwwnFj57IDUzukB3zspAh5DuNDR8OWFTbQlwZ59D
  THqg2gp81U6uZi0pSJB1rl6Voe3y5n1LUF742ZogveLiOU5X7Yo6txvTKrTw0SApo
  8vwsfE9e56AdOLtt2diVsAeMsBpFoIUz7OfE1amuyoGDL5UpjDK9oNVU5
  -----END RSA PRIVATE KEY-----

Note that the variable secure_tls__fake_key is exactly that from the start of The Pledge. Save and exit the vault editing; you’ll note, if you want to do the exercise, that vars/main.yml is now an unreadable encrypted mess of shit. Hooray! Go ahead and commit that fucker if you want! (Don’t really, because then you just have to append to the commit, etc.)

Your tasks/main.yml then becomes a pretty straightforward set of copies with associated setup:

---
- file: path=/etc/ssl/certs state=directory owner=root group=root mode=0755
- file: path=/etc/ssl/private state=directory owner=root group=root mode=0710
- file: path=/usr/share/ca-certificates owner=root group=root mode=0755 state=directory
- copy: src=fake.crt dest=/usr/share/ca-certificates/fake.crt owner=root group=root mode=0644
- copy: content="" dest=/etc/ssl/private/fake.key owner=root group=root mode=0644
- lineinfile: line=fake.crt regexp=^fake.crt state=present dest=/etc/ca-certificates.conf
- command: update-ca-certificates

That’s the bulk of it for ansible – use the secure_tls role in any playbook you choose, and rock and roll.

Well, almost.

Lets say that you’ve now got a playbook that looks like this:

---
- name: install tls certificates and keys
  remote_user: ubuntu
  hosts: wherever
  sudo: yes
  roles:
  - secure_tls

And you run it like ansible-playbook playbook.yml. As soon as ansible finds that secure_tls/vars/main.yml is a vault, it’ll stop. Because you didn’t use the magic words. Try ansible-playbook --ask-vault-pass playbook.yml. Great! Except now you have to type in that absurdly strong password each time. And somehow get that password to everyone else on your team that might use ansible to deploy things. Or at least get it to all your build agents that run ansible for you. Which means that it has to be in ansible or docker because you have playbooks that spin those agents up, right? And so its a catch-22! Which is where GPG comes in. (Most of the rest of the GPG section is lovingly recreated from the Ansible Cookbook 2014 with a few extra steps for GPG novices.)

.vault-password.txt

Take that super strong high-entropy password you used when you created the vault in the first place, and save it at the root level of your playbooks repository in .vault-password.txt, and then immediately add .vault-password.txt to .gitignore. In fact, do those in the opposite order.

Now encrypt it; the hex codes passed via –recipient to –encrypt-files are the public keyring IDs of keys you’ve imported (via gpg --import colleague.pub) – keys are obviously visible via gpg --list-keys.

gpg --recipient 9C430338 --recipient ABC2EA36 --encrypt-files .vault-password.txt

Note that if you need to add recipients, you’ll have to re-encrypt the original plaintext with all the recipients. Also, YMMV as to how you want to manage the fact that anyone with write access could overwrite the .gpg encrypted version of the file. On the other hand, version control, bitches.

Caveats

One last thing: ansible currently (1.7) doesn’t support multiple vault passwords in a single playbook run: so use the same password for all your vaults in a single set of playbooks. Re-key if necessary.

The Prestige

At this point, this should be fairly mundane and certainly not the subject of a sequel to the excellent “The Prestige” which you should go see if you haven’t. And don’t bother bringing up “Now You See Me.” Please.

Your playbooks repository in Github, assuming you’ve committed and pushed everything, looks vaguely like:

drwxr-xr-x ... .git
-rw-r--r-- ... .gitignore
-rw-r--r-- ... .vault-password.txt.gpg
-rw-r--r-- ... playbook.yml
drwxr-xr-x ... roles
drwxr-xr-x ...  |-- secure_tls

Anyone with their key on the recipients list for the .gpg file can now just pgp decrypt, and ansible-playbook --vault-password-file .vault-password.txt and away you go.

Aliases

Of course, you can make things tricksier too:

alias ansible-playbook-vault='test -e .vault-password.txt || gpg --decrypt-files .vault-password.txt.gpg; ansible-playbook --vault-password-file .vault-password.txt $@'

alias apv='ansible-playbook-vault'

And there you have it – secure storage of both private TLS keys and the password for the vault containing the private TLS keys all in a public git repository.

Don't Steal

My new favorite OSX .kext.

➜  ~  cat /System/Library/Extensions/Dont\ Steal\ Mac\ OS\ X.kext/Contents/Info.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>BuildMachineOSBuild</key>
<string>14A328</string>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>Dont Steal Mac OS X</string>
<key>CFBundleGetInfoString</key>
<string>Copyright © 2006,2009 Apple Inc.  All rights reserved.

The purpose of this Apple software is to protect Apple copyrighted materials from unauthorized copying and use. You may not copy, modify, reverse engineer, publicly display, publicly perform, sublicense, transfer or redistribute this file, in whole or in part.  If you have obtained a copy of this Apple software and do not have a valid license from Apple to use it, please immediately destroy or delete it from your computer.</string>
<key>CFBundleIdentifier</key>
<string>com.apple.Dont_Steal_Mac_OS_X</string>

HOWTO Switch JIRA to Crowd SSL

Two things: not only do you have to switch up the crowd configuration file in WEB-INF/classes, you also have to update the database for any Crowd directories you’ve got inside JIRA;

update cwd_directory_attribute set attribute_value='https://crowd.local/crowd' where directory_id=10000 and attribute_name='crowd.server.url';

And obviously then bounce JIRA.

What happens when you don't defensively check that you're getting a valid OAuth key from a remote server and base64-encode an HTML error page

The correct encoding should be 128 bytes. Decoding the entire page is left as an exercise to the reader.

rO0ABXQK9DwhRE9DVFlQRSBodG1sIFBVQkxJQyAiLS8vVzNDLy9EVEQgWEhUTUwgMS4wIFRyYW5zaXRpb25hbC8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9UUi94aHRtbDEvRFREL3hodG1sMS10cmFuc2l0aW9uYWwuZHRkIj4NCjxodG1sIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIj4NCjxoZWFkPg0KPE1FVEEgTkFNRT0ia2V5d29yZHMiIENPTlRFTlQ9ImJibWgtYm94ZG93biI+DQo8TUVUQSBIVFRQLUVRVUlWPSJDQUNIRS1DT05UUk9MIiBDT05URU5UPSJOTy1DQUNIRSI+DQo8bWV0YSBodHRwLWVxdWl2PSJjb250ZW50LXR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD1JU08tODg1OS0xIj4NCjxtZXRhIGh0dHAtZXF1aXY9IlByYWdtYSIgY29udGVudD0ibm8tY2FjaGUiPg0KPG1ldGEgaHR0cC1lcXVpdj0iRXhwaXJlcyIgY29udGVudD0iLTEiPg0KPG1ldGEgaHR0cC1lcXVpdj0iY2FjaGUtY29udHJvbCIgY29udGVudD0ibm8tc3RvcmUiPiANCg0KPHRpdGxlPkJsYWNrYm9hcmQgVGVtcG9yYXJpbHkgVW5hdmFpbGFibGU8L3RpdGxlPg0KPHN0eWxlPg0KZGl2IHsNCglmb250LWZhbWlseTp0YWhvbWEsYXJpYWwsdmVyZGFuYTsNCglmb250LXNpemU6MTZweCA7DQp9DQphOmxpbmsgew0KZm9udC1mYW1pbHk6dGFob21hLGFyaWFsLHZlcmRhbmE7DQogIGNvbG9yOiBibHVlIDsNCiAgYmFja2dyb3VuZC1jb2xvcjogdHJhbnNwYXJlbnQgOw0KICB0ZXh0LWRlY29yYXRpb246IG5vbmUgOw0KZm9udC1zaXplOjEzcHg7DQp9DQphOmhvdmVyIHsNCiAgY29sb3I6IHJlZCA7DQpmb250LWZhbWlseTp0YWhvbWEsYXJpYWwsdmVyZGFuYTsNCiAgIGJhY2tncm91bmQtY29sb3I6IHRyYW5zcGFyZW50IDsNCmZvbnQtc2l6ZToxM3B4Ow0KfQ0KDQphOnZpc2l0ZWQgew0KICBjb2xvcjogcmVkIDsNCmZvbnQtZmFtaWx5OnRhaG9tYSxhcmlhbCx2ZXJkYW5hOw0KICAgYmFja2dyb3VuZC1jb2xvcjogdHJhbnNwYXJlbnQgOw0KZm9udC1zaXplOjEzcHg7DQp9DQphOmFjdGl2ZSB7DQogIGNvbG9yOiBibHVlIDsNCmZvbnQtZmFtaWx5OnRhaG9tYSxhcmlhbCx2ZXJkYW5hOw0KICAgYmFja2dyb3VuZC1jb2xvcjogdHJhbnNwYXJlbnQgOw0KZm9udC1zaXplOjEzcHg7DQp9DQoNCjwvc3R5bGU+DQo8L2hlYWQ+DQo8Ym9keSBiZ2NvbG9yPSIjRkZGRkZGIj4NCjx0YWJsZSBhbGlnbj0iY2VudGVyIiBpZD0iVGFibGVfMDEiIHdpZHRoPSI2NDAiIGJvcmRlcj0iMCIgY2VsbHBhZGRpbmc9IjAiIGNlbGxzcGFjaW5nPSIwIj4NCgk8dHI+DQoJCTx0ZCBjb2xzcGFuPSI1Ij4NCg0KCQkJPGltZyBpZD0ibWFpbnRfc3BsYXNoXzAxIiBzcmM9ImltYWdlcy9tYWludC1zcGxhc2gtMDJfMDEucG5nIiB3aWR0aD0iNjQwIiBoZWlnaHQ9IjMzMSIgYWx0PSIiIC8+PC90ZD4NCgk8L3RyPg0KCTx0cj4NCgkJPHRkIHJvd3NwYW49IjMiPg0KCQkJPGltZyBpZD0ibWFpbnRfc3BsYXNoXzAyIiBzcmM9ImltYWdlcy9tYWludC1zcGxhc2hfMDIucG5nIiB3aWR0aD0iMzQiIGhlaWdodD0iMTQ5IiBhbHQ9IiIgLz48L3RkPg0KCQk8dGQgY29sc3Bhbj0iMyIgd2lkdGg9IjU3NCIgaGVpZ2h0PSIxMDIiIHZhbGlnbj0idG9wIj4NCgkJCTxkaXY+QmxhY2tib2FyZCBpcyB0ZW1wb3JhcmlseSB1bmF2YWlsYWJsZS4gIFdlIGFyZSBhd2FyZSBvZiB0aGUgc2VydmljZSBpbnRlcnJ1cHRpb24gYW5kIHdlIGFwb2xvZ2l6ZSBmb3IgdGhlIGluY29udmVuaWVuY2UuPGJyPjxicj4NCgkJCU91ciBzdGFmZiBpcyB3b3JraW5nIG9uIHRoZSBpc3N1ZS4gV2Ugd2lsbCBoYXZlIHRoZSBzaXRlIHVwIGFuZCBydW5uaW5nIGFnYWluIGFzIHNvb24gYXMgcG9zc2libGUuICBQbGVhc2UgdHJ5IGFnYWluIGF0IGEgbGF0ZXIgdGltZS48L2Rpdj4NCg0KPC90ZD4NCgkJPHRkIHJvd3NwYW49IjMiPg0KCQkJPGltZyBpZD0ibWFpbnRfc3BsYXNoXzA0IiBzcmM9ImltYWdlcy9tYWludC1zcGxhc2hfMDQucG5nIiB3aWR0aD0iMzIiIGhlaWdodD0iMTQ5IiBhbHQ9IiIgLz48L3RkPg0KCTwvdHI+DQoJPHRyPg0KCQk8dGQgd2lkdGg9IjI4NyIgaGVpZ2h0PSIxNiIgdmFsaWduPSJ0b3AiPg0KCQkJPGRpdj48YSBocmVmPSJodHRwOi8va2IuYmxhY2tib2FyZC5jb20vZGlzcGxheS9JRFNSL1N0dWRlbnQrYW5kK0ZhY3VsdHkrU3VwcG9ydCtSZXNvdXJjZStDZW50ZXIiPlN0dWRlbnQgJiBJbnN0cnVjdG9yIFJlc291cmNlczwvYT48L2Rpdj4gICAgICAgICAgICA8L3RkPg0KCQk8dGQgcm93c3Bhbj0iMiI+DQoNCgkJCTxpbWcgaWQ9Im1haW50X3NwbGFzaF8wNiIgc3JjPSJpbWFnZXMvbWFpbnQtc3BsYXNoXzA2LnBuZyIgd2lkdGg9IjI3IiBoZWlnaHQ9IjQ3IiBhbHQ9IiIgLz48L3RkPg0KCQk8dGQgd2lkdGg9IjI2MCIgaGVpZ2h0PSIxNiIgdmFsaWduPSJ0b3AiIGFsaWduPSJyaWdodCI+DQoJCTxkaXY+PGEgaHJlZj0iaHR0cDovL21hbmFnZWRob3N0aW5nLmJsYWNrYm9hcmQuY29tIj5CZWhpbmQgdGhlIEJsYWNrYm9hcmQgKGFkbWluaXN0cmF0b3JzIG9ubHkpPC9hPjwvZGl2Pgk8L3RkPg0KCTwvdHI+DQoJPHRyPg0KCQk8dGQ+DQoJCQk8aW1nIGlkPSJtYWludF9zcGxhc2hfMDgiIHNyYz0iaW1hZ2VzL21haW50LXNwbGFzaF8wOC5wbmciIHdpZHRoPSIyODciIGhlaWdodD0iMzEiIGFsdD0iIiAvPjwvdGQ+DQoJCTx0ZD4NCg0KCQkJPGltZyBpZD0ibWFpbnRfc3BsYXNoXzA5IiBzcmM9ImltYWdlcy9tYWludC1zcGxhc2hfMDkucG5nIiB3aWR0aD0iMjYwIiBoZWlnaHQ9IjMxIiBhbHQ9IiIgLz48L3RkPg0KCTwvdHI+DQo8L3RhYmxlPg0KPGJyPg0KPC9ib2R5Pg0KPC9odG1sPg0K