Cheatsheet: Hashicorp Vault REST API commands - in bash with curl and jq

Submitted by Sean Wingert on Wed, 08/19/2020 - 16:54

Here is a cheatsheet / list of Hashicorp Vault commands that I created as notes for myself. All of these commands are public information, via https://www.vaultproject.io/api.

Vault Config and Cheatsheet

These instructions help you quickly set up vault using Docker. It also has a cheatsheet of vault commands for convenience. See official documentation for most vault documentation.

Installing a Single Vault Production Container

Provision a VM

We need a VM only because we need Docker, which we'll use for vault, ansible, etc. If you're going serverless (IKS, EKS, ECS), this section is not required.

# create restricted bash
sudo cp -f /bin/bash /bin/rbash

# create a system account for administration
sudo groupadd -g 2000 foo
sudo groupadd -g 1111 fwd
sudo useradd -u 2750 -g 2000 -d /home/foo -m -s /bin/bash foo
sudo useradd -u 1111 -g 1111 -d /home/fwd -m -s /bin/rbash fwd
sudo bash -c 'echo foo:$(openssl rand -base64 32) | chpasswd'
sudo bash -c 'echo fwd:$(openssl rand -base64 32) | chpasswd'
sudo mkdir -p ~foo/.ssh
sudo mkdir -p ~fwd/.ssh
sudo chmod 0700 ~foo/.ssh
sudo chmod 0700 ~fwd/.ssh
sudo tee ~foo/.ssh/authorized_keys >/dev/null <<EOF # Created by John Smith at provisioning time. See jenkins > app > vault > readme
ADD ALL PUBLIC KEYS HERE!!!
EOF
sudo cp ~foo/.ssh/authorized_keys ~fwd/.ssh/authorized_keys
sudo chmod 0600 ~foo/.ssh/authorized_keys
sudo chmod 0600 ~fwd/.ssh/authorized_keys
sudo chown -R foo:foo ~foo/.ssh
sudo chown -R fwd:fwd ~fwd/.ssh
sudo usermod -a -G foo foo

# restrict permissions on the fwd account
sudo tee ~fwd/.bash_profile >/dev/null <<\EOF
# Get the aliases and functions  
if [ -f ~/.bashrc ]; then  
. ~/.bashrc 
fi  
# User specific environment and startup programs  

# Harden. Prevents common commands from working, like ls, echo, etc.
PATH=/home/fwd/programs  
export PATH
EOF
sudo chown fwd:fwd ~fwd/.bash_profile


# allow foo to run all commands without a password
sudo sed -i.bkp '/^%wheel/c%foo  ALL=(ALL)       NOPASSWD: ALL' /etc/sudoers

# Disable passwords for root and everyone else
# WARNING: Ensure you can log in with an ssh key BEFORE this!!!
sudo sed -i \
  -e 's/#PasswordAuthentication yes/PasswordAuthentication no/'    \
  -e 's/PasswordAuthentication yes/PasswordAuthentication no/'     \
  -e 's/#PermitRootLogin no/PermitRootLogin no/'                   \
  -e 's/#PermitRootLogin yes/PermitRootLogin no/'                  \
  -e 's/PermitRootLogin yes/PermitRootLogin no/'                   \
  -e 's/#PermitRootLogin without-password/PermitRootLogin no/'     \
  -e 's/PermitRootLogin without-password/PermitRootLogin no/'      \
  -e 's/#PermitRootLogin forced-commands-only/PermitRootLogin no/' \
  -e 's/PermitRootLogin forced-commands-only/PermitRootLogin no/'  \
  -e '/^GSSAPIAuthentication yes/cGSSAPIAuthentication no'         \
  -e '/^#UseDNS yes/cUseDNS no'                                    \
  -e 's/#LogLevel INFO/LogLevel DEBUG3/'                           \
/etc/ssh/sshd_config

# restart sshd to pick up the changes. This does not disconnect current connections.
systemctl restart sshd

# install docker following these instructions: https://docs.docker.com/install/linux/docker-ce/centos/
# in short, do this
sudo yum remove -y docker docker-client docker-client-latest docker-common \
            docker-latest docker-latest-logrotate docker-logrotate docker-engine
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager -y --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.i

# add foo to docker group to run docker without typing sudo
sudo usermod -a -G docker foo

# log out and back in again
exit
ssh foo@vault

# allow docker to start automatically on reboot
sudo systemctl enable docker

# start docker
sudo systemctl start docker

# install docker compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

# set the hostname
sudo hostnamectl set-hostname vault.foo.com

# install our apps' requirements
sudo yum install -y vim curl rsync bzip2 bzip2-libs

sudo yum install epel-release -y
sudo yum install jq -y
jq --version

# configure the firewall
sudo firewall-cmd --list-all

Install the Vault container

Vault:

# git clone or rsync (just the jenkins dir) the docker-compose file to the provisioned VM
rsync -Pua ~/git/dev-ops-tasks/jenkins vault:

# login into vault host
ssh vault
cd jenkins/app
run docker-compose up

# read vault's initialization status
export VAULT_ADDR="http://localhost:8200"
curl -sS --request GET $VAULT_ADDR/v1/sys/init | jq .

# initialize vault
# #####################################################################
# WARNING: You MUST now record these values OUTSIDE of vault!!!!!
# WARNING: If you lose these, you lose all data in vault.
# #####################################################################
tee foo.json <<\EOF
{
  "secret_shares": 3,
  "secret_threshold": 2
}
EOF
curl -sS --request PUT --data @foo.json $VAULT_ADDR/v1/sys/init | jq .

# set this before continuing
export VAULT_TOKEN="VALUE_FROM_ABOVE_OUTPUT"

# unseal vault
# NOTE: If you 3 master keys, you must do this 3 times, with ONLY 1 key in the json payload at a time.
tee key1.json <<< '{ "key": "1234" }'
tee key2.json <<< '{ "key": "3923" }'
tee key3.json <<< '{ "key": "2342" }'
curl -sS --request PUT --data @key1.json $VAULT_ADDR/v1/sys/unseal | jq .
curl -sS --request PUT --data @key2.json $VAULT_ADDR/v1/sys/unseal | jq .
curl -sS --request PUT --data @key3.json $VAULT_ADDR/v1/sys/unseal | jq .

# SHRED AND DELETE THE KEY FILES!!!!
shred -f -n 99 --remove -z key{1,2,3}.json

Create CA

IMPORTANT: After you run these steps, sshv will stop working because it will get certificates signed by the NEW users ca, but the hosts are still signed with the old users CA, so the hosts don't trust the new user CA certs. So to avoid downtime, run the steps below but do NOT overwrite the existing CA in Vault. Instead get the new public keys and deploy them to the existing hosts /etc/ssh/user-ca-keys.pub

# Prepare
sudo mkdir -p /usr/local/sshca/{users,hosts}
sudo chmod 1771 /usr/local/sshca        # sticky bit (first 1) prevents non-owners from renaming files. # 11 allows cat of .pub files
sudo chmod 774 /usr/local/sshca/{users,hosts} 
sudo chgrp -R $(id -g -n) /usr/local/sshca # recursively change group to your group

# Create two CAs: 1 to sign hosts and 1 to sign users. These are openssh, not openssl, CAs. No password since Vault can't decrypt it.
sudo ssh-keygen -N '' -t rsa -b 4096 -C "SSH CA for hosts created on TODAY'S DATE by John Smith" -f /usr/local/sshca/hosts-ca
sudo ssh-keygen -N '' -t rsa -b 4096 -C "SSH CA for users created on TODAY'S DATE by John Smith" -f /usr/local/sshca/users-ca

# Protect them
sudo chmod 400 /usr/local/sshca/{hosts-ca,users-ca}         # only allow the owner to read these
sudo chmod 444 /usr/local/sshca/{users-ca.pub,hosts-ca.pub}

# Enable Vault ssh secrets engine
tee foo.json <<\EOF
{
    "type": "ssh"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "type": "ssh" }' $VAULT_ADDR/v1/sys/mounts/hosts-ca | jq .
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "type": "ssh" }' $VAULT_ADDR/v1/sys/mounts/users-ca | jq .

Delete previous CA keypairs

curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/hosts-ca/config/ca | jq .
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/users-ca/config/ca | jq .

Configure Vault with them

tee foo.json <<EOF
{
    "generate_signing_key" : false,
    "private_key"          : "$(sudo cat /usr/local/sshca/hosts-ca | tr '\n' '*' | sed 's/\*/\\n/g')",
    "public_key"           : "$(sudo cat /usr/local/sshca/hosts-ca.pub | tr '\n' '*' | sed 's/\*/\\n/g')"
}
EOF
# Create (it can't Update) your keypair in Vault
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/hosts-ca/config/ca | jq .

tee foo.json <<EOF
{
    "generate_signing_key" : false,
    "private_key"          : "$(sudo cat /usr/local/sshca/users-ca | tr '\n' '*' | sed 's/\*/\\n/g')",
    "public_key"           : "$(sudo cat /usr/local/sshca/users-ca.pub | tr '\n' '*' | sed 's/\*/\\n/g')"
}
EOF
# Create (it can't Update) your keypair in Vault
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/config/ca | jq .

Validate (read their public keys)

curl -sS --request GET $VAULT_ADDR/v1/hosts-ca/public_key
curl -sS --request GET $VAULT_ADDR/v1/users-ca/public_key

Cleanup

Shred the CA's because now they're in Vault instead.

sudo rmdir /usr/local/sshca/{users,hosts}
sudo shred -f -n 99 --remove -z /usr/local/sshca/*
sudo rm -rfv /usr/local/sshca

Active Directory

Enable ldap auth method

Run these commands from vault:

# enable ldap auth method
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "ldap"}' $VAULT_ADDR/v1/sys/auth/ldap | jq .

# manually get the public key in PEM format of the Active Directory CA
tee pem <<EOF
-----BEGIN CERTIFICATE-----
MIIKIjCCCQqgAwIBAgIRAOte8RRtBz3NCAAAAABTG3EwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMDA4MTEwODQxMjhaFw0yMDExMDMwODQx
MjhaMGYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
Ew1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgTExDMRUwEwYDVQQDDAwq
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCOtOkw
bduvWPL7MDXH8kjbXupowPFpWQdQtMdRxb/Skkxwi7lXVgaADAVdGYCmax5K+11+
B43jzf4TYxTisR2G51DWQOMcqPD/ReZCP2I+PJqSS5jSUJA9zdRpDyqRVeYnsIzu
S/RukODIEx7gCJsN4vn/eEnjbPgh5yFJbAbG4v+PsSZUFOow97ihIDb4tAm1kNLQ
ho/fZ1N2MkvTgxeYzpNrx61BRpEhBGQsUpbaqtABdZeRTEQr4lh6L7bPHAbF2hlg
o8AB/c2CIWJRkiQRXwi4em8vycclYqSPvMgmYcn1UoQQJ0q5nKj2MT0A8RlqQkBa
NRz+o64RMI9TjfkDAgMBAAGjggbtMIIG6TAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l
BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUCGbV2JvkbVTp
o+g2KPKWSdZqfKMwHwYDVR0jBBgwFoAUmNH4bhDrz5vsYJ8YkBug630J/SswaAYI
KwYBBQUHAQEEXDBaMCsGCCsGAQUFBzABhh9odHRwOi8vb2NzcC5wa2kuZ29vZy9n
dHMxbzFjb3JlMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2cvZ3NyMi9HVFMx
TzEuY3J0MIIEqAYDVR0RBIIEnzCCBJuCDCouZ29vZ2xlLmNvbYINKi5hbmRyb2lk
LmNvbYIWKi5hcHBlbmdpbmUuZ29vZ2xlLmNvbYIJKi5iZG4uZGV2ghIqLmNsb3Vk
Lmdvb2dsZS5jb22CGCouY3Jvd2Rzb3VyY2UuZ29vZ2xlLmNvbYIGKi5nLmNvgg4q
LmdjcC5ndnQyLmNvbYIRKi5nY3BjZG4uZ3Z0MS5jb22CCiouZ2dwaHQuY26CDiou
Z2tlY25hcHBzLmNughYqLmdvb2dsZS1hbmFseXRpY3MuY29tggsqLmdvb2dsZS5j
YYILKi5nb29nbGUuY2yCDiouZ29vZ2xlLmNvLmlugg4qLmdvb2dsZS5jby5qcIIO
Ki5nb29nbGUuY28udWuCDyouZ29vZ2xlLmNvbS5hcoIPKi5nb29nbGUuY29tLmF1
gg8qLmdvb2dsZS5jb20uYnKCDyouZ29vZ2xlLmNvbS5jb4IPKi5nb29nbGUuY29t
Lm14gg8qLmdvb2dsZS5jb20udHKCDyouZ29vZ2xlLmNvbS52boILKi5nb29nbGUu
ZGWCCyouZ29vZ2xlLmVzggsqLmdvb2dsZS5mcoILKi5nb29nbGUuaHWCCyouZ29v
Z2xlLml0ggsqLmdvb2dsZS5ubIILKi5nb29nbGUucGyCCyouZ29vZ2xlLnB0ghIq
Lmdvb2dsZWFkYXBpcy5jb22CDyouZ29vZ2xlYXBpcy5jboIRKi5nb29nbGVjbmFw
cHMuY26CFCouZ29vZ2xlY29tbWVyY2UuY29tghEqLmdvb2dsZXZpZGVvLmNvbYIM
Ki5nc3RhdGljLmNugg0qLmdzdGF0aWMuY29tghIqLmdzdGF0aWNjbmFwcHMuY26C
CiouZ3Z0MS5jb22CCiouZ3Z0Mi5jb22CFCoubWV0cmljLmdzdGF0aWMuY29tggwq
LnVyY2hpbi5jb22CECoudXJsLmdvb2dsZS5jb22CEyoud2Vhci5na2VjbmFwcHMu
Y26CFioueW91dHViZS1ub2Nvb2tpZS5jb22CDSoueW91dHViZS5jb22CFioueW91
dHViZWVkdWNhdGlvbi5jb22CESoueW91dHViZWtpZHMuY29tggcqLnl0LmJlggsq
Lnl0aW1nLmNvbYIaYW5kcm9pZC5jbGllbnRzLmdvb2dsZS5jb22CC2FuZHJvaWQu
Y29tghtkZXZlbG9wZXIuYW5kcm9pZC5nb29nbGUuY26CHGRldmVsb3BlcnMuYW5k
cm9pZC5nb29nbGUuY26CBGcuY2+CCGdncGh0LmNuggxna2VjbmFwcHMuY26CBmdv
by5nbIIUZ29vZ2xlLWFuYWx5dGljcy5jb22CCmdvb2dsZS5jb22CD2dvb2dsZWNu
YXBwcy5jboISZ29vZ2xlY29tbWVyY2UuY29tghhzb3VyY2UuYW5kcm9pZC5nb29n
bGUuY26CCnVyY2hpbi5jb22CCnd3dy5nb28uZ2yCCHlvdXR1LmJlggt5b3V0dWJl
LmNvbYIUeW91dHViZWVkdWNhdGlvbi5jb22CD3lvdXR1YmVraWRzLmNvbYIFeXQu
YmUwIQYDVR0gBBowGDAIBgZngQwBAgIwDAYKKwYBBAHWeQIFAzAzBgNVHR8ELDAq
MCigJqAkhiJodHRwOi8vY3JsLnBraS5nb29nL0dUUzFPMWNvcmUuY3JsMIIBBAYK
KwYBBAHWeQIEAgSB9QSB8gDwAHcAB7dcG+V9aP/xsMYdIxXHuuZXfFeUt2ruvGE6
GmnTohwAAAFz3OV8BgAABAMASDBGAiEAuMafz7C3f1ldRKddqTuIEgh4GPYnAJvf
dD/oD4H9II8CIQDxQAJbAvrShgUPB4o84ZEPCxkiFQ/oyQFlsT+ANl55WgB1AMZS
oOxIzrP8qxcJksQ6h0EzCegAZaJiUkAbozYqF8VlAAABc9zle+MAAAQDAEYwRAIg
CY5ReICPtboDfbRlx2R3C9i5bXXGbBmA+lHaKqyCNqICIA4BcPAEKS4eVdvLqx3o
LIIJYH1AffvlLNeQcWwud/7wMA0GCSqGSIb3DQEBCwUAA4IBAQBXQ6QFIGrLPH0B
jCvJ5Vk/hvEOh8xWaRvoxRC/PHn/AL+kudLGwIykvqhZSMgXGr6vBYarHYAWlFoC
LoAnq8aKOO2y9DHY5UpTEfPz2kuVPomea+Prqr2kNCsUWIg8XNQaOEo3VJmm+rk9
lpX4YYhR5Mh5vdODK/lc9dfMzMezqMirruFeQKb4eXPa1ctCCi1Cw+gKUcYbmkno
hGxRcbQTk+VxGoH8Kz6VK1DlWZwGHjytDX49s5siGfdJAtULiXrOd4JaZqJaXYdS
3o6ApvyQx/X/FeCciUR/WjXqiJcr1Pt2MGDQqlCwOG19+JmmMEnT+l+f9AaWXk/A
JKt3fL67
-----END CERTIFICATE-----
EOF
pem="$(cat pem)"

# reformat for later
pem_vault_format="$(cat pem | tr '\n' '*' | sed 's/\*/\\n/g')" # download the LDAP server's PEM and replace its newlines with literal \n
echo "${pem_vault_format}"

# use that cert and test LDAP
docker run --rm -i --name openldap-test --network=vault.local --entrypoint=bash emeraldsquad/ldapsearch -s <<EOF
  tee -a /etc/openldap/ldap.conf <<< "TLS_CACERT    /etc/openldap/cert.pem"
  tee /etc/openldap/cert.pem &>/dev/null <<< "${pem}"
  timeout 4 /usr/bin/ldapsearch -x -H ldaps://foo.bar.com:1636 \
  -D "AD_USERNAME_FOR_VAULT" -w "AD_PASSWORD_FOR_VAULT" -b "CN=Users,DC=foo,DC=bar,DC=com" \
    -s sub "objectClass=organizationalPerson"
EOF

# Configure Vault's connection to LDAP
tee foo.json <<EOF
{
    "url": "ldaps://foo.bar.com:636",
    "tls_min_version": "tls12",
    "token_ttl": "720m",
    "token_max_ttl": "720m",
    "token_bound_cidrs": "",
    "token_num_uses": 999,
    "userattr":"sAMAccountName",
    "userdn":"CN=Users,DC=foo,DC=bar,DC=com",
    "groupdn":"CN=Users,DC=foo,DC=bar,DC=com",
    "groupfilter":"(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))",
    "binddn":"vault-bind@foo.bar.com",
    "bindpass":"YOUR_VAULT_BIND_PASSWORD",
    "groupattr":"memberOf",
    "certificate":"${pem_vault_format}",
    "insecure_tls":false,
    "starttls":true
}
EOF

curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/auth/ldap/config

# confirm
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/ldap/config | jq .

# test a login
export VAULT_ADDR="http://localhost:8200"
token="$(curl -sS --request POST --data '{"password": "MOTION_PRO_PASSWORD"}' $VAULT_ADDR/v1/auth/ldap/login/foo | jq .auth.client_token )"
printf "export VAULT_ADDR=%s\nexport VAULT_TOKEN=%s" "${VAULT_ADDR}" "${token}"

# set a basic policy
# Write a new JSON policy file: https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/devpol.json
# https://learn.hashicorp.com/vault/getting-started/policies
# Version 1 of KV, which we don't currently use, would be "secret/*"
# Version 2 of KV, which we do use, is "secret/data/*"
tee policy1.json <<\EOF
{
  "policy": "path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
  }

  path "secret/metadata/*" {
    capabilities = ["read", "list", "create", "update", "delete" ]
  }

  path "auth/token/lookup" {
    capabilities = ["read"]
  }

  path "auth/token/lookup-self" {
    capabilities = ["read"]
  }

  path "auth/token/renew-self" {
    capabilities = ["read","update"]
  }

  path "auth/approle/role/jenkins/*" {
    capabilities = ["create", "update", "read", "delete", "list"]
  }

  path "hosts-ca/sign/role1" {
    capabilities = ["create", "update", "read", "delete", "list"]
  }

  path "users-ca/sign/role1" {
    capabilities = ["create", "update", "read", "delete", "list"]
  }"
}
EOF

# Reformat: https://github.com/hashicorp/vault/issues/582#issuecomment-533334354 (my post)
# The trick is to see that "policy" is actually a single value: "path..." (where nested quotes are escaped) in HCL format
cat policy1.json | tr '\n' ' ' | sed 's/"/\\\"/g; s/\\"policy\\": \\"/"policy": "/; s/}\\"/}"/' | tee tmp; mv tmp policy1.json

# Create/Update a new policy
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @policy1.json $VAULT_ADDR/v1/sys/policies/acl/policy1 | jq .

# List policies
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/policy | jq .

# Map policy1 to AD group
# NOTE: We use nested groups, but due to bug https://github.com/hashicorp/vault/issues/6325, use child group member names, not the parent
# EXAMPLE: Dont use WFFS-Ops, use Administrators
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"policies": "policy1"}' $VAULT_ADDR/v1/auth/ldap/groups/vault-all-secrets

# Create role for hosts
# Provide role parameters
tee foo.json <<\eof
{
  "key_type": "ca",
  "allowed_users": "*",
  "allowed_domains": "foo.com,bar.com,local",
  "ttl": "8760h",
  "max_ttl": "8760h",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "allow_user_certificates": false,
  "allow_host_certificates": true,
  "allow_subdomains": true,
  "allow_user_key_ids": true,
  "key_id_format": "hosts-ca-default_key_id_format-{{token_display_name}}{{role_name}}{{public_key_hash}}"
}
eof

# Create / Update the role
curl -sS --header "x-vault-token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/hosts-ca/roles/role1 | jq .

# Read the role
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/hosts-ca/roles/role1 | jq .

# create role for users
# Provide role parameters
tee foo.json <<\eof
{
  "key_type": "ca",
  "allowed_users": "*",
  "ttl": "2m0s",
  "max_ttl": "262800m",
  "allowed_extensions": "permit-pty",
  "default_extensions": {
        "permit-pty": "",
        "permit-port-forwarding": ""
  },
  "allow_user_certificates": true,
  "allow_host_certificates": false,
  "allow_user_key_ids": true,
  "key_id_format": "users-ca-default_key_id_format-{{token_display_name}}{{role_name}}{{public_key_hash}}"
}
eof

# Create / Update the role
curl -sS --header "x-vault-token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/roles/role1 | jq .

# Read the role 
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET --data @foo.json $VAULT_ADDR/v1/users-ca/roles/role1 | jq .

Allow users to request signatures

# Set parameters
tee foo.json <<\EOF
{
  "public_key"       : "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII1FSg9Yf866tJmMpD2iEbwDCxfxJAlZBhUb0BhEzW+M user1@host2",
  "ttl"              : "1m",
  "valid_principals" : "foo,user1,root",
  "critical_options" : { "":"" },
  "extension"        : { "":"" },
  "cert_type"        : "user",
  "key_id"           : ""
}
EOF

# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/sign/role1 | jq .

Create an approle for jenkins

# Create/Enable the AppRole authentication method at path foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "approle"}' $VAULT_ADDR/v1/sys/auth/approle | jq .

# Attach desired policies to the Approle
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
         --data '{"policies": "all-staff,network-administrators,domain-administrators"}' \
         $VAULT_ADDR/v1/auth/approle/role/jenkins | jq .

# Get metadata about the AppRole
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/approle/role/jenkins | jq .

# Fetch the identifier of a role (i.e., RoleID)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/approle/role/jenkins/role-id | jq .

# Create a new secret identifier (i.e., SecretID) under the role
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/auth/approle/role/jenkins/secret-id | jq .

# Log in with RoleID (analagous to username) and SecretID (analagous to password) created above
tee foo.json <<\EOF
{
  "role_id": "0bjsdf33-85ca-c776-0635-14jsdkfjsfd3",
  "secret_id": "86e02232-aebf-ja81-25bd-ksjdfkjs4f32"
}
EOF
curl -sS --request POST --data @foo.json $VAULT_ADDR/v1/auth/approle/login | jq .

Enable version 2 secret store

CRUD: Secrets Engine

# Create / Enable an engine (replace kv as needed--eg ssh)
# NOTE: You probably want "version": "2" (version2 maintains change history unlike version1)
tee foo.json <<\EOF
{
    "type": "kv",
    "options": {
        "version": "2"
    }
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/sys/mounts/secret | jq .

Configure PKI

# Enable root
tee foo.json <<\EOF
{
    "type": "pki",
    "default_lease_ttl": "87600h",
    "max_lease_ttl": "87600h"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "pki"}' $VAULT_ADDR/v1/sys/mounts/pki-root1 | jq .

# Verify
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/mounts/pki-root1/tune | jq .

# Troubleshooting
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/mounts | jq .
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/mounts/pki-root1 | jq . # CAREFUL!!!!!!!!!!!!!

# Enable intermediate. NOTE: path is different!
tee foo.json <<\EOF
{
    "type": "pki",
    "default_lease_ttl": "43800h",
    "max_lease_ttl": "43800h"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "pki"}' $VAULT_ADDR/v1/sys/mounts/pki-intermediate1 | jq .
# Troubleshooting
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/mounts/pki-intermediate1 | jq .
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/mounts | jq .

# Create root CA
tee foo.json <<\EOF
{
    "common_name": "root1-ca.foo.bar.com",
    "format": "pem",
    "ttl": "87600h",
    "max_ttl": "87600h",
    "private_key_format": "der",
    "key_type": "rsa",
    "key_bits": 4096,
    "ou": "FSS",
    "organization": "FOO",
    "country": "US",
    "locality": "Austin",
    "province": "TX"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-root1/root/generate/exported | jq .

# Create a role (apply to BOTH root and intermediate CA--at least for now)
tee foo.json <<\EOF
{
    "ttl": "4680h",
    "max_ttl": "4680h",
    "allowed_domains": ["foo.bar.com","foo.bar.com"],
    "allow_localhost": true,
    "allow_any_name": true,
    "allow_glob_domains": true,
    "key_usage": [],
    "ext_key_usage": ["ServerAuth","ClientAuth","OCSPSigning"],
    "key_type": "rsa",
    "key_bits": 4096,
    "ou": "FSS",
    "organization": "FOO",
    "country": "US",
    "locality": "Austin",
    "province": "TX"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-root1/roles/role1 | jq .
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-intermediate1/roles/role1 | jq .

# Verify values
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/pki-root1/roles/role1 | jq .

# List certs
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request LIST $VAULT_ADDR/v1/pki-root1/certs | jq .

# Read the CA's cert
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/pki-root1/cert/ca | jq -r '.data.certificate' | tee cert

# Configure CRL and OSCP URLs
tee foo.json <<\EOF
{
    "issuing_certificates": ["http://127.0.0.1:8200/v1/pki-root1/ca"],
    "crl_distribution_points": ["http://127.0.0.1:8200/v1/pki-root1/crl"],
    "ocsp_servers": ["http://127.0.0.1:8200/todo/pki-root1/ocsp"]
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-root1/config/urls | jq .

# Configure CRL and OSCP URLs
tee foo.json <<\EOF
{
    "issuing_certificates": ["http://127.0.0.1:8200/v1/pki-intermediate1/ca"],
    "crl_distribution_points": ["http://127.0.0.1:8200/v1/pki-intermediate1/crl"],
    "ocsp_servers": ["http://127.0.0.1:8200/todo/pki-intermediate1/ocsp"]
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-intermediate1/config/urls | jq .

# Create intermediate CSR (for the root CA we just created). Note: path is pki-int, not pki
tee foo.json <<\EOF
{
    "common_name": "intermediate1-ca.foo.bar.com",
    "format": "pem_bundle",
    "private_key_format": "der",
    "key_type": "rsa",
    "key_bits": 4096,
    "ou": "FSS",
    "organization": "FOO",
    "country": "US",
    "locality": "Austin",
    "province": "TX"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-intermediate1/intermediate/generate/exported | jq ".data.csr" | tr -d '"' | tee csr
sed -i "s/.*-----BEGIN CERTIFICATE REQUEST/-----BEGIN CERTIFICATE REQUEST/" csr

# Create / Sign the intermediate CA
tee foo.json <<EOF
{
    "csr": "$(cat csr)",
    "format": "pem_bundle",
    "ttl": "46800h",
    "private_key_format": "der",
    "key_type": "rsa",
    "key_bits": 4096,
    "ou": "FSS",
    "organization": "FOO",
    "country": "US",
    "locality": "Austin",
    "province": "TX"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-root1/root/sign-intermediate | jq .data.certificate | tr -d '"' | tee cert

# Shred the private key
shred -f -n 99 --remove -z csr

# "Set" (activate) the intermediate CA. NOTE: /pki-intermediate not /pki
tee foo.json <<EOF
{
    "certificate": "$(cat cert)"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-intermediate1/intermediate/set-signed | jq .

# List the intermediate cert
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request LIST $VAULT_ADDR/v1/pki-intermediate1/certs | jq .

# Create a signed cert with private key
tee foo.json <<\EOF
{
    "common_name": "foo.bar.com",
    "ip_sans": "192.168.1.1",
    "format": "pem",
    "private_key_format": "pem",
    "private_key_format": "pem",
    "ttl": "4680h"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/pki-intermediate1/issue/role1 > results

# Based on the last command, save the parts
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/pki-root1/cert/ca | jq -r '.data.certificate' > root.pem
jq -r '.data.ca_chain|.[]' < results > intermediates.pem
jq -r '.data.certificate' <results >cert.pem
jq -r '.data.private_key' <results >key.pem
cat intermediates.pem root.pem > ca-chain.pem
chmod 0600 {root,intermediates,cert,key,ca-chain}.pem

# Create tarball of the parts
tar -czvf bundle.tgz ca-chain.pem root.pem intermediates.pem cert.pem key.pem
chmod 0600 bundle.tgz

# delete and shred those parts, since they contained a private key
shred -f -n 99 --remove -z results ca-chain.pem root.pem intermediates.pem cert.pem key.pem

Create secrets for jenkins

# Create the vault secret containing the Github token that Jenkins needs to do git clone
tee foo.json <<\EOF
{
    "data": {
        "username": "jenkins@foo.bar.com",
        "password": "PASSWORD"
    }
}
EOF

# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/secret/data/active-directory/jenkins | jq .

# Create the ssh-keys path
tee foo.json <<\EOF
{
    "data": {
        "description": "The ssh passphrase for the named private key. The actual keys are (currently) in Github.",
        "name_deleteme": "foo-swinger",
        "passphrase_deleteme": "skdjfksjdfkjaskdfjkasjdfkjasdkfj",
        "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\aksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\naksjdfkljasdflkjaslkdfjlkasjdflkjaslkdfjlaksdjflkajsdflkjaslkdfjlkasjdflkjasldkfj\nmaCIVTSBm0vySE12+EOPJBy689Y7ib1iJ85Aoz65DCTpidVKYfYbHHilYl/EmrzggYX7Yb\n4kbkBUQHyC0fiDT/mnu2hsQdIsL7tjC0h10Px/kQ4HHotRXCKjKOkryXCOBYCh4jT923+U\nVHBj+Ku8wf3cyVXzzwxIOU6eqfjOoWvmNUrrU8ORXSPvom+j9Jycsgo5D/ZUOKl88+lMec\nY/qDRlufYBcBZ9iOMlgmXtBCogbCL2+/ud/vWzwvRdoICBUO2hSPyJefXDxcbC0HRBpLD+\nVGwMJwqL2UuS1qszqxKRSAYK332O4hZNwjd6Zbir46iQAAB4A/X0vSXFVyVNtW8ywgq7Kn\nE2MqyWCU/VpnLVu3QsKwV/7aWFShNwriFs5L3hqbGMPR9AqG/pFPkNwPK8jiDSLaZtpX3W\nDKAfyWvQkJ3Xd3u/xAJqXHdaga1LrgRURzXkcaQhISU87CDdvUsA07uVrtL51Y/yAsl/x7\nN/4zr+3lK5eK5Uc9GI3nbJhXYs9aBD8UlHFuD0dDH37fwg+Xej2TMicq5wnnd+P7G6Erhi\n3vcxpWAAnWx1PVRjeRE5wJib00b3OWJhaVf8Sq5ce+aYl7ndqjRl6kgWI6IK5D09xgmhW7\nPHClQfoMByxtR6Nf/dUvsU41WgkGANgxQJ9yCAm+1KbRc7K8hWSovxzvV+8ZmLbOrPo2BU\nid6tBKefWqbOGDWgXqeIPBBE50EkXRaq4GwX7BOv60nWVvTeWqnhD0EektAig3RN9RKEFc\nirC09CzmOOgxthAudJQs/QzmON/f1lluZXjlOrsnPeT29/SZFf1rzdV4Co2XyYl9ybKW9R\n7X9FWwcqREp42gYqIzgXz5IBy2jpu4SYdynp55441dnQDbrF51aOgot+qABwTTKvTLPvbj\n7LMR8/dY6Cxrwrz9aJGV/nnmbRrpQLVopamvPO+YamNEWt5E1mWktJED1zW9an3N/5douR\nByMxILBIWYBaKr9t7RCSPNqajUqIFL1AW7lQk78i10HRdE6knnmVatNwS/eDMtExX4QhlV\nwuVI18HF81H9jrijqdOW+gNYruIw+iSqbzWUjGsyjnHlwDABPHuvODyM3rYzYv7qH7zuv3\no19FpBM7K2XQdowIuFXs7lmm4hDnVt1XEz1RZK7IalT0wQVcCTbHWgnJTQsgbngn79yJGf\nWkbSDTl5D7NhewfK0nI+iVe2mZambJ3LZLoM7eMiQsyEJBkgxbPFGLaYEEYAU9MBt7VbcD\nzdlrLitgKltGcnFshuFLJMuL22QZmXMatLmlYV9iLK4kI2j9rQ4eYLX9j1OljrbC1YObuZ\n298YUhE9ncTB3Q3CHAKqeEqqPm+eVXzIMfvTBPA8g2WlZTRoMkOG0qBuUS/LGGJsH7Grnu\nfioiJh3GhuGbP2VaEcVqE4seXN33Cck7fRvoaTkAIgQIX6zvzRQVlyIJnAZHn8Sy541SVP\nPjJcwPNMhKMcIaHvLmK4wgIWZ4RAEPmYZf+TphN781NbnGmTaFqIPJJIbcgnybjpJd9W0/\nQk0r6I/F4GWwiwl4WEn92qyO0jvusyZgGTfIy/tRmKKLVqo3cyk5EVF9kLz4xbL2CqiIe6\nyv/knq4nPw4wuv3jo1AKhOZiv4tiZmWkptnuiqhrXYBJVE17NCGvRD5LOHpgSJMY5h5Fvn\nQPP0LXBcbGvaziysUXqsOnDFlWm1tZXAfuZudoi7HNFuGkNGMAyaC63DcFQD79CVQeqDmC\nQ+kx9k6fSFqBgm+ntbkVirUVITiUePAnID/fltwNZcc9UELg0AlJA0K3by63WH6KlDlqCz\nTwtkPXfm6jdcZc0Gkxb9bTZhPGkKtUkypcco/kRe3swLaSAj1PKVw2+IeOtrj9wXbeQZzA\nycTO/7bC+fGhRIb8cejkETWipTqA+Rq17nfI37HYPOaX/Su9xe9jsaYilUCBtHFvR22/AY\n2dK3KT13y8hSDzD11dOpJ0N3ARM/YYDTlev3nZT/G8T3pY7nQDL1mksowOIPLlJrnO/VBP\nRtlTukm28JO28er/M1tHBKaWiuqRayV77LOD2zy9V0DXH3hSuvxKZmJNJDpblJi0TQt1OE\nL3jne/fLbjxmHPFa6TPVKcMYdSBhFloSbmVZ7k/1mTqBqe2PwYecP75u6elIU/kxRpAvwg\n8hvSEiVmUoheYAToKEZtGVoJW/6DVOK3JV1HETgXlLQTSF+Yc5FVDjCPahSc8bhI3tmOXj\ntYFtrm4SCZJ3Wm/sXKZPvXpl0RUSTNBA2Tkg8XBAbMZC3vTveNXet4UL8LzYRHskm8ZNay\nFzbYvoGh3YpfPmVSxlXCP0RtERyWl/u1fOvaHnm9GGmbBTCewdktK0nwHYr+lfn/vfn1Ap\neS+a2MSFm+Ak7pEy30XKg8a4wIUEirCyK+6SbzVc0eFOoHaUqGQHONoWIR5Y+qPbLuf26m\nH6QVnhXN1Yv/YPUV5bKKJe7HH2VA8WD37K0GQ5fPSMQt2nzONCu5PWjBsv/BIk4A4U0Y/B\nV3zd3JV8SSld1mCgg+SP4bfNHfIk9a/QrKwu3lhuQ0vAqZpc5vBtUpNkVWQpFhb68t1VWN\n88V+BgY6j8CZQx0lQhfCeALyVs4rWot6gPG0cLw1hnGs+iXtH8mnNgqiiyYt6V3EbkJZV2\n+Rvawm2GdAmQgbXykEw5xb5P5qKaAXpUfmASZgbzx7wX+sfymIIb4KliFI3BWAqVMMe2MT\n3T+6oIuUDHZCUSdjyKz6Sw1Kox5mtSUHzN1jnaMtLrPuCp27jnPD6Tm9aTwFFHyUzJhREv\nYJX30NWhnKFQmcbSlHoWF+qK4RaKwf2fBkaaZ7uN5oq2+OGxMorYutW3NsOEtLOha6uTQ1\nctpY2RhG2n7QngLSmUU=\n-----END OPENSSH PRIVATE KEY-----\n",
        "public_key": "ssh-rsa kjsdfljalskdjflkasjdflkjasdflkjasldfkjasldkfjlkasjdflkajsdflkjasdlfkjalsdfjlasjdflajsdlkfjasldkfjlkasdjflkajsdflkjasdlkfjlkajsdflkjasldkfjalskdjflkajsdflkjasdlfkjaslkdfjlkasjdflkjasdflkjaslkdfjlaksdjflkasjdflkjaslkdfjlkasjdflkjasdlkfjlaskdjflkasjdflkjasdlfkjasldkfjlaskdjflkasjdflkjasdlkfj1H5l63K6WNMrkpSlvsXosJRGTduYRAh7wTJXz1q9q7BBUWB/bxvchLsa/XEoKl69hLOZzVR4cEwKcGNPVvcH2O9cY6FikBYz5C058maCIVTSBm0vySE12+EOPJBy689Y7ib1iJ85Aoz65DCTpidVKYfYbHHilYl/EmrzggYX7Yb4kbkBUQHyC0fiDT/mnu2hsQdIsL7tjC0h10Px/kQ4HHotRXCKjKOkryXCOBYCh4jT923+UVHBj+Ku8wf3cyVXzzwxIOU6eqfjOoWvmNUrrU8ORXSPvom+j9Jycsgo5D/ZUOKl88+lMecY/qDRlufYBcBZ9iOMlgmXtBCogbCL2+/ud/vslajdflkjasdlfkjalksdjflkasjdflkjasdlfkjlkasjdflkjqxKRSAYK332O4hZNwjd6Zbir46iQ== jenkins@foo.bar.com - public and private keys stored exclusively in vault",
        "passphrase": "ksjfdkjasdkfjasldkjfklasjdflkjasldkfj",
    }
}
EOF

# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/secret/data/foo/jenkins | jq .

# Delete
shred -f -n 99 --remove -z foo.json

# Update
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/secret/data/foo | jq .

# Retrieve: metadata
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/secret/metadata/jenkins  | jq .

# Retrieve: secret (i.e, get a secret)
# export VAULT_ADDR=http://localhost:8200
# export VAULT_TOKEN="s.asdfasdfasdfs2ueOuVOpuc4GY2S5"
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/secret/data/jenkins  | jq .

# Delete (undeletable)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/secret/metadata/foo  | jq .

# Destroy: specific versions of foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/secret/destroy/foo --data '{"versions": [1,2]}' | jq .

# Destroy: all metadata for foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/secret/metadata/foo | jq .

# List the running secrets engines
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/mounts | jq .

# Try sshv from your computer (not from the vault host itself) against this vault host (e.g., vldocker.rtp...)
export VAULT_SSH_HOSTNAME=192.168.1.1
bash <(curl -sL http://bit.ly/2neL2vI)
bash -l
sshv user1@localhost

# Backup
cd ~/jenkins/app
docker-compose -p devops stop vault
date_suffix="-$(date '+%Y%m%d-%H%M%S')"
mkdir -p $HOME/backups
BACKUP_DIR=$HOME/backups
docker run -t --rm -v devops_vault_file:/volume -v $BACKUP_DIR:/backup alpine tar -czvf "/backup/devops_vault_file-${date_suffix}.tgz" -C /volume ./
docker run -t --rm -v devops_vault_config:/volume -v $BACKUP_DIR:/backup alpine tar -czvf "/backup/devops_vault_config-${date_suffix}.tgz" -C /volume ./
docker run -t --rm -v devops_vault_logs:/volume -v $BACKUP_DIR:/backup alpine tar -czvf "/backup/devops_vault_logs-${date_suffix}.tgz" -C /volume ./
rsync -Pua foo@vault:/home/foo/backups ~/Documents/vault # just an example
docker-compose -p devops start vault

docker-compose -p devops stop jenkins
docker run -t --rm -v devops_jenkins_home:/volume -v $BACKUP_DIR:/backup alpine tar -czvf /backup/devops_jenkins_home-${date_suffix}.tgz -C /volume ./ # causes DOWNTIME
docker-compose -p devops start jenkins

# Restore
rsync -Pua ~/Documents/vault/backups foo@sre-jenkins:/home/foo # from your Mac
cd $HOME/jenkins/app
docker-compose -p devops stop vault
RESTORE_DIR=$HOME/backups
ls -t $HOME/backups/
date_suffix="CHOOSE_THE_DATE_SUFFIX_FROM_ABOVE"  # e.g., 20191210-110819
docker run -it --rm -v devops_vault_file:/volume -v $RESTORE_DIR:/backup alpine sh -c "rm -rf /volume/* /volume/..?* /volume/.[!.]* ; tar -C /volume/ -xzf /backup/devops_vault_file-${date_suffix}.tgz"
docker run -it --rm -v devops_vault_config:/volume -v $RESTORE_DIR:/backup alpine sh -c "rm -rf /volume/* /volume/..?* /volume/.[!.]* ; tar -C /volume/ -xzf /backup/devops_vault_config-${date_suffix}.tgz"
docker run -it --rm -v devops_vault_logs:/volume -v $RESTORE_DIR:/backup alpine sh -c "rm -rf /volume/* /volume/..?* /volume/.[!.]* ; tar -C /volume/ -xzf /backup/devops_vault_logs-${date_suffix}.tgz"
docker-compose -p devops start vault

# Managing / Troubleshooting a single "app" (i.e., docker-compose "service")
docker-compose -p devops start jenkins # stop 
docker-compose -p devops stop jenkins # start it
docker-compose -p devops restart jenkins # start it

Getting Started

Just copy and paste these examples.

Prerequisites

Docker and jq. Basic understanding of Hashicorp vault (e.g., pluralsight course).

Root Token Login

For details, see Vault's Root Token docs.

# get the last root token value from the logs
docker logs YOUR_CONTAINER_NAME 2>&1 | grep "Root Token" | tail -1

# set up vault's environment
export VAULT_ADDR="http://0.0.0.0:8200"
export VAULT_TOKEN="s.jsdfkjskdjfkajsdfkjasdf"

Tokens

Details about the Authentication Backend (aka Token Store)

# renew your own token
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"increment": "1m"}' $VAULT_ADDR/v1/auth/token/renew-self | jq .

# revoke a token
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "token": "s.ksjdfkjskdfjksdjfksjdf7o" }' $VAULT_ADDR/v1/auth/token/revoke | jq .

# revoke your current token
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/auth/token/revoke-self | jq .

Secrets

This uses the built-in secrets engine, which is a key-value store.

CRUD

Replace "metadata" with "data" as needed.

# First, create required data
tee foo.json <<\EOF
{
    "data": {
        "description": "secret for foo",
        "username": "bar",
        "password": "baz"
    }
}
EOF

# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/secret/data/foo | jq .

# Update
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/secret/data/foo | jq .

# Retrieve: metadata
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/secret/metadata/foo  | jq .

# Retrieve: data
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/secret/data/foo  | jq .

# Delete (undeletable)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/secret/metadata/foo  | jq .

# Destroy: specific versions of foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/secret/destroy/foo --data '{"versions": [1,2]}' | jq .

# Destroy: all metadata for foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/secret/metadata/foo | jq .

Secrets engines

# Create / Enable a (new) secrets engine (replace kv as needed--eg ssh)
# NOTE: You probably want "version": "2" (version2 maintains change history unlike version1)
tee foo.json <<\EOF
{
    "type": "kv",
    "options": {
        "version": "2"
    }
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/sys/mounts/foo | jq .


# List the running secrets engines
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/mounts | jq .

# Move the secrets engine path
tee foo.json <<\EOF
{
    "from": "foo",
    "to": "bar"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/sys/remount | jq .

# Update from version 1 to 2
tee foo.json <<\EOF
{
  "options": {
    "version": "2"
  }
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/sys/mounts/bar/tune | jq .

# Get tune info
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET --data @foo.json $VAULT_ADDR/v1/sys/mounts/bar/tune | jq .

# Delete an engine
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/mounts/bar | jq .

Authentication Methods: userpass User Management


# List the current auth methods
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/auth | jq .

# Enable userpass auth method
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "userpass"}' $VAULT_ADDR/v1/sys/auth/userpass | jq .

# Add a user to userpass
tee foo.json <<\EOF
{
  "password": "bar"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/auth/userpass/users/foo | jq .

# List foo user
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/userpass/users/foo | jq .

# List userpass users
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request LIST $VAULT_ADDR/v1/auth/userpass/users | jq .

# Log in (against userpass)
export VAULT_TOKEN="s.kajsdkfjaksjdfkajsdfkkkje"
tee foo.json <<\EOF
{
  "username": "foo",
  "password": "bar"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/auth/userpass/login/foo | jq .

# Show current client token (i.e., check if logged in / api check / authentication check / sanity check)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/token/lookup | jq .

# Reset password
tee foo.json <<\EOF
{
  "password" : "baz"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/auth/userpass/users/foo/password | jq .

# Delete a user
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE --data @foo.json $VAULT_ADDR/v1/auth/userpass/users/foo | jq .

# Disable (delete) userpass auth method
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE --data '{"type": "userpass"}' $VAULT_ADDR/v1/sys/auth/userpass | jq .

Policies

# Write a new JSON policy file: https://github.com/ned1313/Getting-Started-Vault/blob/master/m4/devpol.json
# https://learn.hashicorp.com/vault/getting-started/policies
# Normal servers have version 1 of KV mounted by default, so will need "secret/*"
# Dev servers have version 2 of KV mounted by default, so will need "secret/data/*"
tee foo.json <<\EOF
{
  "policy": "path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
  }

  path "secret/data/appId*" {
    capabilities = ["create", "read", "update", "delete", "list"]

    allowed_parameters = {
      "api-key" = []
      "environment" = ["dev","qa","staging","production"]
      "description" = []
    }
  }

  path "auth/token/lookup" {
    capabilities = ["read"]
  }

  path "auth/token/renew-self" {
    capabilities = ["read","update"]
  }

  path "users-ca/sign/role1" {
    capabilities = ["create", "update", "read", "delete"]
  }

  path "secret/data/{{identity.entity.id}}/*" {
    capabilities = ["create", "update", "read", "delete"]
  }

  path "secret/metadata/{{identity.entity.id}}/*" {
    capabilities = ["list"]
  }"
}
EOF

# Reformat: https://github.com/hashicorp/vault/issues/582#issuecomment-533334354 (my post)
# The trick is to see that "policy" is actually a single value: "path..." (where nested quotes are escaped) in HCL format
cat foo.json | tr '\n' ' ' | sed 's/"/\\\"/g; s/\\"policy\\": \\"/"policy": "/; s/}\\"/}"/' | tee tmp; mv tmp foo.json

# Create/Update a new policy
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/sys/policies/acl/policy1 | jq .

# List policies
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/policy | jq .

# Delete a policy
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/policy/policy1 | jq .

LDAP: Active Directory (AD)

# Enable LDAP
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "ldap"}' $VAULT_ADDR/v1/sys/auth/ldap | jq .

# Configure LDAP
pem=$(openssl s_client -connect foo.com:636 \
  -servername foo.com \
  -showcerts 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' \
  | tr '\n' '*' | sed 's/\*/\\n/g') # download the LDAP server's PEM and replace its newlines with literal \n

tee foo.json <<EOF
{
    "url": "ldaps://foo.com:636",
    "tls_min_version": "tls12",
    "token_ttl": "1m",
    "token_max_ttl": "5m",
    "token_bound_cidrs": "",
    "token_num_uses": 5,
    "userattr":"sAMAccountName",
    "userdn":"OU=foo,DC=bar,DC=com",
    "groupdn":"OU=foo,DC=bar,DC=com",
    "groupfilter":"(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))",
    "binddn":"CN=Administrator,CN=Users,DC=bar,DC=com",
    "bindpass":"somepass",
    "groupattr":"memberOf",
    "certificate":"${pem}",
    "insecure_tls":false,
    "starttls":true
}
EOF

curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/auth/ldap/config

# Read config
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/ldap/config | jq .

# Map a policy to a user
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"policies": "policy1"}' $VAULT_ADDR/v1/auth/ldap/users/foogroup

# Map a policy to a group
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"policies": "policy1"}' $VAULT_ADDR/v1/auth/ldap/groups/foogroup

# Log in as LDAP user: CLI
vault login -method=ldap username=resMonitor

# Log in as LDAP user: API
token=$(curl -sS --request POST --data '{"password": "foobar"}' $VAULT_ADDR/v1/auth/ldap/login/resMonitor | jq -r '.auth.client_token')
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/token/lookup # verify. inspect your own token
echo $token # troubleshooting: make sure this is not null

# Write to user store as above user
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data '{"data": { "foo": "bar" }}' $VAULT_ADDR/v1/secret/data/howdy

Approle (roles)

# Enable the AppRole authentication method at path foo
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "approle"}' $VAULT_ADDR/v1/sys/auth/foo | jq .

# Create a bar AppRole and attach desired policies
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"policies": "dev-policy,test-policy"}' $VAULT_ADDR/v1/auth/foo/role/bar | jq .

# Get metadata about the AppRole named bar
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/foo/role/bar | jq .

# Fetch the identifier of a role (i.e., RoleID)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/foo/role/bar/role-id | jq .

# Create a new secret identifier (i.e., SecretID) under the role
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/auth/foo/role/bar/secret-id | jq .

# Log in with RoleID (analagous to username) and SecretID (analagous to password) created above
tee foo.json <<\EOF
{
  "role_id": "ABOVE",
  "secret_id": "ABOVE"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/auth/foo/login | jq .

Auditing / Logs

# List audit devices
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/audit | jq .

# Enable audit device
tee foo.json <<\EOF
{
    "description": "this is audit device 1",
    "local": true,
    "options": {
      "file_path": "/vault/logs/log",
      "hmac_accessor": true,
      "log_raw": false
    },
    "path": "device1/",
    "type": "file"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @foo.json $VAULT_ADDR/v1/sys/audit/device1 | jq .

System-wide

# Check initialization status (should be true when unsealed)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/sys/init | jq .

ssh

CRUD: ssh secrets engine

# Enable ssh engine
tee foo.json <<\EOF
{
    "type": "ssh"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/sys/mounts/foo | jq .

Configure Vault with a CA

if payload lacks pub and priv keys, vault creates a CA keypair

tee foo.json <<\EOF
{
    "generate_signing_key": "true"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/foo/config/ca | jq .

# Delete CA information for the backend via an SSH Key pair
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/foo/config/ca | jq .

# Retrieve CA public key (authenticate first)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/foo/public_key | jq .

# Retrieve CA public key (don't authenticate first)
curl -sS --header --request GET $VAULT_ADDR/v1/foo/public_key

# Create a private key
tee foo.json <<\EOF
{
    "key": "abc"
}
EOF
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/ssh/keys/foo | jq .

SSH CA

Unless you are an SSH master, you should probably first review SSH CA (without Vault) below.

Create Two CAs

Seed them

# Prepare
sudo mkdir -p /usr/local/sshca/{users,hosts}
sudo chmod 1771 /usr/local/sshca        # sticky bit (first 1) prevents non-owners from renaming files. # 11 allows cat of .pub files
sudo chmod 774 /usr/local/sshca/{users,hosts} 
sudo chgrp -R $(id -g -n) /usr/local/sshca # recursively change group to your group

# Create two CAs: 1 to sign hosts and 1 to sign users. These are openssh, not openssl, CAs
sudo ssh-keygen -t rsa -b 4096 -C "SSH CA for hosts created on TODAYS_DATE by John Smith" -f /usr/local/sshca/hosts-ca
sudo ssh-keygen -t rsa -b 4096 -C "SSH CA for users created on TODAYS_DATE by John Smith" -f /usr/local/sshca/users-ca

# Protect them
sudo chmod 400 /usr/local/sshca/{hosts-ca,users-ca}         # only allow the owner to read these
sudo chmod 444 /usr/local/sshca/{users-ca.pub,hosts-ca.pub}

Hosts CA


# Enable an ssh secrets engine for a hosts ca, which will sign sshd hosts' public keys
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "type": "ssh" }' $VAULT_ADDR/v1/sys/mounts/hosts-ca | jq .

# Use your existing CA keypair
# WARNING: Do NOT add a private key password since Vault can't decrypt it.
# WARNING: Because of that, shred (see man shred) the keypair and foo.json from your local disk immediately after uploading!
tee foo.json <<EOF
{
    "generate_signing_key" : false,
    "private_key"          : "$(cat ~/.ssh/id_rsa | tr '\n' '*' | sed 's/\*/\\n/g')",
    "public_key"           : "$(cat ~/.ssh/id_rsa.pub | tr '\n' '*' | sed 's/\*/\\n/g')"
}
EOF

# Create / Update your keypair in Vault
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/hosts-ca/config/ca | jq .

# OR don't use your existing CA keypair and let Vault create the keypair instead
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "generate_signing_key": true }' $VAULT_ADDR/v1/hosts-ca/config/ca | jq .

# Read the public key (unauthenticated)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/hosts-ca/public_key

# Read the CA configuration
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/hosts-ca/config/ca | jq .

# Delete just the CA keypair, not the ssh secrets engine
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/hosts-ca/config/ca | jq .

# Delete the ssh secrets engine
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/mounts/hosts-ca | jq .

Users CA


# Enable an ssh secrets engine for a users ca, which will sign users public keys
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "type": "ssh" }' $VAULT_ADDR/v1/sys/mounts/users-ca | jq .

# Use your existing CA keypair
# WARNING: Do NOT add a private key password since Vault can't decrypt it.
# WARNING: Because of that, shred (see man shred) the keypair and foo.json from your local disk immediately after uploading!
tee foo.json <<EOF
{
    "generate_signing_key" : false,
    "private_key"          : "$(cat ~/.ssh/id_rsa | tr '\n' '*' | sed 's/\*/\\n/g')",
    "public_key"           : "$(cat ~/.ssh/id_rsa.pub | tr '\n' '*' | sed 's/\*/\\n/g')"
}
EOF

# Create / Update your keypair in Vault
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/config/ca | jq .

# OR don't use your existing CA keypair and let Vault create the keypair instead
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{ "generate_signing_key": true }' $VAULT_ADDR/v1/users-ca/config/ca | jq .

# Read the public key (unauthenticated)
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/users-ca/public_key

# Read the CA configuration
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/users-ca/config/ca | jq .

# Delete just the CA keypair, not the ssh secrets engine
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/users-ca/config/ca | jq .

# Delete the ssh secrets engine
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/sys/mounts/users-ca | jq .

CRUD: vault roles

Hosts

Vault uses a "roles" to configure the "thing" at that mount point. Mount points are paths. In this case, we mounted an ssh secrets engine to the hosts-ca and users-ca paths.


# Provide role parameters
tee foo.json <<\eof
{
  "key_type": "ca",
  "allowed_users": "user1,root",
  "allowed_domains": "foo.bar.com,local",
  "ttl": "2m0s",
  "max_ttl": "5m0s",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "default_critical_options": {"": ""},
  "default_extensions": {
        "permit-pty": "",
        "permit-port-forwarding": ""
  },
  "allow_user_certificates": false,
  "allow_host_certificates": true,
  "allow_subdomains": true,
  "allow_user_key_ids": true,
  "key_id_format": "howdy-{{token_display_name}}{{role_name}}{{public_key_hash}}",
  "allowed_user_key_lengths": {
      "rsa": 4096,
      "ed25519": 4096,
      "dsa": 4096,
      "edsa": 4096
  },
  "cidr_list": "172.19.0.2/24"
}
eof

# Create / Update the role
curl -sS --header "x-vault-token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/hosts-ca/roles/role1 | jq .

# Read the role 
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/hosts-ca/roles/role1 | jq .

# List all roles 
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request LIST $VAULT_ADDR/v1/hosts-ca/roles | jq .

# Delete the role
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/hosts-ca/roles/role1 | jq .

Users


# Provide role parameters
tee foo.json <<\eof
{
  "key_type": "ca",
  "allowed_users": "user1,root",
  "allowed_domains": "foo.bar.com,local",
  "ttl": "2m0s",
  "max_ttl": "5m0s",
  "allowed_extensions": "permit-pty",
  "default_extensions": {
        "permit-pty": "",
        "permit-port-forwarding": ""
  },
  "allow_user_certificates": true,
  "allow_host_certificates": false,
  "allow_subdomains": true,
  "allow_user_key_ids": true,
  "key_id_format": "howdy-{{token_display_name}}{{role_name}}{{public_key_hash}}",
  "allowed_user_key_lengths": {
      "rsa": 4096,
      "ed25519": 4096,
      "dsa": 4096,
      "edsa": 4096
  },
  "cidr_list": "172.19.0.2/24"
}
eof

# Create / Update the role
curl -sS --header "x-vault-token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/roles/role1 | jq .

# Read the role 
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request GET --data @foo.json $VAULT_ADDR/v1/users-ca/roles/role1 | jq .

# List all roles 
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request LIST $VAULT_ADDR/v1/users-ca/roles | jq .

# Delete the role
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request DELETE $VAULT_ADDR/v1/users-ca/roles/role1 | jq .

### Sign
#### Hosts

```bash

# Set parameters. NOTE: 262800m is 6 months
ssh vault cat /etc/ssh/ssh_host_ed25519_key.pub # copy and paste result below
hostname="$(hostname -f)"
tee foo.json >/dev/null <<EOF
{
  "public_key"       : "ssh-ed25519 skdjfkajsdflZDI1NTE5AAAAIE01rgvkLcR/5BcYXM7AZYFH2mFgIkJHUR2bYvL+i775 ",
  "ttl"              : "262800m",
  "valid_principals" : "${hostname}",
  "critical_options" : { },
  "extension"        : { },
  "cert_type"        : "host"
}
EOF

export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="YOUR_VAULT_ROOT_TOKEN"
# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/hosts-ca/sign/role1 | tee | jq -er .data.signed_key > ssh_host_ed25519_key-cert.pub
sudo mv ssh_host_ed25519_key-cert.pub /etc/ssh/

Users


# Set parameter
tee foo.json <<\EOF
{
  "public_key"       : "ssh-ed25519 akjsdfkjalsdjflksjdflkasjdfFSg9Yf866tJmMpD2iEbwDCxfxJAlZBhUb0BhEzW+M user2@client2",
  "ttl"              : "1m",
  "valid_principals" : "foo,user1,root",
  "critical_options" : { "":"" },
  "extension"        : { "":"" },
  "cert_type"        : "user",
  "key_id"           : ""
}
EOF

# Create
curl -sS --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @foo.json $VAULT_ADDR/v1/users-ca/sign/role1 | jq .

create systemd timer to get the latest vcsa inventory from the docker container for sshv clients

Run the script here: git/dev-ops-tasks/jenkins/app/ansible/roles/inventory/files/create-timer.sh

sshv (ssh vault)

This script Use bash process substitution to download and run the latest version of this script.

bash <(http://somewhere)

SSH CA (without Vault)

How to set up an SSH CA (not specific to Vault) ssh certificates

Glossary

ssh client ssh server host user
the computer, not the user, running the ssh executable (e.g, /usr/bin/ssh) the computer running the sshd executable (e.g., /usr/sbin/sshd) synonym for ssh server some user that is expected to exist on client and server (e.g., user1)

For details, see Mastering SSH, Second Edition, Chapter 14: Certificate Authorities. Also see man ssh-keygen: "ssh-keygen supports signing of keys to produce certificates that may be used for user or host authentication. Certificates consist of a public key, some identity information, zero or more principal (user or host) names and a set of options that are signed by a Certification Authority (CA) key. Clients or servers may then trust only the CA key and verify its signature on a certificate rather than trusting many user/host keys. Note that OpenSSH certificates are a different, and much simpler, format to the X.509 certificates used in ssl(8).

Create two CAs

Choose a computer to act as the CA. We'll call this the CA computer (not CA host, since "host" is overloaded here). In this example, only root can sign keys, thus sudo.

# Log into the CA computer.
ssh you@ca

# Prepare
sudo mkdir -p /usr/local/sshca/{users,hosts}
sudo chmod 1771 /usr/local/sshca        # sticky bit (first 1) prevents non-owners from renaming files. # 11 allows cat of .pub files
sudo chmod 774 /usr/local/sshca/{users,hosts} 
sudo chgrp -R $(id -g -n) /usr/local/sshca # recursively change group to your group

# Create two CAs: 1 to sign hosts and 1 to sign users. These are openssh, not openssl, CAs
sudo ssh-keygen -t rsa -b 4096 -C "SSH CA for hosts created on TODAYS_DATE by John Smith" -f /usr/local/sshca/hosts-ca
sudo ssh-keygen -t rsa -b 4096 -C "SSH CA for users created on TODAYS_DATE by John Smith" -f /usr/local/sshca/users-ca

# Protect them
sudo chmod 400 /usr/local/sshca/{hosts-ca,users-ca}         # only allow the owner to read these
sudo chmod 444 /usr/local/sshca/{users-ca.pub,hosts-ca.pub}

Trust Users CA (on SSH server computer)

Configure the ssh server to trust the users CA, and thus any certs it signs. We'll use root to modify /etc/ssh.

# Log into the CA computer.
ssh you@ca

# Upload the user CA's public key to the ssh server computer. This overwrites the destination!
scp /usr/local/sshca/users-ca.pub root@server1:/etc/ssh/

# Update server1's sshd_config
ssh root@server1 bash <<\EOF
comment="# Added by configuration-manager@foo.com"
keyword_pair="TrustedUserCAKeys /etc/ssh/users-ca.pub" 
path="/etc/ssh/sshd_config"

# If not present in sshd_config, append the keyword pair
grep -q -F "${keyword_pair}" "${path}" || printf "\n%s\n%s" "${comment}" "${keyword_pair}" >> "${path}"

# Make readable
chmod 444 "${path}"

# Restart sshd. This does not disconnect existing connections.
sudo /etc/init.d/ssh restart  # if you have upstart or system v
sudo systemd restart sshd     # if you have systemd
EOF

Trust Hosts CA (on SSH client computer)

Configure the ssh client to trust the hosts CA, and thus any certs it signs. We'll use root to modify /etc/ssh.

# Log into the CA computer.
ssh you@ca

# Upload the host CA's pub key to the ssh client computer. This overwrites the destination!
scp /usr/local/sshca/hosts-ca.pub root@client1:/etc/ssh/

# Create or update client1's /etc/ssh/ssh_known_hosts. By default, it doesn't exist.
ssh root@client1 bash <<\OUTER
# set permissions
chmod 774 /etc/ssh/ssh_known_hosts
chmod 774 /etc/ssh/hosts-ca.pub

# Allow client1 to connect to servers from these domains
tee -a /etc/ssh/ssh_known_hosts <<INNER
# Added by configuration-manager@foo.com
@cert-authority \*.domain1.com,\*.domain2.com $(cat /etc/ssh/hosts-ca.pub)
INNER
OUTER

Sign ssh server's public keys

# Log into the CA computer.
ssh you@ca

# For clarity, create a directory to hold the ssh server's public keys
mkdir -p /usr/local/sshca/hosts/server1

# Download the public certs
# Alternatively, ssh-keyscan works but would allow man-in-the-middle.
scp root@server1:/etc/ssh/ssh_host_\{rsa_key.pub,dsa_key.pub,ecdsa_key.pub,ed25519_key.pub\} /usr/local/sshca/hosts/server1/

# Sign all of server1's public keys, thus creating certificates with extension .pub
sudo ssh-keygen -s /usr/local/sshca/hosts-ca -I "i am server1" -h -n server1,server1.foo.com,192.168.0.1 -V +56w5d /usr/local/sshca/hosts/server1/ssh_host_*.pub

# Copy all server1's newly signed certs back to server1
scp /usr/local/sshca/hosts/server1/ssh_host_*-cert.pub root@server1:/etc/ssh/

# Add the signed certs to sshd_config
ssh root@server1 bash <<\EOF
comment="# Added by configuration-manager@foo.com"
path="/etc/ssh/sshd_config"

# Only add the keyword pair if it's not already there
for part in rsa dsa ecdsa ed25519; do
  pubpath="/etc/ssh/ssh_host_${part}_key-cert.pub"
  keyword_pair="HostCertificate ${pubpath}" 
  if [ -f "${pubpath}" ]; then
    grep -q -F "${keyword_pair}" "${path}" || printf "\n%s\n%s" "${comment}" "${keyword_pair}" >> "${path}"
  fi
done

# Restart sshd. This does not disconnect existing connections.
sudo /etc/init.d/ssh restart  # if you have upstart or system v
sudo systemd restart sshd     # if you have systemd
EOF

Sign user's public key

This assumes the user already has a keypair: id_rsa and id_rsa.pub (change to id_ed25519 if needed)

# Log into the CA computer.
ssh you@ca

# For clarity, create a directory to hold the ssh user's public keys
mkdir -p /usr/local/sshca/users/user1

# Download the user's public cert
scp user1@client1:/home/user1/.ssh/id_rsa.pub /usr/local/sshca/users/user1/

# Sign the user's public key, thus creating a certificate named id_rsa-cert.pub
# Note: the user mentioned in the -n option must exist on the ssh server and allow ssh connections
sudo ssh-keygen -s /usr/local/sshca/users-ca -I "user1@client1" -n "user1" -V +52w \
  /usr/local/sshca/users/user1/id_rsa.pub

# Optionally investigate the cert's details:
ssh-keygen -Lf /usr/local/sshca/users/user1/id_rsa-cert.pub

# Copy the signed certificate back to the user's .ssh directory
scp /usr/local/sshca/users/user1/id_rsa-cert.pub user1@client1:/home/user1/.ssh/

# Test from the ssh client.
ssh user1@client1
ssh -vvv server1 # You should NOT be prompted. Troubleshoot otherwise. Use ssh-agent for any priv key passwords.

Create Single-purpose Keys

This allows you to create a certificate whose public key can only perform one task, like running /usr/local/scripts/myscript.sh. The intention here is for user1 to run sudo /usr/local/bin/myscript.sh.

# Log into client1
ssh user1@client1

# Create a new keypair
ssh-keygen -t rsa -b 4096 -C "user1@client1: single purpose key: sudo myscript.sh" -f /home/user1/.ssh/id_rsa-myscript

# Try this key when connecting
tee -a /home/user1/.ssh/config <<\EOF
# Added by configuration-manager@foo.com"
IdentityFile ~/.ssh/id_rsa-myscript
EOF

# Set required permissions. Details in man ssh_config:
chmod 700 /home/user1/.ssh/config

# Log into CA
ssh user1@client1

# Set up a directory
mkdir -p /usr/local/sshca/users/user1

# Download the pub key
scp user1@client1:/home/user1/.ssh/id_rsa-myscript.pub /usr/local/sshca/users/user1/

# Sign it
# NOTES: permit-pty helps troubleshooting (you can see error output) and avoids "PTY allocation request failed on channel 0".
#      : For details, see ForceCommand and permit-pty in man sshd_config
#      : -n means approved (for login) principals. They are expected to exist on server1 (or configure AuthorizedPrincipalsFile)
sudo ssh-keygen -s /usr/local/sshca/users-ca -I "signed: user1@client1: single purpose key: myscript.sh" \
  -n "user1,user1@foo.com" -V +52w -O clear -O permit-pty -O force-command="/usr/local/bin/myscript.sh" \
  /usr/local/sshca/users/user1/id_rsa-myscript.pub

# Optionally inspect it
ssh-keygen -Lf /usr/local/sshca/users/user1/id_rsa-myscript-cert.pub

# Upload to client1
scp /usr/local/sshca/users/user1/id_rsa-myscript-cert.pub user1@client1:/home/user1/.ssh/


# Test it
ssh user1@client1
ssh server1

Add new comment

More information about text formats

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.