Re-Creating My Homelab Public Key Infrastructure

A while back I wanted to get more into Public Key Infrastructure and learn more how it works, and how I can create a more secure homelab even for internal traffic. It started out fairly benign, from work I learned that Windows server has a built in PKI suite with Active Directory Certificate Services. The link to the write up on that will be here.

If you are a fellow home labber you will know that you don't really understand a new technology until you do it over again. When you initially deploy new features to learn you do not know what features you need, you just browse reddit and find out what is the best. Well you will learn soon enough that always what you have deployed is not the best for your use case. This is the case with my PKI (Public Key Infrastructure).

When I initially started my journey I though ADCS (Active Directory Certificate Services) was the way to go, mainly due to the automatic enrollment and provisioning of certificates. I have since learned not only that I am hating Windows more and more, but also most of my certificates that I am provisioning are on servers and not clients. ADCS does not provide much for auto provisioning of certificates, with the only option being NDES or Network Device Enrollment Service. NDES enrolls certificates via SCEP, which is not only an already basic protocol, but I still need to access a web page sign in and get an access code in order to provision certificates.

I also have started moving away from my Windows stack as I have a Domain Controller at home and would love to de-commission it since I am not utilizing any of its advanced features since I don't need to manage multiple computers. I also find Windows to be too bloated and hide a lot of important configuration away from the user. The core focus of Windows is to provide a Next -> Next -> Next -> Finish experience which is not tailored to many people. I find in the long run a headache to remember all the small details and power shell scripts you need to run on Windows in order for it to be secure and reliable.

My end goal for my Public Key Infrastructure is the ability to provision short lived certificates to all of my services. The world is slowly trying to move to short lived certificates to get rid of Certification Revocation Lists, and Online Certificate Status Protocol. Feel free to read more about it from this paper written in 2012. The short of it is we can worry less about revoking certificates if they already last a few days. I think this is a verry good idea as I can already see more than half of my traffic in my lab is just on CRLs and OCSP queries.

Praise be Hashicorp Vault

A while back I created a Hashicorp Vault instance to provide better secrets management to my Kubernetes cluster I have a blog post if you want to learn more. Hashicorp Vault is verry extensible to fit your secret management needs. They also have a PKI secret management that will allow you to provision certificates via the vault API, or what I need the most is ACME. ACME or Automatic Certificate Management Environment, is a protocol that verifies control over a domain over two main ways, either a TXT record in the DNS, or a simple webserver serving a challenge which the DNS name points to. It is a really easy, and really popular way of provisioning certificates automatically. Automation is the name of the game as it allows for short lived certificates.

Configuring The New PKI Infrastructure

As I want to move off of Windows ADCS, I created a brand new PKI infrastructure with a new root certificate. Once again as I wanted this to be extra secure, I went ahead and created an offline root certificate. One of the main hurdles when creating certificate authorities is how you store your private key. Your private key is used to sign certificates, and thus need to be in a secure place or someone could sign their own keys to malicious software/site. One of the most recommended ways is by encrypting a USB with software such as Veracrypt, and loading your private key in there. This is exactly what I decided to do.

For the purposes of my short tutorial the configs and keys will be placed in /root/ca. Make sure to make the directory, as well as the serial file to increment certificate serial numbers:

mkdir -p /root/ca /root/ca/certs /root/ca/crl /root/ca/newcerts /root/ca/private
touch /root/ca/index.txt
echo 1420 > serial

Here is the command that I used to create my private key, with the openssl configuration:

# OpenSSL root CA configuration file.

[ ca ]
# `man ca`
default_ca = CA_root

[ CA_root ]
# Directory and file locations.
dir               = /root/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
# Match names with Smallstep naming convention
private_key       = $dir/root_ca_key
certificate       = $dir/root_ca.crt

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 25202
preserve          = no
policy            = policy_strict

[ policy_strict ]
# See the POLICY FORMAT section of `man ca`.
# Change these to add in restrictions to your CA on what certificates it can sign I disabled this as I ran into some encoding problems
#countryName             = match
#organizationName        = match
commonName              = supplied

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
commonName                      = Common Name
countryName                     = Country Name (2 letter code)
0.organizationName              = Organization Name

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

Create and sign the new private key:

# Generate a private key (needs a passphrase, don't forget the passphrase)
openssl genrsa -aes256 -out /root/ca/root_ca_key 4096

# Sign a 10-year certificate
openssl req -config /root/ca/root.cnf -key /root/ca/root_ca_key -days 3650 -new -x509 -sha256 -extensions v3_ca -out /root/ca/root_ca.crt

Once the new root authority has been set up, we need to also set up Vault as an intermediary certificate authority. To do this we need to enable and configure the PKI secrets engine. Upon going into the secrets engine it will prompt you to configure the new PKI engine. Here you will need to generate an intermediate CSR (Certificate Signing Request), so you can sign it with your root certificate. This will also create a new private key for the intermediate CA on vault. Once you generated and downloaded the CSR, go to your root CA, and sign the cert with the below command.

openssl ca -config /root/ca/root.cnf -keyfile /root/ca/root_ca_key -cert /root/ca/root_ca.crt -extensions v3_intermediate_ca -days 1095 -notext -md sha256 -in /root/ca/intermediate_ca.csr.pem -out /root/ca/intermediate_ca.crt

I signed my intermediate for 3 years, if you would like you can extend this up to the length of your root certificate, though I think 3 years is a good length. Once the CSR is signed you can then go back into the vault pki secret engine configuration and click on upload a certificate and upload the certificate you just signed.

Complete the CA Chain within Vault

One other thing that you need to do now that you have configure the PKI secret engine, is to add in the root certificate. When you issue certificates within vault it will respond with not only your certificate but then entire certificate chain. This is a necessity as your clients wont trust the certificate unless they are given a valid chain all the way to a certificate they trust, such as the root certificate. You can add in the certificate chain after the fact though it is easier for automations if you can just have vault respond with the whole chain.

It is very easy to add in a certificate, you don't even need to specify the certificate in the chain it will figure out which ones are intermediaries and which ones are the root. Get your root certificate and go to the Issuers tab in vault. Here you will click on the Import button and upload the PEM encoded root certificate. Once imported you should see a tag on the certificate noting it is the root certificate. This is all you need to do for vault to give you the entire certificate chain.

Configuring the Secrets Role

Like I mentioned in my article adding Hashicorp vault to my Kubernetes cluster vault roles define how credentials / certificates can be generated, or retrieved. For the PKI secrets engine the roles define how certificates can be signed. We will be adding a simple role to provision web certificates.

To create a role, just go into the Roles tab and clock on Create role + Here you can configure the role name, and some uses for the certificate. Under Domain handling this is the domains this certificate is allowed to be for. In the Allowed Domains enter your domain you want to sign certificates for such as internal.examle.com or even just example.com. I would check two boxes in here, Allow subdomains and Allow wildcard certificates. This will allow you to make service.internal.example.com and a wildcard certificate for *.internal.example.com. Finally feel free to add in additional Subject Fields to configure locality settings for your certificate.

Adding in an ACME Endpoint

Like I mentioned above, ACME or Automatic Certificate Management Environment is a way to automate control over a domain. It works in one of two ways, either adding in a TXT record into the domain with a challenge. The other way which is what I would be using more is to host a challenge file on a website the domain points to. If you are a fellow nerd like me you can read more about RFC 8555

To explain this a little more in depth, the client will reach out to an ACME server and request a certificate with certain attributes. If the ACME server is willing to issue a certificate it will present the client with a list of possible challenges to complete. The client will set up the challenges (Either TXT record or Web Challenge), and submit to the ACME server the challenge it has completed. The ACME server will verify the challenge has been complete and respond with an acknowledgement. The client will then request the issued certificate from the ACME server completing the ACME interaction.

To set this up is way simpler than it sounds. Vaults PKI secret engine by default comes with a RFC 8555 compliant ACME server you just need to configure and enable it. To start out go to the PKI configuration setting and add in the Mount's API path. This may be a little differently depending on your set up but this looked like the following https://<vault-domain>:8200/v1/<pki-mount>. Once that is in there check the box for ACME enabled. Within the Allowed roles specify the web-certificates role you just created. Next I highly suggest enabling EAB I will explain it more in the next paragraph but set EAB policy to always-required. Finaly for your Global URLs Use the following:

• Issuing Certificates: https://<vault-domain>:8200/v1/<pki-mount>/ca
• CRL distribution points: https://<vault-domain>:8200/v1/<pki-mount>/crl
• OCSP Servers: https://<vault-domain>:8200/v1/<pki-mount>/ocsp
• Delta crl distribution points: https://<vault-domain>:8200/v1/<pki-mount>/crl/delta

Once finished click Save on the bottom, and in the bottom right you should see 4 saved sucessfully boxes, and you can configure your ACME clients!

What is an EAB? An EAB or External Account Binding, adds in a small authentication layer to our ACME request. When you submit the request you add in they eab-kid, and the eab-hmac-key. This is higly suggested for internal PKI infrastructure to add in a bit of added security, incase there was some DNS spoofing going on. For public facing ACME servers such as letsencrypt there is no need to add EAB. In order to obtain a new EAB key id, and HMAC key run the following command within vault: vault write <pki-mount>/acme/new-eab You can also assign roles to EABs when you create them with this command: vault write <pki-mount>/roles/<role>/acme/new-eab

ACME Clients

My preferred ACME client of choice is Certbot. By default Certbot allows you to get certificates from the LetsEncrypt CA. They also automatically provision cron jobs to aide in automatic certificate renewals. Though we can edit some configuration settings and allow it to fit to our needs. In this post we will mainly be focusing on Web Challenges. There are two ways Certbot can complete Web Challenges, either via Standalone or Webroot. Standalone is perfect if you are not using port 80. It will spin up a temporary standalone server bound to port 80 and serve the web challenge. If you were to use Webroot, you need to specify which directory houses the web root directory. Certbot will then add the file in that directory, which will be hosted via your pre-existing web server on port 80.

If you are confused on which one to use, it boils down to if you have something already bound to port 80. A lot of times people have web servers on port 80 that will redirect users to the HTTPS counterpart. In this case you would use webroot. I will have examples below of HA-Proxy configurations that explain the webroot configurations.

Certbot Renewal Hook

If you are deploying your application behind HA-Proxy, and let HA-Proxy handle the certificate, you need to combine your key and cert chain into one file. Normally this would be a pretty manual process though Certbot allows for custom scripts during the certificate deployment step. Below is a Certbot Renewal Hook I created for my HA-Proxy applications.

Put this in a file at /etc/letsencrypt/renewal-hooks/deploy/haproxy-pem.sh

Then run chmod +x /etc/letsencrypt/renewal-hooks/deploy/haproxy-pem.sh

#!/bin/bash
# Automatically combine privkey + fullchain for HAProxy

TARGET_DIR="/etc/haproxy/certs"

# Ensure target directory exists
[ -d "$TARGET_DIR" ] || mkdir -p "$TARGET_DIR"

# Combine into single PEM
cat "$RENEWED_LINEAGE/privkey.pem" "$RENEWED_LINEAGE/fullchain.pem" > "$TARGET_DIR/$RENEWED_DOMAINS.pem"

# Set permissions
chown haproxy:haproxy "$TARGET_DIR/$RENEWED_DOMAINS.pem"
chmod 600 "$TARGET_DIR/$RENEWED_DOMAINS.pem"

# Check if HAProxy is active, then reload
if systemctl is-active --quiet haproxy; then
    echo "HAProxy is running — reloading..."
    systemctl reload haproxy
else
    echo "HAProxy is not running — skipping reload."
fi

When Certbot runs the bash script it will give it two environment variables.

• RENEWED_LINEAGE - Contains Certbot directory where it houses the private key and certificate chain
• RENEWED_DOMAIN - Contains the domain name in the renewal request

We use these to find where the key and certificate chain are to combine them into a file in /etc/haproxy/certs/<domain>.pem

Certbot Commands

I will give the two certbot commands that you will need, for both a standalone server, and a webroot. These are both fairly simple commands, and they will automatically set up a renewals cron job which will check if a certificate needs to be renewed.

Certbot - Standalone

sudo certbot certonly \
  --standalone \
  --preferred-challenges http \
  --server https://<vault-domain>:8200/v1/<pki-mount>/acme/directory \
  --eab-kid "<EAB-Key-ID>" \
  --eab-hmac-key "<EAB-HMAC-Key>" \
  -d <domain>

Certbot - Webroot

sudo certbot certonly \
  --webroot --webroot-path /var/lib/letsencrypt/ \
  --preferred-challenges http \
  --server https://<vault-domain>:8200/v1/<pki-mount>/acme/directory \
  --eab-kid "<EAB-Key-ID>" \
  --eab-hmac-key "<EAB-HMAC-Key>" \
  -d <domain>

If everything works out you should see Certbot succeeds, and provisions a new certificate in /etc/letsencrypt/live/<domain>/fullchain.pem

Closing Remarks

This is such a cool project to do and use if you are interested at all in Public Key Infrastructure. With vault I can automate tons of certificate requests and renewals and only have to worry about rotating my large important certificates such as my vault intermediate certificate, and my root certificate every 10 years.

Along side the above benefits this also moves me away from using windows within my environment and adding more programmability within my infrastructure. Hopefully in the future I will make a blog post on how to automate the configurations needed for automatic certificate requests, this should be pretty easily done with Ansible. With regards to my Kubernetes cluster, I have no need to provision certificates on there since most certificates that go on there are production certificates that I need everyone to accept so I use letsencrypt instead of my PKI stack.

I can definitely do more with this current PKI stack than what I would be able to do with Windows. I can also have more specificity in my certificates, possibly moving to a Elliptic Curve key stack, though I am not sure if that will conflict with some of my future projects such as 802.1x port authentication.