Sign in

Sentry on-premise deplyoment via Helm3 on Kubernetes with custom CA certificates

Sometimes, when you are searching for some solution, you may not be able to find the one matching your use case exactly. This was my case as well.

We use Sentry (https://sentry.io/) as application monitoring platform, installed as on-premise on Kubernetes cluster using Helm. Until we used Let’s Encrypt certificates everything worked fine, but sh.. happens and someone decided to use our own Root CA.

Small change in the beginning …

Oops! Something went wrong.
Oops! Something went wrong.
Oops! Something went wrong (with certificates :))

If we open the web pod log, the error message is clear and obvious:

MaxRetryError: HTTPSConnectionPool(host=’sentry.domain.com’, port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)],)”,),))

It is logical, the app calls self URL which is behind the ingress secured with certificate which CA is not trusted by the app itself.

The web pod is based on Ubuntu, so you can use kubectl cp command to copy the CA certificate(s) to the pod filesystem (/usr/local/share/ca-certificates/ folder) and run update-ca-certificates and then sentry upgrade commands.

But this change is not persistent, every time the Kubernetes recreates the pod from deployment manifest our certificates are lost.

Deployment patch

Create the yaml file with the ConfigMap with the your CA certificate(s) in PEM format, for example:

apiVersion: v1
kind: ConfigMap
metadata:
name: cm-ca-erts
namespace: app-namespace
data:
root-ca.crt: |
—-—--BEGIN CERTIFICATE-—-—-
...
-----END CERTIFICATE-----
intermediate-ca.crt: |
—-—--BEGIN CERTIFICATE-—-—-
...
-----END CERTIFICATE-----

Create the yaml file with the patch of the Sentry Web Deployment:

spec:
template:
spec:
containers:
- name: sentry-web
command: ["/bin/bash","-c","cp /ca-certs/*.crt /usr/local/share/ca-certificates/; update-ca-certificates; sentry run web"]
volumeMounts:
- mountPath: /ca-certs
name: ca-certs
volumes:
- name: ca-certs
configMap:
name: cm-ca-certs

Let’s clarify it

  • Created config map will be mounted to the pod filesystem into /ca-certs folder
  • Certificates are then copied to the /usr/local/share/ca-certificates folder and update-ca-certificates command is executed to refresh (and add all new) trusted certificates
  • Finally, Sentry Web is started

Create the deploy script where you install or upgrade your Sentry instance and apply the the new yaml files, for example:

#!/bin/bash# install or upgrade Sentry
helm upgrade --install sentry sentry/sentry -f values.yaml
# apply the config map
kubectl apply -f cm-ca-certs.yaml
# patch the deployment
kubectl patch deployment sentry-web --patch "$(sentry-web-patch.yaml)"

That’s it, every time you redeploy or install Sentry instance, it is “patched” to support your own CA certificates.

Of course, there are some cons, for example whenever you make a small change in the deployment, where the web service redeployment is not required, it is redeployed anyway.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store