Sentry on-premise deployment via Helm3 on Kubernetes with custom CA certificates

Sometimes, when you are searching for some solution, you may not be able to find the one matching your use case exactly. This was my case as well.

We use Sentry (https://sentry.io/) as application monitoring platform, installed as on-premise on Kubernetes cluster using Helm. Until we used Let’s Encrypt certificates everything worked fine, but sh.. happens and someone decided to use our own Root CA.

Small change in the beginning …

So we’ve changed Kubernetes ingress to use our own certificates. Nothing difficult, but after that we sometimes (for example registration of new user after email invitation) got message, that “something went wrong”…

Oops! Something went wrong (with certificates :))

If we open the web pod log, the error message is clear and obvious:

MaxRetryError: HTTPSConnectionPool(host=’sentry.domain.com’, port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)],)”,),))

It is logical, the app calls self URL which is behind the ingress secured with certificate which CA is not trusted by the app itself.

The web pod is based on Ubuntu, so you can use kubectl cp command to copy the CA certificate(s) to the pod filesystem (/usr/local/share/ca-certificates/ folder) and run update-ca-certificates and then sentry upgrade commands.

But this change is not persistent, every time the Kubernetes recreates the pod from deployment manifest our certificates are lost.

Deployment patch

Custom CA certificates are not supported in Sentry Helm chart, so for those who don’t want to modify the chart (which is a bit complex), here is the option.

Create the yaml file with the ConfigMap with the your CA certificate(s) in PEM format, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-ca-erts
  namespace: app-namespace
data:
  root-ca.crt: |
    —-—--BEGIN CERTIFICATE-—-—-
    ...
    -----END CERTIFICATE-----
  intermediate-ca.crt: |
    —-—--BEGIN CERTIFICATE-—-—-
    ...
    -----END CERTIFICATE-----

Create the yaml file with the patch of the Sentry Web Deployment:

spec:
  template:
    spec:
      containers:
      - name: sentry-web
        command: ["/bin/bash","-c","cp /ca-certs/*.crt /usr/local/share/ca-certificates/; update-ca-certificates; sentry run web"]
        volumeMounts:
        - mountPath: /ca-certs
          name: ca-certs
      volumes:
      - name: ca-certs
        configMap:
          name: cm-ca-certs

Let’s clarify it

• Created config map will be mounted to the pod filesystem into /ca-certs folder
• Certificates are then copied to the /usr/local/share/ca-certificates folder and update-ca-certificates command is executed to refresh (and add all new) trusted certificates
• Finally, Sentry Web is started

Create the deploy script where you install or upgrade your Sentry instance and apply the the new yaml files, for example:

#!/bin/bash
# install or upgrade Sentry
helm upgrade --install sentry sentry/sentry -f values.yaml
# apply the config map
kubectl apply -f cm-ca-certs.yaml
# patch the deployment
kubectl patch deployment sentry-web --patch "$(sentry-web-patch.yaml)"

That’s it, every time you redeploy or install Sentry instance, it is “patched” to support your own CA certificates.

Of course, there are some cons, for example whenever you make a small change in the deployment, where the web service redeployment is not required, it is redeployed anyway.