We use Sentry (https://sentry.io/) as application monitoring platform, installed as on-premise on Kubernetes cluster using Helm. Until we used Let’s Encrypt certificates everything worked fine, but sh.. happens and someone decided to use our own Root CA.
Small change in the beginning …
So we’ve changed Kubernetes ingress to use our own certificates. Nothing difficult, but after that we sometimes (for example registration of new user after email invitation) got message, that “something went wrong”…
If we open the web pod log, the error message is clear and obvious:
MaxRetryError: HTTPSConnectionPool(host=’sentry.domain.com’, port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)],)”,),))
It is logical, the app calls self URL which is behind the ingress secured with certificate which CA is not trusted by the app itself.
The web pod is based on Ubuntu, so you can use
kubectl cp command to copy the CA certificate(s) to the pod filesystem (/usr/local/share/ca-certificates/ folder) and run
update-ca-certificates and then
sentry upgrade commands.
But this change is not persistent, every time the Kubernetes recreates the pod from deployment manifest our certificates are lost.
Custom CA certificates are not supported in Sentry Helm chart, so for those who don’t want to modify the chart (which is a bit complex), here is the option.
Create the yaml file with the ConfigMap with the your CA certificate(s) in PEM format, for example:
Create the yaml file with the patch of the Sentry Web Deployment:
- name: sentry-web
command: ["/bin/bash","-c","cp /ca-certs/*.crt /usr/local/share/ca-certificates/; update-ca-certificates; sentry run web"]
- mountPath: /ca-certs
- name: ca-certs
Let’s clarify it
- Created config map will be mounted to the pod filesystem into /ca-certs folder
- Certificates are then copied to the /usr/local/share/ca-certificates folder and
update-ca-certificatescommand is executed to refresh (and add all new) trusted certificates
- Finally, Sentry Web is started
Create the deploy script where you install or upgrade your Sentry instance and apply the the new yaml files, for example:
#!/bin/bash# install or upgrade Sentry
helm upgrade --install sentry sentry/sentry -f values.yaml# apply the config map
kubectl apply -f cm-ca-certs.yaml# patch the deployment
kubectl patch deployment sentry-web --patch "$(sentry-web-patch.yaml)"
That’s it, every time you redeploy or install Sentry instance, it is “patched” to support your own CA certificates.
Of course, there are some cons, for example whenever you make a small change in the deployment, where the web service redeployment is not required, it is redeployed anyway.