How to setup Mattermost on the Kubernetes engine Constellation

Kubernetes is the most popular cloud orchestration platform, making it simple to deploy and manage containerized and distributed applications.

Constellation is the most secure Kubernetes distribution, providing full encryption of data through confidential computing technology. For this reason, we recently set up a Mattermost on Constellation.

Mattermost is an open-source, self-hostable online project management software with file sharing, search, and other integrations. It is designed as an internal chat for organizations and companies and it is an open-source alternative to Slack and Microsoft Teams.

In this post, we will showcase how you can deploy Mattermost, a PostgreSQL cluster, and MinIO object storage on our confidential Kubernetes distribution Constellation; including database and object storage setup. The repo for this deployment also includes CI/CD pipelines, automatic SQL backups, and a backup restore pipeline. You can find it on GitHub. So, feel free to follow along and discover how to set this up for yourself!

Deployments

Most components in the repo are deployed via Helm charts. There are two charts in the repo, mm-infra and mm-app. mm-infra bundles charts that need to be deployed once per cluster, like operators and CSI drivers. mm-app bundles charts that are deployed once per environment (e.g., staging, dev, prod). An environment in this context is a namespace. Consequently, changes to the mm-infra deployment affect all environments. If you have higher requirements for stability than us, you might want to deploy mm-infra once per environment or use separate clusters for each environment altogether.

Operators

The mm-infrachart deploys multiple operators and allows you to configure which ones are deployed when you run helm upgrade. Deploying all charts at once can be done by selecting "all" when running the GitHub Actions workflow included in the repository.

The following are all the operators deployed as part of mm-infra, as well as the commands to deploy them:

Zalando's postgres-operator and its web UI for SQL cluster creation/management:

helm upgrade psql-op --install --atomic --set postgresOperator.enabled=true --
create-namespace --namespace psql-op mm-infra

cert-manager for automatic Let's Encrypt certificates. This automatically also deploys external-dns for automatic creation of DNS entries:

helm upgrade cert-manager --install --set certManager.enabled=true --create-
namespace --namespace cert-manager mm-infra

To setup access to your DNS provider for external-dns you need to create a secret:

kubectl create secret generic godaddy-access --from-env-file=./godaddy_creds -
n external-dns

ingress-nginx for traffic routing/TLS:

helm upgrade nginx-ingress --install --atomic --set ingressNginx.enabled=true 
--create-namespace --namespace storage mm-infra

azuredisk-csi-driverand the respective storage class for transparent disk encryption. We also have the analogous GCP driver:

helm upgrade storage --install --atomic --set azurediskCsiDriver.enabled=true 
--create-namespace --namespace storage mm-infra

mattermost-operator for the mattermost deployment:

helm upgrade mm-op --install --atomic --set mattermostOperator.enabled=true --create-namespace --namespace mm-op mm-infra

Lastly, the minio-operator needs to be deployed. However, this is done via a kubectl plugin like so:

wget -q https://github.com/minio/operator/releases/latest/download/kubectl-minio_linux_amd64.zip sudo unzip -o kubectl-minio_linux_amd64.zip sudo chmod +x ./kubectl-minio export PATH=$PATH:$PWD kubectl minio init

Apps

After installing the operators, you can deploy the mattermost app to different environments. Each deployment includes:

• A Mattermost resource. It's a custom resource defined by the mattermost-operator. Mattermost needs to connect to object storage (here: MinIO) and a SQL database (here: PostgreSQL). In the case of the database, a postgres connection string is saved in a secret. The MinIO connection is defined via a URL and bucket key inside the Mattermost resource. The credentials for the connection are provided in another secret.
• A Tenant resource, defined by the minio-operator. Each tenant is independent object storage that can hold multiple buckets, has its own service, etc. The resource defines some default credentials by referencing the storage-configuration secret, which buckets should be created on startup, the size of the tenant, and which users should have access to it. The usernames in the Tenant resource are names of secrets defined here.
• A postgresql resource, defined by the postgres-operator. The resource configures databases and users that should be created automatically on startup, the size of the cluster, the size of the underlying disks, what storage class to use for storage management, and how backups are recovered. For example, if you scale the mattermost-server by increasing the number of mattermost users, you will likely want to increase the specs for the SQL cluster.

SQL Backups

The postgres-operator manages backups for us. It will create a new backup once per day and save it to a separate bucket in MinIO using a dedicated user account. To configure backup creation, this ConfigMap is used. Backup creation is a feature of Spilo, an individual project the postgres-operator relies on. Therefore, reading about Spilo's environment variables is helpful if you want to modify the backup config.
The ConfigMap for backup configuration is available as environment variables in the pods through the postgres-operator. The relevant field of the postgres-operator is the pod_environment_configmap.

Restoring backups is configured in the postgresql resource by setting the clone key. If the clone key is present, data from the configured backup location is automatically put into the database.

Backup Quirk

While the database is correctly restored by adding the clone key, in our case, Mattermost needed some convincing until it accepted the backup. During regular operation, the postgres connection string is put into an environment variable called MM_CONFIG inside the Mattermost pod. When the pod starts for the first time, it will use the value of MM_CONFIG to connect to the DB. However, once it is successfully connected to the DB, it will read the settings for the mattermost-server from the DB. These settings also include a postgres connection string. At this point, the value in MM_CONFIG is no longer used by the mattermost-server. The connection string from the DB is used instead. Therefore, if the backup you just restored used different credentials for the database or the database was hosted under a different address, the connection will not work.

The simplest solution for our case was to ensure the address stays the same between backups. Unfortunately, passwords change since the postgres-operator creates new users every time a new cluster is created. Therefore, you can't set the password externally. To update the password, we manually read the settings for the mattermost-server from the DB after a backup was restored. Then, we update the password of the respective PostgreSQL user to match the password from the settings string.

So now, to initiate a backup restore, you can use one of the pipelines that we included as part of the repo.

CI/CD Pipelines

The repository includes GitHub Actions workflows to automate deployments and partially automate backup restoration. If you fork the repo, you can run these after updating the secrets. Three pipelines are relevant for managing the app:

• Deploy mm-infra: This workflow will upgrade the existing mm-infra deployment and install a new one if none exists already.
• Deploy mm-app: This workflow will upgrade the existing mm-app deployment for the configured environment and install a new one if none exists already.
• Restore backups: This workflow will restore the latest backup from the selected environment by deleting the current SQL cluster and creating a new one based on the backup. To migrate backups between environments, you will have to download the backup data from the backup-bucket in environment A and upload it to the backup-bucket in environment B. Before you start the upload to the backup-bucket in environment B, you should shut down the SQL cluster in environment B. You should also delete existing backups in environment B. After the upload is finished, run the CI action "Restore SQL backup" in environment B.

Encryption

Deploying chat applications in the public cloud requires careful data security and privacy considerations. Enabling e2e message encryption for Mattermost's chat via available plugins is essential in securing communication. However, these plugins don't address infrastructure-based threats. With the rise of cloud computing, attacks coming through the infrastructure are alarmingly becoming more and more numerous. For example, researchers discovered alarming cross-tenant vulnerabilities in Microsoft Azure. The Cybersecurity Insiders 2020 Cloud Security Report also found that organizations ranked malicious insiders (36%) among the top 10 cloud security threats. And supply chain attacks are a rapidly emerging threat, with a recorded 742% increase in volume over the past three years.

In the case of the e2e encryption plugin, the authors describe an example of how a privileged attacker with access to the infrastructure might break e2e encryption by serving malicious JavaScript to break encryption.

Constellation alleviates this problem by encrypting data in transport, at rest, and in use, relying solely on widely known projects and technologies to implement these features:

• In transport: traffic inside the cluster is encrypted using a Wireguard VPN managed by Cilium. Traffic outside the cluster is secured via TLS.
• At rest: all persistent storage is backed by our custom CSI driver that transparently encrypts block storage by utilizing dm-crypt.
• In use: main memory is encrypted by using AMD SEV-SNP-based confidential VMs on Azure.

Lastly, possibly the most crucial feature: the cluster's identity, integrity, and encryption are verifiable using remote attestation. That means we can trust our cluster and, consequently, our Mattermost deployment because we can verify it.

Combining these features allows us to run completely isolated, private, and always-encrypted Mattermost deployments in the public cloud while protecting against cross-tenant, malicious CSP insider, and supply chain attacks. In this way, we can be sure our internal communications are private and stay private!

Conclusion

This post outlined how to set up a Mattermost deployment on Constellation, a confidential Kubernetes distribution. The setup we showcased works as-is but can be adapted to fit your needs. Furthermore, while the setup is built on Constellation, it can be easily deployed on any Kubernetes cluster. In that case, however, you will lose the cluster VPN, storage encryption, memory encryption, and remote attestation.

Let us know what you would like us to deploy on confidential Kubernetes next!

Author: Otto Bittner