How to build and deploy confidential computing microservice applications with EGo & MarbleRun

In this article, we are going to look at how cloud applications are built today and how the principles of confidential computing can be applied to them. We will also show you how easily this can be achieved by using our tools EGo and MarbleRun.

When we started our development cycle some time ago, we asked ourselves one central question "How would you like to build confidential computing applications?". What we found is that there is a large community called the Cloud Native Landscape, who have built a vast ecosystem of open-source tool stacks. When people develop new cloud services, they take those dev ops development tools and build their applications in a microservice architecture. This allows for separating tasks into logical parts which can then be scaled up individually. Furthermore, any part can be exchanged, which makes it a great foundation for an agile development lifecycle.

From this, we took away two things: First, we wanted to facilitate exactly this type of development process for confidential computing apps. Secondly, if we want to help make the cloud a more secure space, we knew we had to offer a technology based on the same principles, so anyone can jump on board and adopt our tools for the cloud native space. Our main concern was to make everything as easy and as convenient as possible, allowing for a seamless integration into an existing tool stack. What we essentially ended up with are two tools: one tool to build confidential microservices and another tool for all the dev ops of administering and deploying microservices to your cloud.

The easy way to build confidential Microservices

Our solution for building confidential cloud apps is called EGo, which is a compiler based on the Go programming language. There were many reasons to choose Go: First off, it’s one of the leading languages used by cloud native developers. Secondly, it’s easy to get into and allows you to do a lot in a very short time. Since so much in the cloud native space was already written in Go, it was only natural to adopt the language for confidential computing development. But concerning CC development, what we found was a very tedious process of taking the specific compiled Go binary, building the enclave around it and packaging the entire thing into a container. To us, this did not feel right, which is why we came up with EGo.

With EGo, you don’t just build a Go binary — you build the whole enclave. Your process still follows the essential Go commands of “build” and “run”, but adds “sign” in between, which gives you the benefits of confidential computing. To illustrate this, we demonstrated these steps using Hashicorp Vault as a showcase for more mature projects. This popular tool is a way to store secrets in the cloud native space. In essence, you start with “ego env make”, which wraps the build process and injects EGo in the right spots. Afterwards, just add “sign” and now you have a vault server running inside an enclave, ready to be deployed in the cloud.

So, what does EGo get you?

With EGo, you get secure enclaves that are convenient and straightforward to use. Each enclave created with EGo is lightweight, since it only consists of your application code, some Go runtime and a small compatibility layer for the confidential computing technology. Because EGo is built on top of Open Enclave, all low-level content it based on already established standards — everything you build with EGo can potentially run everywhere. EGo also offers you the same build and debug process as you normally would have with Go — depending on your needs, you can run your binary on its own, the Open Enclave simulation mode or in the full enclave. Building Microservices this way makes it easy to bring them to the cloud for scaling things up, which is where MarbleRun comes into play.

The challenges of confidential microservices

When we started to deploy Go microservices that run enclaves inside the cloud, we suddenly faced an entirely new set of challenges. First and foremost, we had a cluster of numerous services which each ran inside separate enclaves — but this is not something an end-user would care about. Ideally, a user would receive one single attestation for the whole cluster, and this gets even more important when scaling things up. You want one concise statement. Secondly, there is the issue of service communication: if you want to enable data sharing between parts of the same service, how does one part know that the other is actually running inside an enclave and it’s safe to share the data? Also, if you want to distribute secrets, e.g., if there is a web front end and you would like all instances to use the same certificate, how can this certificate be distributed? Keep in mind, once the certificate leaves the enclave boundaries, it is not considered confidential anymore.

For these reasons, we needed a solution to distribute secrets in a confidential way. This was especially difficult with sealing, which is one of the key properties of confidential computing. In one of our first attempts, we deployed a confidential app in the cloud. This deployment was then rescheduled, so it was shut down on one node and restarted on another node. This completely failed because the sealing key had changed. Naturally, our node was sealed with the physical key of the first node and then the app tried to unseal it with the physical key of the second node.

We needed to make sure that some kind of communication was possible, and data could be unsealed independently from single nodes. Still, even with virtual sealing keys, we wanted our connections to be terminated inside enclaves, so you never have any connection that transfers data in plain to the outside. Or for example, what about code updates? Imagine there is a new update of our enclave code, how do we ship it to all parts of the cluster without violating the trust of the clients? In a nutshell, our goal was to enable a transfer from individually encrypted enclaves for each part of an app toward an overarching remote attestation statement that guarantees the security of an entire cluster.

The service mesh for confidential computing

MarbleRun is our solution to address all these concerns. With MarbleRun, you can bring attestation to your whole cluster --- and this in turn brings you end-to-end confidentiality and verifiability for an entire structure of microservices. As stated in the introduction, our goal was to keep everything cloud native and allow for updates. If you are familiar with service meshes, you might ask yourself "can't there be a way to solve these problems with existing service meshes?" and we found that this is not the case. Normally, a service mesh considers the host or at least the host machine as a trusted party, so it terminates the TLS communication in a separate container (a sidecar) and forwards the unencrypted traffic through the host into another container, where the data is then being processed. This hurts the confidentiality, because secrets are distributed in plain and end up in the non-enclave world. But our approach was not to replace any of the existing systems in place --- we wanted to augment the existing ecosystem to keep the cool features that are already available. We want you to be able to deploy your favorite service mesh and just add MarbleRun to help you turn it into a confidential service mesh.

Imagine you have several services in a microservice architecture. Each of these services is running in an enclave, so the first thing we do with MarbleRun is to inject a small piece of code inside this enclave. This takes place on the data plane layer or service mesh layer, which handles all the tasks a service mesh needs to do. On top of this data plane layer, which is typically referred to as the control plane, we added an additional service. The control plane takes care of orchestrating all services and implementing the confidential computing features discussed previously. By using this separate-layer approach, the security solutions can be implemented without having to adjust the application itself.

From a dev-ops perspective, this adds quite a bit of convenience. The dev ops engineer does not really need to know very much about CC. Instead, they can use whatever tooling they had and just inject this small data plane code into their application --- everything else, such as scaling things up, updating code, building the application itself can be done as per usual. And from the user perspective, this is even better: in just one step, you can verify that everything runs in an enclave and everything is encrypted. You can verify the topology of the mesh as well as the identity of the cluster and verify what kind of services are running.

What does MarbleRun get you?

With MarbleRun, your entire cluster becomes end-to-end encrypted and end-to-end verifiable. If you are on the outside and care about the security of the microservice you use, you do not need to worry about individual services anymore. With its cloud-first approach, MarbleRun is compatible with all common service meshes and tools such as Kubernetes, helm, linkerd, and much more. It is built on the same Open Enclave standards as EGo, so together they deliver very fast and easy deployments. We currently support the library OSes Graphene and Occlum, which opens up diverse possibilities for the lift and shift. Applications can be heterogenous, with services based on EGo, Graphene and Occlum running in the same mesh. On top of that, MarbleRun is cloud agnostic, so if your cloud provider supports confidential technology, MarbleRun should work there out of the box.

This blog post was delivered as a talk by Moritz Eckert (Chief Architect, Edgeless Systems) at the Open Confidential Computing Conference 2021. You can watch the full recording on YouTube.