One of the key pieces of the Entando platform is the Entando operator that allows us to automate all of the tasks that go into managing a modular architecture on Kubernetes. There are a couple of things that make the Entando Operator different from many other Kubernetes operators. The first is that it is written entirely in Java and the second is that the operator uses Quarkus for many of the tasks that it performs.

In this post we’ll go over how we use Quarkus in the operator, the benefits it brings, and some lessons we learned along the way.

Why Quarkus?

When we built the operator we made the choice to do everything in Java (look for a future blog post on this topic). When thinking about the operator we had a few main requirements in mind:

1. Modularity is key. The platform is evolving. We need the ability to add new capability quickly and in a low risk way
2. Speed is important. Many of the actions the operator takes have a user or developer waiting on the other end
3. Resource consumption is important. Resource utilization in Kubernetes has a direct relationship to cost and total cost of ownership
4. Focus on standards. Use the tools and techniques available in Kubernetes and Java so that the operator can be easily consumed by the internal team and developers using the platform

With those constraints in mind we found that Quarkus helped address many of our needs. Paired with the “Kubernetes Job” pattern we had the tools to achieve many of our goals. Some of the key features that led us to this choice include:

1. Quarkus keeps us in the Java based ecosystem
2. The Quarkus run to completion jobs start very quickly. Usually in less than 50ms
3. The jobs consume significantly less memory than a traditional JVM. Helps keep our cluster level resource consumption lower
4. We are able to take advantage of extensibility in both the code and the image builds. We’ve got a set of operator-common code that we use across all of the operator controllers (not necessarily specific to Quarkus but useful)

The primary alternative we considered was building the operator and the jobs in Go. The ultimate choice to go with Java and Quarkus has some trade offs when compared with Go

1. Go is “more native” to Kubernetes. It is the default language for most Kubernetes infrastructure code
2. Go would be smaller than even a Quarkus based image and would start as fast or faster in most cases
3. The available documentation for Go for operators and Kubernetes controllers is much greater than for a Java based implementation

In the end the speed of Quarkus and the value of keeping the entire platform in Java led us to go with the final architecture you see in the platform today.

How it Works

When you deploy an Entando instance you’re creating a deployment that includes the operator. The operator then reads the configuration provided in the rest of the helm template and deploys your instance of the platform.

During that deployment the operator creates Kubernetes Jobs that manage the tasks required to complete installation of the platform. In particular, the jobs that are created are Run to Completion jobs and the containers that are deployed use Quarkus. 

The operator uses those Quarkus based jobs to complete many parts of the installation including:

1. Restoring a database backup for an application or microservices
2. Linking an Entando microservice (plugin) to an application
3. Deploying the Entando App Engine
4. Deploying a database in a container or connecting an app to an external database
5. Deploying an instance of Keycloak for an Entando App

We also use a run to completion job at runtime when a user deploys a microservice (plugin) from the Entando Component Repository (ECR). The workflow in this case is:

1. User hits install in the App Builder
2. API call is made to the Entando component manager
3. A custom resource is deployed for the microservice by the Entando cluster infrastructure service
4. Operator is listening for that custom resource in the namespace or cluster
5. Operator spins up a Quarkus based run to completion job for the microservice
6. Once the microservice is deployed the operator spins up a Quarkus based run to completion job to link the new microservice to the Entando application
7. End user gets notified and the services are available for consumption in the Entando App

When you’re deploying an Entando instance you’ll see the pods for these jobs start and complete if you’re watching the namespace. Some examples are highlighted in the screenshot below.

Lessons Learned

There are a number of things that we learned along the way about building an operator using Quarkus and run to completion jobs. This pattern and the tools we selected have many of the same tradeoffs and requirements that you see in normal Java projects.

However, the design choices we made amplify some of the challenges, and some aspects of the project ecosystem were more critical than we originally anticipated. This is a quick list of some of the challenges we ran into and things to be aware of. 

1. Pipelines, pipeline design, and CI/CD are critical and need to be reliable. Invest earlyPipelines are critical in every project but the modular design of the operator made the pipelines even more critical in this architecture 
2. Value pipeline reliability very highly. If your pipelines aren’t solid or reliable you’ll suffer
3. For example, if we change something in a base image or common project we may need to get builds through as many as 10 projects. Early on in the lifecycle our pipelines and build infrastructure was a little flaky. This led to a lot of discussions about “babysitting the builds”. You don’t want to be a build babysitter. Get this right early on and it’ll pay dividends.
4. Make sure you understand and plan for the constraints requirements for building native imagesIt takes a long time and a lot of resources to build a native image. Make sure your build environment is ready for it
5. Consider cloud based build systems that you can scale as needed
6. Develop an integration test plan early. Even if you know it will need to evolveCreate an environment where you can automate the deployment and integration testing of your operator and your jobs. 
7. Unit tests are always important but in this case the interplay between git satKubernetes, the jobs, and the operator means that you’ve got to run it all to understand how it all works together
8. Think about how you’ll test and develop. In particular, the role of JVM based images (if any) in your test and development environmentsWe have docker builds for both native and JVM. The ability to do most development and testing in a JVM based build can save a ton of time and resources. This has been a great choice (and it looks like it is a common practice in Quarkus projects)
9. Educate your teams on how and when to use both native and JVM based builds
10. Make a native-focused test plan to complement any work you do in JVM builds. If all of your testing is in JVM builds you’ll eventually run into a problem

Links and Other Resources

The Entando Operator is open source. If you want to learn more or dive into the code, checkout some of the links below for the Quarkus run to completion jobs

Some of our Quarkus controllers

1. https://github.com/entando-k8s/entando-k8s-database-service-controller
2. https://github.com/entando-k8s/entando-k8s-keycloak-controller
3. https://github.com/entando-k8s/entando-k8s-plugin-controller
4. https://github.com/entando-k8s/entando-k8s-app-controller

You can find the other code for the operator and the rest of the architecture in our two GitHub organizations:

1. https://github.com/entando-k8s
2. https://github.com/entando

Deploy an instance yourself and look for those Quarkus run to completion jobs during deployment

1. https://dev.entando.org/

Build better apps faster with Entando.

When release cycles take months instead of weeks, your business is left unable to deliver modern online experiences. You might run containerized deployments and have backend modular, but for many organizations, the frontend monolith remains. And it’s slowing innovation. 

So that’s why we created a platform to help you get your ideas to market faster.

Entando is the leading modular application platform for building enterprise web apps on Kubernetes. We want to change the way enterprises think about building their apps, sites, and portals in order to innovate more quickly.

With Entando, you can:

1. Leverage customized blueprints using the Entando Component Generator (built on JHipster) to quickly create micro frontends and microservices and assemble them into an app.
2. Reuse UI/UX components across multiple projects via the Entando Component Repository, saving money and increasing development speed. 
3. Scale apps quickly and effectively with Entando’s custom Kubernetes operator, automating the deployment of high availability, self-healing applications. 
4. Centrally manage user roles with Entando Identity Management (built on Keycloak).

Entando is open source with available enterprise support and services. Begin developing on the platform today, and get a quote to see how our team can help your enterprise build better apps, sites, and portals--faster.