Why Use Kubernetes as an Application?

Authors: Paul Carroll, Margaret Yang

Why Kubernetes?

Kubernetes is a powerful and popular container orchestration platform. When you are developing an enterprise application, you can use Kubernetes to containerize it and manage its lifecycle, which brings with it many benefits.

Kubernetes as an Application

In most cases the application users do not see Kubernetes itself, they see only the user interfaces and APIs provided by the application running on Kubernetes. But Kubernetes is extensible: A large community of developers and companies actively build extensions and plug-ins that add capabilities such as security, monitoring, and management to Kubernetes. Application developers can extend Kubernetes as well, often blurring the lines between the application and Kubernetes itself. It is relatively easy to use CustomResourceDefinitions and Controllers as part of an application running on Kubernetes, such that Kubernetes itself becomes the API for your application and kubectl becomes your CLI. Rather than creating and exposing a REST interface from scratch to manage data stored in a database, the Kubernetes API service can be used to manage data stored as CustomResources. Application users are Kubernetes users, application data is Kubernetes resources, and access is controlled by Kubernetes roles.

Kubernetes as Application Extension

The Challenge

Exploiting Kubernetes as part of the application rather than simply hostingthe application offers desirable capabilities to the developer. APIs, CLIs, data storage with fine grained access control — all of these are robustly provided by Kubernetes. However, application users access to Kubernetes itself is risky, akin to giving users operating system accounts on the server running an application so that the file system can be exposed as part of the application. The users with operating system accounts could find ways to install personal programs, access directories they should not have access to, or even destabilize the entire system. Similarly, Kubernetes users could create their own Pods running cryptocurrency miners, define Ingresses that preempt traffic that is intended for the application, or destabilize the cluster by consuming all available resources.

To use Kubernetes as part of an application requires special care to ensure application users, even application administrators, are safely restricted.

The most obvious approach is to simply define custom Kubernetes ClusterRoles along with your application that grant permissions only on the specific resources used by your application. For example, the application’s ClusterRoles can allow only create, update, and delete permissions on ConfigMaps, Secrets, or custom resource types provided by your application. The ClusterRoles would be bound to application users in specific namespaces monitored by your application.

The problem is that access management systems such as IBM Cloud Private’s IAM component are not aware of custom ClusterRoles provided by installed applications. If you grant a Team access to a Namespace, a RoleBinding is created for the Team members granting one of the default k8s roles — admin, edit, or view, not a custom ClusterRole from one installed application. But if the cluster administrator avoids using IAM to grant namespace access, they also lose the ability to leverage the Teams defined by IAM. Directly managing the RoleBindings from individual users to application ClusterRoles in each namespace becomes tedious at best.

All is not lost — it is possible to reconfigure Kubernetes to allow truly treating it as part of the application, including handling access through the Kubernetes default roles without giving users carte blanche! Kubernetes has four default user-facing roles: cluster-admin, admin, edit, and view. So that the cluster’s access management system can continue using these default roles, the cluster administrator will modify admin, edit, and view so that they are scoped down to application management permissions rather than full Kubernetes management permissions.

The Solution

Lets consider a hypothetical “WidgetManager” application that works by watching ConfigMaps (a standard Kubernetes resource type) and Widgets (a custom resource type defined by a CustomResourceDefinition included in the application), both managed by the application users. The application admin users will be able to create, read, update, and delete these two resources in their assigned namespaces, while less privileged application users will have read-only permissions on these resources in their assigned namespaces.

To set this up as the cluster administrator you will:

1. modify the Kubernetes default ClusterRoles
2. install the application
3. add application users as Kubernetes users
4. create some Namespaces for some different groups of application users
5. bind application users to the normal Kubernetes ClusterRoles in their assigned Namespaces

The WidgetManager application itself will create a ClusterRole granting Widget create/delete/update permission, specifying the rbac.authorization.k8s.io/aggregate-to-edit label so that any user with edit or higher role will be allowed to manage Widgets. A separate ClusterRole is also included granting Widget get/list/watch permissions, specifying the rbac.authorization.k8s.io/aggregate-to-view label so that any user with view or higher role will be allowed to view Widgets.

The key is the modification of the Kubernetes default ClusterRoles so that the only Kubernetes resources they provide permissions on are the ones that are used by the application. The Kubernetes default ClusterRole editinherits from ClusterRole system:aggregate-to-edit, which is where all the Kubernetes resource permissions are defined. As the cluster administrator setting up the WidgetManager application, you do not want users able to manage Deployments, Pods, Ingresses, etc. So you will modify the rules in ClusterRole system:aggregate-to-edit to have edit-verbs on only one resource: ConfigMaps.

• $ kubectl edit clusterRoles system:aggregate-to-edit to manually modify and delete all rules and resources except the one granting permissions on ConfigMaps.
• $ kubectl describe clusterRoles system:aggregate-to-edit to verify the alteration in ClusterRole system:aggregate-to-edit.

Name:          system:aggregate-to-edit
Labels:        kubernetes.io/bootstrapping=rbac-defaults
               rbac.authorization.k8s.io/aggregate-to-edit=true
Annotations:   rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
    Resources  Non-Resource URLs  Resource Names  Verbs
    ---------  -----------------  --------------  -------
    configmaps []                 []              [create delete deletecollection patch update]

• Repeat the same modifications to the system:aggregate-to-view and system:aggregate-to-admin roles to ensure no other permissions get added!
• $ kubectl describe clusterRoles edit to verify the same changes are reflected in default ClusterRole edit.

Name:         edit
Labels:       kubernetes.io/bootstrapping=rbac-defaults
              rbac.authorization.k8s.io/aggregate-to-admin=true
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
   Resources  Non-Resource URLs  Resource Names  Verbs
   ---------  -----------------  --------------  -----
   configmaps []                 []              [create delete deletecollection patch update get list watch]
   widgets    []                 []              [create delete deletecollection patch update get list watch]

The edit role gets its ConfigMaps permissions from the system:aggregate-to-edit role that you have modified, and the Widgets permissions from the application’s own ClusterRole that also specifies aggregate-to-edit.

Of course this doesn’t affect the cluster administrator — the cluster-adminrole continues to use wildcards for both the resources and the permissions on the resources. It doesn’t inherit or aggregate from the modified roles.

Summing Up

Using Kubernetes as an integral part of your application rather than as merely a container orchestration platform is achievable, and can allow creating a fully featured application quite easily. Rather than creating capabilities from scratch, leverage the capabilities of Kubernetes and make them your own! Several IBM products such as the IBM Cloud Pak for Multicloud Management use this approach, extending the Kubernetes API to provide application-level functionality. If you find your users taking liberties with the platform then the approach described here can provide relief, though it’s worth contacting your application support liaison first to ensure they will support the change — changes to Kubernetes are changes to the Application after all!