The future of VM’s is Kubernetes

Kubernetes is the most powerful orchestration layer for containers, that is a fact. but is it just for containers?

“Kubernetes a platform to build platforms”, not just running workloads. You can introduce to kubernetes new objects (CRD), declare them with a manifest, control them with Kubernetes controllers, expose them with Kubernetes services, and connect them with the same network construct of kubernetes.

So, if you combine the statements above, why not use Kubernetes to manage VM’s as well?

The Answer, vSphere with Tanzu, VM service!

If you want to learn more about vSphere with Tanzu and the Kubernetes services integrated into vSphere, you can read about it here. Once you have that Kubernetes service builds in your hypervisor cluster, or should I say Supervisor Cluster, now you can extend the workloads that will be managed and run on that platform. When we announced vSphere with Tanzu, we exposed different services that can run and leverage the Kubernetes service directly. On ESXi, the main one is Tanzu Kubernetes Grid that allows you to provision and manage Kubernetes clusters with ClusterAPI on top of the Supervisor Cluster. Other services like Harbor and Data Persistence Platform, which run as vSphere Pods, allow you to run applications directly on the Supervisor Cluster.

As from vSphere 7 U2a, we added a new VM service that allows you to manage VMs with native Kubernetes API on top of the Supervisor Cluster. Just like any other service we enabled, once you configure the VM service on a supervisor namespace, you will be able to declare a VM object that will be post configured according to a configmap object and exposed with a network service object. Everything Kubernetes!

The service allows you to extend the DevOps experience and get a single API to manage any workload on vSphere with the Kubernetes Supervisor Cluster.

A quick demo of the service and capabilities:

Lets digg deep to the service configuration:

VM service configurationthe virtual infrastructure admin must pre-configure several parameters to set the ground for VM to be provisioned. Those parameters are part of the resource control and management and without that configuration, we wouldn’t be able to provision VMs. Let’s go over those parameters:

  1. Namespace service – with namespace service, we can enable specific SSO users to create vSphere namespaces with Kubernetes API. We must set resources boundaries to prevent so that namespace workloads can’t expand and consume all of the vSphere cluster resources. Those resource boundaries will be applied as quota policy on the new namespace automatically.
Namespace service enablement page
kubectl command to create new namespace
Qouta policies on the namespace

2. Machine class – Machine class is the size of the VM that will be created as part of the VM service. The configuration should state which machine classes the DevOps engineer can state in the manifest file.

Machine class namespace configuration

After bounding the machine classes to the Namespace, we can query Kubernetes for “VirtualMachineClass” and “VirtualMachineClassBindings” objects to see the classes that we can use as part of our declaration file in a namespace context:

Virtual machine classes
Vm classes that are bound to vmsvc namespace

3. Content library – The VM image files that will be the base for the VM need to be stored in a content library and bound into the namespace. Once configured, the DevOps engineer can state the OVA image he wants in the manifest of the VM object.

** The first version of the VM service will be shipphed with CentOS support followed by Ubuntu and other OVA **

Content library namespace configuration

After bounding the content library to the Namespace, we can query Kubernetes for “ContentSourceBindings” and “VM Image” objects to see the content library and the OVA we can use as part of our declaration file in vmsvc namespace context:

VM Images exposed to Kubernetes as VM Images

4. Configure VM manifest – Now, the DevOps engineer can create declaration files to provision and manage the VM’s with the Kubernetes API. Let’s go over a simple example of a YAML for VM declaration:

kind: VirtualMachine
    my-selector: ubuntu-services
  name: vmsvc-ubuntu-01
  - networkName: ""
    networkType: nsx-t
  className: best-effort-xsmall
  imageName: ubuntu-20.04-vmservice-v1alpha1.20210210.enabled
  powerState: poweredOn
  storageClass: k8s-storage-policy
    configMapName: vmsvc-ubuntu-cm-01
    transport: OvfEnv

We declare a “VirtualMachine ” type of object that’s the CRD for a virtual machine in the Supervisor Cluster. In the manifest file we also declare the network type, in this case NSX, the class and the image of the VM and the storage policy that indicates the datastore for the VM that will be provisioned.

The VM will be cloned from the ova as a unconfigured VM. We need to configure the VM to suit our needs of the specific app it is part of. For that, we use cloud-init post configuration in Linux OS. In the file below, configuring the cloud-init steps in another YAML file:

  - default
  - name: vmware
    lock_passwd: false
    passwd: '--------------------'
    shell: /bin/bash
ssh_pwauth: true

  - content: |
      VMSVC Says Hello World
    path: /home/vmware/helloworld

      source: deb [arch=amd64] $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

package_update: true

  - docker-ce
  - docker-ce-cli

  - sudo docker run --name nginx -p 80:80 -d

In the post configuration file, we can see a user and password configuration, print a new test file in a specific location. In this use case, as part of the demo video, I have installed docker on that Ubuntu, and I’m running a simple nginx web page exposing the container with port 80

To integrate that configuration into our VM service imprinting the file out in Base64 format and then incorporate that into a new config map object

base64 format
apiVersion: v1
kind: ConfigMap
    name: vmsvc-ubuntu-cm-01
    namespace: vm-operator
  user-data: I2Nsb3VkLWNvbmZpZwp1c2VyczoKICAtIGRlZmF1bHQKICAtIG5hbWU6IHZt

I’m referring to this config file in the VM manifest as OvfEnv under VM data, as you can see in the code below (you can see it in the VM manifest up as well)

  configMapName: vmsvc-ubuntu-cm-01
  transport: OvfEnv

Now that we have the VM manifest and the CM manifest, we need to expose the VM somehow, and for that, we will declare a new object (CRD) from kind VirtualMachineService who will expose a service type LoadBalancer with the ports and configurations we need

kind: VirtualMachineService
  name: ubuntu-services
  - name: ssh
    port: 22
    protocol: TCP
    targetPort: 22
  - name: web
    port: 80
    protocol: TCP
    targetPort: 80
    my-selector: ubuntu-services
  type: LoadBalancer

That service will expose the nginx web page from the container in port 80 and allow us to connect with ssh directly to the machine to configure or change anything else.

5. Deploy VM with Kubectl – Now to deploy the VM using Kubectl, and the manifests declared below:

VM manifest with configmap and network load balancer
Kubectl command to create the VM

The Kubectl command will create three Kubernetes objects: VirtualMachine, Configmap and VirtualMachineService. in the vCenter side we can see the new VM object under the namespace configuration <> compute tab in the virtual machine section. in the same way we can see the new network service under network section.

The VM service on vmsvc namespace

On the Kubernetes side we can query for VM, Service and endpoints for that load balancer service to check that everything is configured correctly:

Details about the VM in the Kubernetes side

We exposed port 22 for SSH and port 80 for web lets connect to the VM and brows the load blancer IP:

SSH to the VM
Browse the Nginx web page on the VM

By connecting to the machine with the username and password and browsing the Nginx web page, we know that the cloud-init phase went well, and everything is installed and configured according to what we declared in the cloud-config YAML file.

All the configuration files and YAML file are consolidated in my Git repo:

To summarize, I believe that the future of vSphere is Kubernetes management and control of any workload. It’s a vast frog leap forward in having a single, open, standard de-facto infrastructure orchestration API layer that can be consumed by automation and orchestration tools like vRealize Automation and others to automate everything, including VM’s.

2 thoughts on “The future of VM’s is Kubernetes

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.