Zone Support in vSphere with Kublr

Introduction

In vSphere infrastructure, default dynamic volume provisioner use the shared datastores wich is connected to all Kubernetes nodes. But in some cases, you don’t want to use these datastores, or perhaps you don’t have one, so using the zones can help you create a Kubernetes cluster on that infrastructure.

With enabling feature as zones support in Kubernetes, the cloud provider reads the topology of the cloud infrastructure and zones from vCenter. Kubernetes nodes and volumes will be labeled with zone information. And pod scheduler will use zone labeling to place pods on nodes.

Purpose

Quick overview on how to create VMware vSphere Kubernetes clusters with zone support.

Prerequisites

1. Look at the prerequisites that is required for vSphere.
2. Create and associate Tag Zones and Regions in vCenter

By default the Kublr uses two tag categories: “k8s-zone” and “k8s-region”. Use them to designate a zone and region in vCenter.

Before You Get Started

First download and install Kublr-in-a-box from here.

Creating VMware vSphere Cluster with Zone Support

Creating a cluster with Zone support is not much different from normal. The differences are highlighted in the sections below. Note that creating VMware vSphere cluster with zone support differs depending on the the Kublr version.

Since 1.28.0

1. On the CLUSTER tab, select:

   • Datacenter
   • Resource Pool

   See details on general parameters of VMware vSphere cluster.

   

2. Set Default Storage Class Zones via cluster specification as described below.

3. On the INSTANCES tab, describe at least two instance groups for worker nodes in different zones.

Before 1.28.0

1. On the CLUSTER tab, expand the Advanced Options section.

2. Select the Enable Zone Support option.

   

3. If your cluster does not have a shared Datastore which is connected to all virtual machines specify one of the parameters:

   • Default Storage Class Zones
   • Default Storage Class Datastore

   NOTES

   • You can also define these fields if you want to use a specific Datastore for default StorageClass.
   • Do not specify both sections at the same time.
   • See each option details and example in the sections below.

4. You don’t need to create “StorageClass” - Kublr will create it automatically.

5. On the INSTANCES tab, describe at least two instance groups for worker nodes in different zones.

Default Storage Class Zones

Since 1.28.0

Starting from version 1.28.0, for vSphere zone management, Kublr switches to 3.x version of VMware vSphere Container Storage Plug-in (previously called vSphere CSI Driver).

This implements the following changes:

• region_tag_name and zone_tag_name are deprecated
• topology_categories are now in use

Was:

spec:
  kublrAgentConfig:
    kublr_cloud_provider:
      vsphere:
        region_tag_name: DataCenterName
        zone_tag_name: ESXiHostsName

Defaults: region_tag_name: k8s-region, zone_tag_name: k8s-zone.

Now:

spec:
  kublrAgentConfig:
    kublr_cloud_provider:
      vsphere:
        topology_categories:
        - DataCenterName
        - ESXiHostsName

And:

spec:
  feature:
    system:
      values:
        storageClass:
          allowedTopologies:
            - matchLabelExpressions:
                - key: topology.csi.vmware.com/{category-name}
                  values: {tag}

Migration procedure:

Before start, make sure that you use KCP v1.27.1 and Agent v1.24.13-4 or above. For back compatible generator have expression for determine CPI enforce in Kublr: 1.25.16-1

For agent v1.24.13-4 and previous:

1. Upgrade Kublr Agent to v1.24.13-6 for PV in-tree provision migration complete.
2. Define topology-category and tags on UI.
3. Upgrade Kublr-System to v1.28.0 or above for enable CSI provisioner in default storage class.

Before 1.28.0

This is a zone in which the volume will be dynamically created. This zone will be included into the “allowedTopologies” section to the “StorageClass” and it will look like:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kublr-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: thin
  fstype: ext4
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - zone-a
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Default Storage Class Datastore

NOTE The behavior described in this section is deprecated since Kublr version 1.27.1, not supported since version 1.28.0.

This is the datastore where PersistentVolume will be dynamically created. This datastore will be included into default “StorageClass” and it will look like:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kublr-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/vsphere-volume
parameters:
  datastore: fast-datastore
  diskformat: thin
  fstype: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer