Kublr Release 1.26.0 (2023-06-28)

Kublr Quick Start

To quickly get started with Kublr, run the following command in your terminal:

sudo docker run --name kublr -d --restart=unless-stopped -p 9080:9080 kublr/kublr:1.26.0

The Kublr Demo/Installer docker container can be run on ARM-based PC, such as MacBook M1.

Follow the full instructions in Quick start for Kublr Demo/Installer.

The Kublr Demo/Installer is a lightweight, dockerized, limited-functionality Kublr Platform with a base trial license. It can be used for the following purposes:

  • Test setup and management of a standalone Kubernetes cluster
  • Setting up a full-featured Kublr Platform
  • Trial license use

The Kublr Demo/Installer stores all data about created clusters inside the Docker container. If you delete the container, you will lose all data about the created clusters and the Kublr platforms. However, you will not lose the clusters and platforms themselves.

Overview

The Kublr 1.26.0 release introduces several new features and improvements, including:

  • Support for Kubernetes 1.26
  • Improved SELinux policy
  • Production-ready VictoriaMetrics cluster
  • Production-ready OpenSearch integration
  • High availability mode for Control Plane
  • Updates for MongoDB in Kublr Control Plane

All Kublr components are checked for vulnerabilities using Aquasecurity trivy scaner. In addition to these major features, the release also includes various other improvements and fixes.

Supported Kubernetes Versions

VersionKublr AgentNotes
1.261.26.4-2Default version: v1.26.4
1.251.25.9-12
1.241.24.13-2
1.231.23.17-4Deprecated in 1.27.0
1.221.22.17-10End of support in 1.27.0

Important Changes

  • New versions of Kubernetes:

  • Deprecations:

    • Kubernetes v1.21 (v1.21.14/agent 1.20.14-22) is End of Support
    • Kubernetes v1.22 (v1.22.17 by default) is deprecated and will be removed in Kublr v1.27.0
    • Kublr cert-updater is End of Support and should not be used in Kublr agents v1.22 and above
    • Ubuntu 18.04 / SUSE SLES 12 is End of Support and should be removed from Kublr UI
  • Kubernetes node-role enhancement

    Kublr now applies a “node-role” label to its control plane Nodes. The label key has been renamed from node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane. Kublr also uses the same “node-role” key for a taint applied to control plane Nodes, which has also been renamed to “node-role.kubernetes.io/control-plane”. For more information, refer to the Kubernetes Enhancement Proposal.

    • Introduced the “node-role.kubernetes.io/control-plane” label alongside the “node-role.kubernetes.io/master” label for the “Control Plane” nodes
    • Introduced the “node-role.kubernetes.io/control-plane:NoSchedule” toleration in Kublr Application Deployments
  • Kublr CRDs migrate to free form

  • Seccomp annotation is enabled by default

If you need to disable the default seccomp profile, use the following cluster spec:

spec:
  master:
    kublrAgentConfig:
      kublr:
        security:
          seccompdefault: false
  • SELinux: Added the right context for Kublr secrets and persistent data

  • Kublr Control Plane MongoDB migrates to v5.0.17

  • VictoriaMetrics cluster v0.9.62 and Agent v0.8.37 are now included in Kublr Centralized Monitoring and can be enabled via the UI

Improvements

  • Kublr Agents:

    • Upgraded patch versions of supported Kubernetes versions.
    • Updated Cloud CSI/CPI drivers.
    • Containers running with ContainerD CRI are stopped during containerd restart.
    • Removed KubeDNS addons.
  • Kublr Control Plane:

    • Default usage of Helm 3.12.0 in Kublr operator and feature controller.
    • Added PodDisruptionBudget (PDB) for the Database deployment in Control Plane HA mode.
    • UI improvements:
      • Added switches for ELK/OpenSearch and Prometheus/VictoriaMetric.
      • Added container runtime information to node information pages.
      • Added AWS EC2 spot instance support.
  • Azure:

    • Critical fix: Kublr now considers the next page marker when listing files from Azure secret store.
    • Supported CSI controller CRI migration.
  • vSphere:

    • Forced external CSI driver usage in Kubernetes 1.25 and above.
    • Default usage of CSI external provisioner for Kubernetes 1.25 and above.
    • Migrated CPI config to YAML format.
    • Fixed cluster deletion issue.
  • Centralized Log Collection:

    • Fixed LogMover to stop log collection correctly when the managed cluster is in the wrong state.
    • Enabled Curator by default for SelfHosted deployment.
    • Scheduled ELK pods only on AMD64 nodes.
    • Added SelfHosted OpenSearch deployment.
    • Prepared ELK v1.17.9 images for customer use.
  • Stability, Reliability, and Security:

    • Upgraded default Terraform version to v1.4.6:
      • Used GoVC v0.30.4.
      • Used vSphere plugin v2.3.1.
      • Used VCD plugin v3.1.0.
    • Migrated FluentD ARM compatibility to v3.3.1.

Fixes

  • Memory leak in Kublr cluster-controller and feature-controller.
  • Fixed an issue with Terraform state not refreshing missing folders in vSphere.
  • UI/UX improvements in license management.
  • Fixed an issue with sorting instances by status not working properly in the UI.
  • Removed validation for vSphere default storage in the UI.
  • Display a warning in the UI if cni-calico is selected for Azure clusters.
  • vSphere cluster now shows the selected network on the edit page in the UI.

AirGap Artifacts List

To use Kublr in an airgap environment, you will need to download the following BASH scripts from the repository at https://repo.kublr.com:

You will also need to download the following Helm package archive and Docker images lists:

Supported Kubernetes Versions

v1.26

v1.25

v1.24

v1.23 (Deprecated in 1.27.0)

v1.22 (Deprecated in 1.26.0, End of support in 1.27.0)

Components versions

Kublr Control Plane

ComponentVersion
Kublr Operator1.26.0
Kublr Control Plane1.26.0

Kublr Platform Features

ComponentVersion
Kubernetes
Dashboardv2.7.0
Kublr System1.26.0
LocalPath Provisioner (helm chart version)0.0.24-15
Ingress1.26.0
nginx ingress controller (helm chart version)4.7.0
cert-manager (helm chart version)1.10.2
Centralized Logging1.26.0
ElasticSearch7.10.2
SearchGuard53.6.0
Kibana7.10.2
SearchGuard Kibana plugin53.0.0
SearchGuard Admin7.10.2-53.6.0
OpenSearch (helm chart version)2.6.2
OpenSearch Dashboards(helm chart version)
RabbitMQ3.9.5
Curator5.8.1
Logstash7.10.2
Fluentd3.3.2
Fluentbit1.9.10
Centralized Monitoring1.26.0
Prometheus2.37.8 LTS
Kube State Metrics (helm chart version)5.6.4
AlertManager0.25.0
Grafana8.5.22
Victoria Metrics
Cluster0.9.62
Agent0.8.37
Alert0.6.0

Known Issues and Limitations

  1. For Kublr Kubernetes clusters deployed on vSphere, you may encounter an update error that requires manual unmounting of CD/DVD drive 1 from each virtual machine using the vCenter console.

  2. Elasticsearch is only supported on the AMD64 architecture. If you are using an ARM64 cluster, you can use tag 7.17.9 for the cluster nodes. Refer to the following link for more information: Use Elasticsearch v7.16.3 or Above.

  3. vSphere CSI limitation: Please refer to the VMWare documentation for information on vSphere functionality supported by the vSphere Container Storage Plug-in.

  4. CRI change limitation: When deploying an Ingress controller in hostPort mode (typically for vSphere or BareMetal installations), DNAT rules may not be cleaned properly. This is an open issue in the Felix project. If your Ingress NGINX controller connection times out, please reboot the worker nodes.

  5. SUSE SLES 12 sp5 can’t be run with ContainerD CRI (Docker is end of support in Kuberntes v1.24.0). Kublr can’t find the correct way for automatically installing containerd-ctr utilities, but fully supports Kubernetes installation with preinstalled ContainerD CRI and containerd-ctr utilities.

  6. Google Cloud Platform SUSE SLES 15 sp4 can’t use the Google Cloud Engine CSI PD driver. The root cause is the relocation of /lib/udev to /var/lib/udev on the root file system. Please use custom CSI drivers installation and disable Kublr CSI support via the cluster spec:

spec:
  kublrAgentConfig:
    kublr:
      features:
        csi_drivers:
          auto_behavior: manual