Kublr Release 1.23.0 (2022-10-15)

Kublr Quick Start

sudo docker run --name kublr -d --restart=unless-stopped -p 9080:9080 kublr/kublr:1.23.0

The Kublr Demo/Installer docker container can be run on ARM-based PC, such as MacBook M1.

Follow the full instructions in Quick start for Kublr Demo/Installer.

The Kublr Demo/Installer is a lightweight, dockerized, limited-functionality with base trial license Kublr Platform which can be used to:

  • Test setup and management of a standalone Kubernetes cluster
  • Setup a full-featured Kublr Platform
  • Trial license use

The Kublr Demo/Installer stores all of the data about the created clusters inside the Docker container. If you delete the Docker container you will lose all data about the created clusters and the Kublr platforms. However, you will not lose the clusters and the platforms themselves.

Overview

The Kublr 1.23.0 release brings Kubernetes 1.23 and 1.24, proxy servers supports, upgraded NGINX controler and terraform controller versions.

All Kublr components are checked for vulnerabilities with Aquasecurity trivy scaner.

It also includes new Kublr license management UI, and provides a number of other improvements and fixes.

Important Changes

  • New versions of Kubernetes

  • SELinux policy upgrade

    • (Kublr policy v0.0.5)[https://gitlab.eastbanctech.com/kublr/images/tree/master/selinux]
  • Proxy servers configuration supports

    • OS based proxy support for Kubernetes installation
    • Kublr operator proxy configuration support
  • Kublr license management

    • License management UI
    • Fully functional 30 day trial license is included in the Kublr Demo/Installer
  • Kuberntes Pod Security Policy feature is deprecated and will be removed from Kubernetes v1.25

    • (PSA) Enabled by default starting with Kublr agent v1.23
    • (PSP) Enabled in Kublr agent v1.23, deprecated and disabled in agent v1.24
    • Kublr enforce baseline PodSecurityConfiguration:
      apiVersion: pod-security.admission.config.k8s.io/v1beta1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "baseline"
        enforce-version: "latest"
        audit: "baseline"
        audit-version: "latest"
        warn: "baseline"
        warn-version: "latest"
      exemptions:
        usernames: []
        runtimeClasses: []
        namespaces: [kube-system,kublr]
      
  • Kublr Demo/Installer image is now a docker multiarch AMD64/ARM64 image

  • Kublr cert-updater is deprecated in Kublr v1.23 and is removed from Kublr agent starting with v1.22 and above

  • Kublr backup controller is deprecated in Kublr v1.23 and will be removed from Kublr 1.24.0

  • Kubernetes v1.19 support in Kublr v1.23 is deprecated and will be removed in Kublr v1.24.0

Improvements

  • Upgrade patch versions of supported Kubernetes versions

  • Keycloak v15.1.1 used

  • Kublr Operator:

    • Helm v3.7.2 is used by default
    • Proxy config support for helm/raw repositories
    • Kublr secrets can be used as helm package chart secrets
  • Kublr Terraform controller

    • terraform migrated to v1.1.7

      Currently supported versions: 0.12.30, 0.13.6, 0.14.11, 1.1.7

      Please read an important note above for vSphere based clusters upgrade procedure!

  • Kublr Monitoring:

    • Prometheus v2.37.1 LTS
  • Kublr Logging:

    • RabbitMQ v9.5.3

    • Fluentbit enabled by default (instead of Fluentd)

    • RabbitMQ space limit autofix

    • The following settings are removed from the default template:

      "auto_expand_replicas": "0-1",
      "codec": "best_compression"
      
  • Kublr feature ingress

    • Kubernetes NGINX controller automaticaly migrates to v1.3.0 (helm chart v4.2.3)

      This may affect the applications deployed to the managed clusters; please refer to NGINX Ingress Controller documentation to prepare for the upgrade.

  • Kublr Agent

    • Proxy server configuration support
    • Improved containerd supports
  • vSphere:

    • Master nodes can be located on different ESXi hosts
    • Cloudinit based templates support deployment on StorageDRS cluster
  • AWS:

    • CSI driver v1.5.1 with enabled CSIMigrationAWS feature gates starting with Kublr agent v1.23
    • Use nextToken pagination for all api requests
  • Azure:

    • CSI driver v1.9.0 with enabled CSIMigrationAzureDisk feature gates starting with Kublr agent v1.23
    • Master nodes are included in the services load balancer pools by default
  • Stability, Reliability and security

    • Kublr k8s API proxy use HAProxy v2.6.4; HAProxy is run as a non-root user
    • PSP supported in KCP database components
    • Kublr components migrated to alpine v3.16.2 base image
  • Various UI Improvements

    • PSP configuration selector
    • Proxy server configuration credentials
    • Fluentbit/Fluentd selector

Fixes

  • Increased memory limits for generator and API
  • Kublr cluster-controller stops upgrading k8s to 1.22 if Ingress contoller version is lower than 1.21.0
  • Feature controller: error parsing token: crypto/rsa: verification error
  • Fix a downloading file through a slow connection issue
  • All components behind OAuth proxy sidecar should listen on 127.0.0.1
  • Terraform operations can’t be interrupted
  • Wrong KublrKubernetesReady expression if a node has completed pods
  • Decreasing autoscale group doesn’t work in case if min nodes < the number of AZ
  • Azure
    • Failed to create a Kublr cluster Storage Account with Private endpoints
    • Failed to migrate Storage Account from public endpoint to private endpoint
  • vSphere
    • CloudInit ISO file must be reconciled if none exists
  • Logging and Audit
    • Allow user to override kublr_rollover
  • Monitoring
    • Wrong ingress rule for prometheus in the managed clusters

AirGap Artifacts list

Additionally, you need to download the BASH scripts from https://repo.kublr.com

You also need to download Helm package archives and Docker images:

Supported Kubernetes versions

v1.24

v1.23

v1.22

v1.21

v1.20 (Deprecated in 1.24.0, End of support in 1.25.0)

v1.19 (Deprecated in 1.23.0, End of support in 1.24.0)

Components versions

Kubernetes

ComponentVersionKublr AgentNotes
Kubernetes1.241.24.6-7default v1.24.6
1.231.23.10-7
1.221.22.13-7
1.211.21.14-7
1.201.20.14-7Deprecated in 1.24.0
1.191.19.16-7End of support in 1.24.0

Kublr Control Plane

ComponentVersion
Kublr Operator1.23.0
Kublr Control Plane1.23.0

Kublr Platform Features

ComponentVersion
Kubernetes
Dashboardv2.4.0
Kublr System1.23.0
LocalPath Provisioner (helm chart version)0.0.22-11
Ingress1.23.0
nginx ingress controller (helm chart version)4.2.3
cert-manager (helm chart version)1.5.3
Centralized Logging**1.23.0 *
ElasticSearch7.10.2
Kibana7.10.2
SearchGuard52.3.0
SearchGuard Kibana plugin51.0.0
SearchGuard Admin7.10.2-52.3.0
RabbitMQ3.9.5
Curator5.8.1
Logstash7.10.2
Fluentd3.3.0
Fluentbit1.8.10
Centralized Monitoring1.23.0
Prometheus2.37.1
Kube State Metrics (helm chart version)3.4.2
AlertManager0.22.0
Grafana7.5.10
Victoria Metrics
Cluster0.9.13
Agent0.7.35
Alert0.4.15

Known issues and limitations

  1. ContainerD CRI does not support private registry authentication; use the following cluster spec configuration override as a workaround if needed:

    kublrAgentConfig:
      kublr:
       containerd:
         config:
           plugins:
             "io.containerd.grpc.v1.cri":
               registry:
                 configs:
                   "{{ DOCKER_REPO_URL }}":
                     auth:
                       username: {{ REPO_USER }}
                       password: {{ REPO_PASSWORD }}
                     tls:
                       insecure_skip_verify: true
    
  2. After adding a new managed cluster with centralized logging enabled, a default index pattern is not created automaticaly in Kibana. You will need to add the index pattern manually via Kibana cluster settings.

  3. For Kublr kubernetes clusters deployed on vSphere it is necessary to unmount CD/DVD drive 1 from each virtual machines manually via vCenter console.

  4. vSphere clusters cannot be deleted automaticaly, it is necessary to remove VMs manually and unregister the cluster after that.