Kublr Release 1.21.0 (2021-06-11, NB! Critical known issue, use 1.21.1)

This release has known critical issues, use Kublr 1.21.2 or later instead

Due to docker image gcr.io/kubernetes-helm/tiller:v2.14.3 discontinued in the Google Image repository in August 2021, ( related issue: “Make Tiller Image Available on Docker Hub” ), Kublr may fail to complete cluster create and update.

The cluster hangs in “Creating” or “Updating” state indefinitely or for a very long time, or goes to “Error” state, in all cases with Tiller pod unhealthy due to Tiller image not available.

All versions of Kublr before 1.21.2 (including this one), and Kublr Agent versions earlier than the ones included in Kublr 1.21.2 are affected.

The issue and available solutions are described in the troubleshooting guide on Kublr support portal.

Migration to the latest Kublr Agents and Kublr Control Plane versions or at least Kublr 1.21.2 is recommended.

Additionally, Kublr release 1.21.0 contains a known issue with Azure resources update and removal.

Kublr Quick Start

sudo docker run --name kublr -d --restart=unless-stopped -p 9080:9080 kublr/kublr:1.21.0

Follow the full instructions in Quick start for Kublr Demo/Installer.

The Kublr Demo/Installer is a lightweight, dockerized, limited-functionality Kublr Platform which can be used to:

• Test setup and management of a standalone Kubernetes cluster
• Setup a full-featured Kublr Platform

The Kublr Demo/Installer stores all of the data about the created clusters inside the Docker container. If you delete the Docker container you will lose all data about the created clusters and the Kublr platforms. However, you will not lose the clusters and the platforms themselves.

We recommend using the Kublr Demo/Installer to verify if a Kubernetes cluster can be created in your environment and to experiment with it. To manage a real cluster and experience all features, you can create a full-featured Kublr Platform in a cloud or on-premise.

Overview

The Kublr 1.21.0 release brings Kuberntes 1.20, multiple container runtimes, and major vCloud Director redesign and improvements including migration to the recent versions of vCloud Director API and SDK, org and app networks of differnet types, edge gateways, NAT and firewall configuration. It also adds support for environments that only have access to public internet via proxy server, and provides a number of other improvements and fixes.

Important Changes

• New versions of Kubernetes
  • Kubernetes v1.20 support (v1.20.7 by default)
  • Kubernetes v1.21 technical preview
• Multiple container runtimes support: ContainerD (technical preview) and Docker
• Major vCloud Director redesign and improvements
  • Migrated to the recent version of vCloud Director API and SDK
  • org and app networks of differnet types
  • edge gateways, NAT and firewall configuration
• Support organization proxy settings
• Azure: Migrate to Storage Account version 2 and disable Storage Account public access by default

Improvements

• Upgrade minor versions of k8s
• Ubuntu 20.04 added to the list of options in UI for all clouds
• Grafana dashboards
  • Kublr Grafana Dashboard
• Include yaml2json and jq into the shell container
• Logging and Audit
  • Use non-OSS ELK images by default including X-Pack (Elastic stops supporting OSS images starting with 7.11)
  • Use separate index for Kublr API audit records
  • ELK secure settings can be configured via Kublr cluster spec
• Monitoring
  • create alert rules for elasticsearch
  • collect metrics and provide alerts for Keycloak
• Keycloak
  • Upgrade Keycloak to the latest version, from 10.0.0 to 12.0.4
• Kublr Agent
  • Check permissions on required files and folders
  • Reliable container image pull health checking and problem reporting improved
• Azure
  • Improve support for VMSS with custom images
  • Improve resource removal logic during update/delete cluster when deployed in an existing resource group
  • Suggest the list of available zones in UI
  • UI: do not set bootDiskSize by default
• AWS
  • Disable SSH port by default, tighten the default security group permissions
  • Enable overriding default security group rules individualy
  • Faster cluster deletion with the cluster controller deleting stack components independently
• Stability, Reliability and security
  • Use startupProbe for slowly starting containers
  • Azure: Update azure-sdk-for-go
  • Mongodb Client. Add connection and socket timeouts. Ensure awaiting of mongodb’s start\restart.
  • Upgrade to Go 1.15
  • Improve using of kubectl in SearchGuard init job
  • Kublr API should be restarted automatically if configmap/values.yaml changed
• Various UI Improvements
  • Visualize the status of packages
  • Increase and make configurable tolerance for token Issue, NotBefore and Expiry times
  • Hide functionality from the cluster view when a user does not have access to it
  • Set default k8s API to 6443
  • Add link to docs.kublr.com from RBAC configuration pages
  • Restore Intercom on Keycloak screen in Kublr Box
  • Support i18n settings on Keycloak screen
  • Improve user experience if user doesn’t have rights on any space
  • Lightweight redesign Cluster Statuses table
  • Set padding between Status and Last transition
  • Add empty option for Docker (Binary) Registry override fields
  • Add /ui/ to url of KCP in Kublr Box
  • Only display the cluster console link if it is available
  • AWS: UI should take into account ‘overrideImageID’ when displaying Host OS

Fixes

• Cluster update fails due to too short a timeout in certain situations
• Azure: Cluster does not run with CentOS 7 and 8 (see known issues below)
• AWS: Impossible to update cluster if create failed before cloud formation stack creation was completed
• Logging and Audit
  • Logstash floods the error logs with “logstash.licensechecker.licensereader” messages
• Monitoring
  • Fix missing prometheus annotations for the cluster autoscaler
    • “Kubernetes Workload” grafana dashboard - some charts do not filter by namespace
• UI
  • Fix security Dialogs Text Problems
  • Fix warnings and info color scheme
  • Fix generator validation does not send info messages without warnings\errors
  • If feature is off - the frame should be gray
  • Package status contains an unknown condition
  • Firefox: hovering over spinner cursor changes to text entry, spinners are not clickable
  • AWS: s3BucketName copied when cluster cloned
  • Azure: Operation system cannot be edited in a cloned cluster
  • Some issues editing a cloned cluster
  • Empty string in kublrAgentConfig.kublr.version.k8s
  • Error when changing a type of credentials while creating
• Agent
  • Autoscaling for Azure does not work with 1.17 Kublr Agent
  • Label-master pod requires root access or 755 rights to /var/lib/kubelet
  • Kubelet cannot login to docker.io if credential are specified
  • Fix potential race condition in secret store code
• Security
  • keycloak init container prints keycloak admin password in the output in clear text

AirGap Artifacts list

Additionally, you need to download the BASH scripts from https://repo.kublr.com

• https://repo.kublr.com/repository/arc/controlplane/1.21.0/kublr-load-helm-1.21.0.sh
• https://repo.kublr.com/repository/arc/controlplane/1.21.0/kublr-controlplane-load-images-1.21.0.sh

You also need to download Helm package archives and Docker images:

• https://dl.kublr.com/controlplane/1.21.0/kublr-helm-1.21.0.tar.gz
• https://dl.kublr.com/controlplane/1.21.0/kublr-controlplane-images-1.21.0.tar.gz

Supported Kubernetes versions

v1.21 (technical preview)

• https://repo.kublr.com/repository/arc/agent/1.21.1/kublr-agent-load-gobins-1.21.1-5.sh
• https://dl.kublr.com/agent/1.21.1/kublr-agent-1.21.1-5.tar.gz
• https://repo.kublr.com/repository/arc/agent/1.21.1/kublr-agent-load-images-1.21.1-5.sh
• https://dl.kublr.com/agent/1.21.1/kublr-agent-images-1.21.1-5.tar.gz

v1.20

• https://repo.kublr.com/repository/arc/agent/1.20.7/kublr-agent-load-gobins-1.20.7-13.sh

• https://dl.kublr.com/agent/1.20.7/kublr-agent-1.20.7-13.tar.gz

• https://repo.kublr.com/repository/arc/agent/1.20.7/kublr-agent-load-images-1.20.7-13.sh

  (optional if the control plane 1.21.0 images are imported already)

• https://dl.kublr.com/agent/1.20.7/kublr-agent-images-1.20.7-13.tar.gz

  (optional if the control plane 1.21.0 images are imported already)

v1.19

• https://repo.kublr.com/repository/arc/agent/1.19.11/kublr-agent-load-gobins-1.19.11-43.sh
• https://dl.kublr.com/agent/1.19.11/kublr-agent-1.19.11-43.tar.gz
• https://repo.kublr.com/repository/arc/agent/1.19.11/kublr-agent-load-images-1.19.11-43.sh
• https://dl.kublr.com/agent/1.19.11/kublr-agent-images-1.19.11-43.tar.gz

v1.18 (Deprecated in 1.22.0)

• https://repo.kublr.com/repository/arc/agent/1.18.19/kublr-agent-load-gobins-1.18.19-22.sh
• https://dl.kublr.com/agent/1.18.19/kublr-agent-1.18.19-22.tar.gz
• https://repo.kublr.com/repository/arc/agent/1.18.19/kublr-agent-load-images-1.18.19-22.sh
• https://dl.kublr.com/agent/1.18.19/kublr-agent-images-1.18.19-22.tar.gz

v1.17 (Deprecated in 1.21.0, End of support in 1.22.0)

• https://repo.kublr.com/repository/arc/agent/1.17.17/kublr-agent-load-gobins-1.17.17-21.sh
• https://dl.kublr.com/agent/1.17.17/kublr-agent-1.17.17-21.tar.gz
• https://repo.kublr.com/repository/arc/agent/1.17.17/kublr-agent-load-images-1.17.17-21.sh
• https://dl.kublr.com/agent/1.17.17/kublr-agent-images-1.17.17-21.tar.gz

Components versions

Kubernetes

Kublr Control Plane

Kublr Platform Features

Known issues and limitations

1. (Critical) During Azure cluster update, Kublr Cluster Controller tries to remove LoadBalancers created by Kubernets cloud controller.

   Please use Kublr 1.21.1 to avoid Azure cluster downtime.

2. Kublr operator crashes if Tiller is missing or upgraded in a managed cluster

3. Canal CNI (default) does not work on AWS images with nm-cloud-config. At the moment it only affects RHEL 8 image.

   As a workaround, remove or disable nm-cloud-config or use a different CNI plugin (e.g. CNI Calico).

4. SELinux is not supported for ContainerD CRI

5. Containerd CRI is only supported for Kublr Agents 1.19, 1.20, and 1.21

6. vCloud Director implementation does not support named disks for persistent volumes by default. vCloud Director CSI driver needs to be installed in the cluster.

7. Beginning November 2, 2020, progressive enforcement of rate limits for anonymous and authenticated Docker Hub usage came into effect. Learn more about the change from the article Understanding Docker Hub Rate Limiting. Kublr clusters use some images hosted on Docker Hub / docker.io (e.g. kubernetesui/dashboard:v2.0.4). As a result some cluster operations may fail due to Docker Hub rate limiting. You can avoid possible issues using one of the following solutions:

   1. If you have a paid Docker Hub account, create a docker.io secret in Kublr Control Plane and add this docker registry to the cluster specification using advanced section in Kublr cluster creation UI.
   2. Override docker.io registry with cr.kublr.com, all imagess needed for cluster installation are mirrored in this repo. Learn more about docker registry override in the Kublr documentation cluster specification reference.

8. (Critical) Due to docker image gcr.io/kubernetes-helm/tiller:v2.14.3 discontinued in the Google Image repository in August 2021, ( related issue: “Make Tiller Image Available on Docker Hub” ), Kublr may fail to complete cluster create and update.

   The cluster hangs in “Creating” or “Updating” state indefinitely or for a very long time, or goes to “Error” state, in all cases with Tiller pod unhealthy due to Tiller image not available.

   All versions of Kublr before 1.21.2 (including this one), and Kublr Agent versions earlier than the ones included in Kublr 1.21.2 are affected.

   The issue and available solutions are described in the troubleshooting guide on Kublr support portal.

   Migration to the latest Kublr Agents and Kublr Control Plane versions or at least Kublr 1.21.2 is recommended.