Deploy Kubernetes in Air-Gapped Environments with Kublr

First things first

To deploy Kubernetes in air-gapped environment (environment with no connectivity) using Kublr Demo/Installer, please use your Kublr license number or request the Kublr evaluation license via email at contact@kublr.com or the schedule demo form.

Additionally, you need to download the BASH scripts from https://repo.kublr.com

You also need to download Helm package archives and Kublr Control Plane Docker images:

Also you need to download BASH scripts for supported Kubernetes versions as described below.

BASH scripts for supported Kubernetes versions

You need to download required Kublr Agent components files based on Kubernetes version from current release notes page: Kublr release v1.29.0

Important notice!

All the provided scripts are checked for use with the Sonatype OSS Nexus. If you are using a different repository, you might need to modify the given scripts.

In this example we will use the following local repository installation based on Sonatype OSS Nexus as described Nexus Repository Installation

System Requirements for Cluster Nodes

  1. It should have x86 64-bit hardware
  2. Hardware recommendation can be found here Hardware recommendation
  3. Minimal supported OS on nodes should be: RedHat Enterprise Linux 7.5+ or Ubuntu 16.04 LTS
  4. It should have root access to each node
  5. Existing RAW repository for uploading Helm and Kublr agent archives and Docker registry should be available from each node
  6. With nodes connected to your network, Kublr Demo/Installer configured IP should be accessible from these nodes (ping)
  7. From your nodes, firewall rules should allow traffic to Kublr Demo/Installer on port 9080
  8. It should have installed curl, containerd and docker packages.
  9. Kublr Demo/Installer should be correctly configured to be accessible in your local network. Note: It is critical that you provide the correct IP address during the virtual machine startup. If you skip this step, please re-run provisioning and configure your firewall rules to deliver traffic to your computer.

Persistence Data Storage for Kublr KCP on Hosts

Component NameNode TypeDefault Storage PathOwner UIDMinimum Disk Size
ETCDmaster/mnt/master-pd04G
Elasticsearch data nodenode/var/lib/kublr/elasticserach/data1000128G
Elasticsearch master nodenode/var/lib/kublr/elasticserach/master10004G
Grafananode/var/lib/kublr/grafana01G
MongoDBnode/var/lib/kublr/mongodb10018G
MySQL DBnode/var/lib/kublr/mysql9998G
Prometheusnode/var/lib/kublr/prometheus025G
RabbitMQnode/var/lib/kublr/rabbitmq9993G

Repository Requirements

Please refer to Nexus Repository Installation and then to Nexus Repository Configuration for example repository setup.

  1. It should have RAW repository (e.g. Sonatype OSS Nexus) for store go binary and Helm packages. At least 15Gb free space is required.
  2. It should have Docker repository (e.g. Sonatype OSS Nexus or Docker Registry) for Docker image management. At least 15Gb free space is required.

Repository Preparation

In this example, 192.168.3.8 is the IP address of the local RAW and docker repositories. You will need to change this to your repository IP or DNS name. You refer to Nexus Repository Installation and then to Nexus Repository Configuration for example repository setup.

Use the downloaded files on external media or download SHELL scripts and run them as is. All necessary archives will be downloaded automatically. Internet access is required.

  1. Upload archives with Kublr agent and Helm packages

    $ bash kublr-agent-load-gobins-1.26.4-2.sh http://192.168.8.36:8081/repository/raw/
    Upload kublr-agent-1.26.4-2.tar.gz to local repo:
    ######################################################################## 100.0%
    
    $ bash kublr-load-helm-1.29.0.sh http://192.168.8.36:8081/repository/raw/
    Processing kublr-helm-1.29.0.tar.gz:
    ######################################################################## 100.0%
    cleaning...
    
  2. Login to your Docker repository

    $ docker login --username admin --password admin123 192.168.3.8:5000
    
  3. Push Kublr Control Plane Docker images into your Docker registry

    $ bash kublr-controlplane-load-images-1.29.0.sh 192.168.3.8:5000
    
  4. Push Kublr Agent Docker images into your Docker registry

    $ bash kublr-agent-load-images-1.26.4-2.sh 192.168.3.8:5000
    

Running the Kublr Demo/Installer in Air-Gap Mode

In order to use Kublr you need to run Kublr Demo/Installer with access to Air-Gapped networks. After preparations described before no internet access is needed.

Specify the addresses of repositories to store artifacts. Please also determine the address of the Docker repository when you run Kublr Demo/Installer.

export HELM_REPOSITORY=https://192.168.3.8:8081/repository/raw
export KUBLR_AGENT_REPOSITORY=https://192.168.3.8:8081/repository/raw
export DOCKER_REGISTRY=192.168.3.8:5000
export KUBLR_VERSION=1.29.0
export KUBLR_LICENSE=<your-kublr-license-number>

docker run -p 9080:9080 -d --restart=unless-stopped --name kublr \
        -e SKIP_TLS_VERIFY=true \
        -e HELM_REPOSITORY=${HELM_REPOSITORY} \
        -e KUBLR_AGENT_REPOSITORY=${KUBLR_AGENT_REPOSITORY} \
        -e KUBLR_LICENSE=${KUBLR_LICENSE} \
        ${DOCKER_REGISTRY}/kublr/kublr:${KUBLR_VERSION}

After that Kublr Bootstrapper (Demo/Installer) should be accessible using URL “https://{Kublr-Demo-Installer-IP}:9080/” - replace “{Kublr-Demo-Installer-IP}” with the actual IP address or hostname of your Kublr Bootstrapper (Demo/Installer).

Determine Your Own IP Address

Creating a cluster in your machine network is simple. Use your machine’s IP address for this network. If you don’t know how to get the IP address, contact your system administrator or read your OS manual.

To install Kublr clusters in a different network on complex network topologies, provide the IP address of your machine on that network.

Creating an Air-Gapped cluster in Kublr

  1. Open the Kublr Bootstrapper (Demo/Installer) UI by accessing the following URL in your web browser: “https://{Kublr-Demo-Installer-IP:9080/}”. Replace “{Kublr-Demo-Installer-IP}” with the actual IP address or hostname of your Kublr Bootstrapper (Demo/Installer) that has been determined earlier.

  2. In section “Credentials”, create the following credentials:

    • Create Docker Registry Credentials, add username, password if needed and add CA cert file, or set insecure. Docker Registry
    • Likewise, create Binary Registry Credentials - make sure you have noted created credentials name. Binary Registry
    • Configure Proxy if needed. Proxy
  3. Click create cluster or platform.

  4. In KCP create mode, expand Advanced options and override docker repository. Besides default, you can override:

    • docker.io
    • gcr.io
    • k8s.gcr.io
    • quay.io
    • elastic.co
  5. Override binary repository. Besided default, you can override:

    • storage.googleapis.com
    • github.com Docker and binary override
  6. In last step, click “Customize Specification” and add the following configuration:

   features:
      controlPlane:
         helmRepositorySecretRef: BINARY_REGISTRY_NAME

Where BINARY_REGISTRY_NAME should be replaced with the name of configured binary registry in section “Credentials” of Kublr Bootstrapper (Demo/Installer) UI

For more detail see On-Premises Installation

More info: Docker images customization