Back to Kublr Support
Image caching with Spegel

Image caching with Spegel

Overview

TODO:

  • agent versions

Starting with Kublr Agent versions 1.27, 1.28, 1.29, 1.30, 1.31, 1.32 Kublr integrates with Spegel (github). Cluster is created with Spegel integration enabled by default.

Spegel starts lightweight registry mirrors on every node of the cluster that can serve images already present on the node to other nodes.

Important notes:

  • Spegel registry mirroring is not a replacement for local registry. Original registries must still be present for cluster to restart after full node replacememt, or when Spegel mirrors are not available for some reason.

  • Spegel only works with Containerd runtime; CRI-O and other runtimes are not supported and Spegel will be disabled by default if they are used.

Registry mirroring architecture with Spegel

Kublr and Spegel use Containerd hosts configuration described in Containerd hosts docs.

Kublr agent will create /etc/containerd/certs.d/_default/hosts.toml file pointing Containerd at http://<current-node>:30020 and http://<master-nodes>:30020 as registry mirrors.

Spegel is installed in the cluster using standard Spegel helm chart and runs as a DaemonSet on each node of the cluster. Two ports are open on each node that can be used to access Spegel registry mirrors:

  • 30020 is a nodePort created by the pod running on the node
  • 30021 is a nodePort created by Spegel Service

After Spegel pod starts on a node, it replaces Kublr-generated /etc/containerd/certs.d/_default/hosts.toml file with its own, generated based on the helm chart configuration. By default it will contain http://<current-node>:30020 and http://<current-node>:30021 as registry mirrors.

This approach allows worker nodes bootstraping using Spegel even before the node is fully initialized and Spegel instance is started on it, therefore benefitting from mirroring images for foundational components like CNI drivers etc.

Configuring Spegel integration

Disable Spegel

Spegel can be disabled by setting cluster.addon.spegel.enabled property in Kublr agent config to false.

When this property is set to false, Spegel helm chart will not be deployed (or will be uninstalled) if it was deployed already, and Kublr agent will generate empty /etc/containerd/certs.d/_default/hosts.toml file.

Note that this property must be set on all nodes of the cluster to affect Kublr-generated hosts.toml on all nodes.

Example cluster configuration:

spec:
  kublrAgentConfig:
    cluster:
      addon:
        spegel:
          enabled: false

Customize Spegel chart configuration

Spegel chart configuration can be customized via cluster.addon.spegel.helm_values section in Kublr agent config:

spec:
  kublrAgentConfig:
    cluster:
      addon:
        spegel:
          helm_values:
            registryFilters:
              - '.*:latest$'
              - '^docker\\.io/'
              - '^ghcr\\.io/'

Helm chart values section can be specified as structured map (as in the previous example) or as a string value containing yaml.

It is recommended to use structured map in most cases as it provides additional level of validation. The only situation when string value with yaml should be used is when it is necessary to unset some values in helm chart using null value. Structured map will be cleaned out of null values by Kublr API, so the only way to pass null to helm is to use string with yaml, e.g. as follows:

spec:
  kublrAgentConfig:
    cluster:
      addon:
        spegel:
          helm_values: |
            resources:
              requests:
                memory: null
              limits:
                memory: null

Disable or override Kublr-generated hosts.toml

Kublr agent uses built-in go template to generate /etc/containerd/certs.d/_default/hosts.toml.

Any built-in templates can be overridden in Kublr agent config in section extensions as described in Kublr cluster specification reference.

Specific template used to generate Cpntainerd catch-all hosts.toml is templates/root_dir/etc/containerd/certs.d/_default/hosts.toml.

To disable the template generation specify empty content as follows:

spec:
  kublrAgentConfig:
    extensions:
      # this key name can be anything, multiple keys will be applied in alphanumerical order
      containerd-default-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/_default/hosts.toml'
        content: ''

To make sure that the file is generated with no mirrors, you can provide non-empty content with comments only:

spec:
  kublrAgentConfig:
    extensions:
      containerd-default-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/_default/hosts.toml'
        content: '# no mirrors'

Provide specific mirror:

spec:
  kublrAgentConfig:
    extensions:
      containerd-default-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/_default/hosts.toml'
        content: |
          [host.'https://my-mirror.internal']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'

Go template functions can be used to generate files content:

spec:
  kublrAgentConfig:
    extensions:
      containerd-default-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/_default_/hosts.toml'
        content: |
          [host.'http://{{.config.node_address_ip}}:30020']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'

Define mirrors/hosts for specific registries

It is possible to specify mirrors for specific registries as documented in Containerd hosts docs and fully customize registry mirror configuration.

For this either Spegel should be fully disabled, or Spegel Containerd mirror configuration function should be turned off so that Spegel does not overwrite customizations:

Spegel Containerd mirror configuration function can be turned off via spegel.containerdMirrorAdd property in the chart values file, for example:

spec:
  kublrAgentConfig:
    cluster:
      addon:
        spegel:
          helm_values:
            spegel:
              containerdMirrorAdd: false
    extensions:
      containerd-default-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/_default/hosts.toml'
        content: |
          [host.'http://{{.config.node_address_ip}}:30020']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'

          [host.'http://{{.config.node_address_ip}}:30021']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'
      containerd-great-registry-hosts-override:
        path: 'templates/root_dir/etc/containerd/certs.d/great-registry.com/hosts.toml'
        content: |
          [host.'https://my-mirror.internal']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'

          [host.'http://{{.config.node_address_ip}}:30020']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'

          [host.'http://{{.config.node_address_ip}}:30021']
          capabilities = ['pull', 'resolve']
          dial_timeout = '200ms'
Questions? Suggestions? Need help? Contact us.