Using VictoriaMetrics instead of Prometheus

Overview

Kublr’s monitoring capability is built on top of Prometheus and Grafana. In that scheme, you can replace Prometheus with the cluster version of VictoriaMetrics.

IMPORTANT NOTICE: Using VictoriaMetrics instead of Prometheus is a technical preview feature and is not recommended in production environments.

VictoriaMetrics can be implemented only within the Kublr platform.

When replacing Prometheus with VictoriaMetrics may be beneficial

Downstate of one of the availability zones for the VictoriaMetrics does not stop the monitoring function.

Therefore, for the large clusters, when Prometheus performs monitoring and it is restarting, this restart may take a long time, and monitoring will not be performed within this time. But stopped monitoring will never be the case when working with VictoriaMetrics.

NOTE: Prometheus dealing with millions of unique time series (aka high cardinality) or high metrics churn rate has higher chances to crash/restart and much longer restart time.

VictoriaMetrics uses up to 7x less RAM than Prometheus when dealing with high cardinality or metrics churn rate.

Why using VictoriaMetrics instead of Prometheus is painless

• API of VictoriaMetrics is fully compatible with Prometheus.
• VictoriaMetrics implements the query language MetricsQL which is an advanced version of Prometheus’s PromQL. Thus any query written in PromQL for Prometheus will work fine in VictoriaMetrics.
• To configure VictoriaMetrics, you can use the Prometheus configuration file without any changes.
• VictoriaMetrics works perfectly with the Kublr native Alert Manager.

Conclusion:

• Any application working with Prometheus (like Grafana) will work perfectly well with VictoriaMetrics without reconfiguration.

How to replace Prometheus with VictoriaMetrics

To enable VictoriaMetrics for your Kublr platform, in the specification, set:

spec:
  features:
    monitoring:
      ...
      values:
        victoriametrics:
          enabled: true
        vmcluster:
          clusterDomainSuffix: cluster.local
          vmstorage:
            persistentVolume:
              size: 128Gi

Where clusterDomainSuffix should correlate with:

  network:
    dnsDomain: cluster.local

NOTE: Clusters connected to the control plane of your platform should have the same clusterDomainSuffix.

This installs VictoriaMetrics cluster Helm chart for your Kublr platform.

VictoriaMetrics will be automatically set as a default data source for Grafana.

NOTE: While working with VictoriaMetrics is a technical preview feature, you should not disable Prometheus when using VictoriaMetrics.

VictoriaMetrics cluster version components

With Kublr, a cluster version of VictoriaMetrics is deployed. VictoriaMetrics is implemented as a sub-chart of the monitoring chart. It includes the following components:

• kublr-monitoring-vmagent - main component of the application responsible for collecting metrics; receives requests from Grafana. By default, 2 agents are implemented to provide fault tolerance.
• kublr-monitoring-vmalert-server - analog of Kublr Alert Manager. In most cases, it is recommended to disable this component of VM and use the native Alert Manager. However it is good to consider that like all other components of cluster VM version, it is horizontally scalable, and the downstate of one of the VM’s nodes does not stop alerting service.
• kublr-monitoring-vmcluster-vminsert, kublr-monitoring-vmcluster-vminsert - additional components to which the vmagent transfers the requests; they are processing different types of requests and allow decreasing the load of the database.
• kublr-monitoring-vmcluster-vmstorage-1..2 - components for storing metrics, each can be scaled, and each supports RESTful API.

Pods with the listed components after installing VictoriaMetrics cluster:

For more information on the components configuration, refer to the values.yaml file of the corresponding VictoriaMetrics Helm charts.

Currently, VictoriaMetrics is implemented with Kublr without its own UI.