Log rollover and data streams

Log rollover and deletion

In Kibana, the index lifecycle policies define when the rollover of a new index should occur and when the old indices should be deleted.

To view the policies:

1. In Kibana, on the left menu, under Management, click Stack management.

2. Click Index Lifecycle Policies.

   

Data streams

Kublr default policy for the index rollover creates a new index every day for every cluster. This means the significant increase in the number of the tiny indexes through which the Elasticsearch needs to search for the data when working with the logs. If there is a need to store data for some long period of time, the huge number of stored indexes may lead to slowing down the work with logs.

To deal with this problem, Elasticsearch has a built-in mechanism of data streams:

A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests.

Kublr supports the data streams feature. By default it is disabled but you can enable it at any moment if necessary.

To enable the data streams feature:

1. Via Kublr, access your cluster specification.

2. IMPORTANT: To avoid problems during migration process, temporarily disable the logstash.

   spec:
      features:
         logging:
            values:
               logstash:
                  replicas: 0
   

3. Set useDataStreams: true

4. If necessary, set rollover maxAge and/or maxSize. If not set, the default values “7d” and “20Gb” will be used.

   spec:
   features:
      logging:
         values:
            elasticsearch:
               useDataStreams: true
               rollover:
                  maxAge: 1d
                  maxSize: 1Mb
            fluentbit:
               enabled: true
            fluentd:
               enabled: false
   

5. Save the new configuration.

6. Wait until upgrade is finished

7. In Kibana > Stack Management > Index Management > Index Templates, check that the kublr-logs template has the Use Datastream option set to “ON”.

8. Scale the logstash back to designed value.

9. Re-enable the logstash.

Data streams support in Kublr includes the kublr-rollover ILM policy that you can access and configure under your needs as alternative to specification editing described in the steps above.

Note Starting from release 1.23, it is planned to rename the kublr-rollover ILM policy > kublr-policy.