Logging operations

Create platform with centralized logging

To create a platform with centralized logging:

1. Initiate creation of Kublr platform. Example of how to do that can be found here.

2. In the DEPLOY FULL KUBLR PLATFORM dialog, click the FEATURES step.

3. Scroll to the Centralized Logging section.

   Centralized Logging is always enabled when creating Platform.

4. Fill out count of master/client/data Elasticsearch nodes. In general, one data node is enough for 2-3 clusters created using Platform, but it depends on the number of logs generated by each cluster.

   Note We highly recommend using Persistence enabled for collecting logs. Otherwise your custom templates and dashboards will be deleted after a restart of the Elasticsearch pod.

   

5. Complete creation of Kublr platform. Centralized Logging feature is installed on the Platform.

6. To open Kubernetes Dashboard, use the platform page, CLUSTER tab, Open Dashboard.

   

7. In Namespace, select “kublr”.

8. Open Pods page and check that all pods are running.

   Note It may take up to 20 minutes for all pods to fully start. RabbitMQ and port-fwd may restart periodically.

   

In the current implementation, logs will be collected for 2 days. If you want to collect logs for longer, follow the steps described in this article.

Create cluster with centralized logging

Centralized logging will be enabled in a managed cluster by default if centralized logging is enabled in the platform from which this cluster is created.

Create cluster with additional self-hosted Elasticsearch/Kibana logging

Centralized Logging is always enabled for your clusters. If you want additional logging for your cluster, follow the steps below:

1. Initiate creation of Kublr cluster. Example of how to do that can be found here.

2. In the ADD CLUSTER dialog, click the FEATURES step.

3. Select the Self-Hosted Logging checkbox.

4. Fill out count of master/client/data Elasticsearch nodes.

5. If necessary, select Persistence enabled, and then optionally type in Data node disk size.

   

6. Complete creation of Kublr cluster. Self-Hosted Logging feature is installed on the cluster.

7. While the cluster is created, go to the cluster page, the CLUSTER tab, scroll to the FEATURES section.

8. In the Logging field, click the link. The Kibana page is opened.

9. In Kibana, enter username/password from KubeConfig file.

   The file can be downloaded from Kublr, the cluster page, CLUSTER tab.

10. In Kibana, create an Index Pattern.

11. To check logs, on the left menu, click Discover.

    

Change parameters to collect Logs for more than 7 days

Cluster installation phase

By default, the curator is configured to delete indexes older than 7 days. You can change this setting by specifying the retentionPeriodDays parameter in custom specifications. Here is an example for a platform:

spec:
  features:
    logging:
      sinks:
        -
          centralLogging:
            retentionPeriodDays: 30

Here is an example for a cluster with SelfHosted Elasticsearch:

spec:
  features:
    logging:
      sinks:
        -
          selfHosted:
            retentionPeriodDays: 30

To use custom specification click Customize Cluster Specification button instead of Confirm and Install when creating a platform or cluster.

After the cluster is deployed

Go to Config Maps. Open kublr-logging-curator-config. Click the Edit button and change unit_count from 7 to required value.

Note: You must independently calculate the resources necessary for your task and your environment.

Enabling X-Pack in Elasticsearch/Logstash/Kibana

See also: Kublr support portal: Customize Elastic Stack

In Kublr 1.21 and later

In Kublr 1.21 and later X-Pack is enabled by default.

In Kublr 1.20.2 and later

By default, centralized logging is preconfigured to use ELK without X-Pack. In Kublr 1.21 Kublr will be preconfigured to use with X-Pack.

To configure enabling of X-Pack in ELK, use custom cluster/platform specification to switch X-Pack (add overwritten values under “logging” section of custom spec):

    logging:
      values:
        elasticsearch:           
           xpackEnabled: true

In Kublr 1.20.1 and earlier

By default, centralized logging is preconfigured to use ELK without X-Pack.

If you want X-Pack installed, use custom cluster/platform specification to switch to images that includes X-Pack and set xpackEnable option equals true (add overwritten values under “logging” section of custom spec):

    logging:
      values:
        elasticsearch:
          cluster:
            xpackIncluded: true
        logstash:
          image:
            name: logstash/logstash
        searchguard:
          enabled: true
          elasticsearch:
            image:
              name: kublr/elasticsearch-sg
          kibana:
            image:
              name: kublr/kibana-sg 

To use custom specifications, click the CUSTOMIZE SPECIFICATION button and set: