To setup RHACM Observability

To install the RHACM operator and enable its observability capabilities, perform the following steps:

1. Log in to your OpenShift instance.
2. Under Ecosystem at Software Catalog
3. Pick the a namespace
4. In the search look for Advanced Cluster Management for Kubernetes
5. Select install if not already installed
6. Once installed, pull it up under Ecosystem at Installed Operators
7. Under MultiClusterHub tab, click Create MultiClusterHub
8. Give it an appropriate name and click Create
9. Go to Storage at StorageClasses
10. Make sure a StorageClass is set to default for deploying the infrastructure. For example, if using ODF, make sure the "ocs-storagecluster-ceph-rdb-virtualization" class is set to the default. If it is not, click the 3 dots on the StorageClass desired and choose "Set as default"
11. Follow the steps described in https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.14/html-single/observability/index#enabling-observability to enable the Observability modules of RHACM

If configuring Thanos with IBM Cloud® Object Storage, the thanos-object-storage.yaml file will resemble the following:

apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: <bucket-name>
      endpoint: s3.us-east.cloud-object-storage.appdomain.cloud
      insecure: true
      access_key: XXXX
      secret_key: XXXX

Service credentials should be created HMAC keys in the IBM Cloud Object Storage instance. The values for access_key and secret_key will be the HMAC ones provided.

Alerting

Overview

Default System alerts are managed by OpenShift's built-in monitoring stack and cover platform-level concerns:
Scope: Single cluster infrastructure monitoring
Namespace: openshift-monitoring on each managed cluster
Components:

• Cluster Monitoring Operator
• Prometheus Operator
• Prometheus (for cluster metrics)
• AlertManager (for cluster alerts)

If default system alerts cannot cover your business scenario, you can setup custom alerts, which can be managed by RHACM's observability stack and support multi-cluster scenarios:
Scope: Multi-cluster application and custom resource monitoring
Namespace: open-cluster-management-observability on the hub cluster
Components:

• Thanos Query (global metrics view)
• Thanos Ruler (centralized alert evaluation)
• AlertManager (multi-cluster alerting)

Creating System Alerts

For system alerts (handled by the openshift-monitoring namespace), complete the following steps to create alerting. Alerting supports integration with PagerDuty, Slack, and Email.

1. Log in to your OpenShift instance.
2. Under Administration at Cluster Settings88 at Alertmanager
3. Either configure one of the existing Receivers listed at the bottom table or click Create Receiever
4. Give an appropriate name
5. Under Receiver type select one from the dropdown
   1. If you select PagerDuty, you need a routing key and URL.
   2. If you select Email, you need an SMTP server.
   3. If you select Slack, you need the webhook URL and channel name (can be a username for direct messages, without the @ symbol).
6. For routing labels, select based on which alerts you want to send a notification for.
   1. For example, severity = critical sends all critical alerts, and severity = warning sends all warning alerts.
   2. You can leave this field blank to have all alerts send out a notificiation

You can click "Alerting rules" tab, which will show you the existing default alerting rules to cover the system alerts. Take alerting rules for ceph storage cluster as an example, as the following shows, which means there should be alert triggered if the storage usage is higher than 75%.

Creating Custom Alerts

For custom alerting (handled by the open-cluster-management-observability namespace):
When default alerting rules do not cover a required scenario, you must create custom alerting rules. The following example creates a Slack alert when a virtual machine in the default namespace is not in a running state. This scenario is not covered by default alerting rules and requires a custom rule in RHACM. Complete the following steps:

1. Verify whether the Prometheus metric kubevirt_vm_info exists. Log in to the observability-thanos-query pod in the open-cluster-management-observability namespace, and run the following curl command to confirm the metric exists: curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=kubevirt_vm_info'.

$ curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=kubevirt_vm_info'

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"kubevirt_vm_info","cluster":"local-cluster","clusterID":"110b2032-b6b9-455a-abd5-9949482b61df","container":"virt-controller","endpoint":"metrics","flavor":"small","instance":"10.129.0.198:8443","job":"kubevirt-prometheus-metrics","machine_type":"pc-q35-rhel9.6.0","name":"centos-stream9-white-mink-62","namespace":"acm-test","os":"centos-stream9","pod":"virt-controller-7744b59dd8-dk7tw","receive":"true","service":"kubevirt-prometheus-metrics","status":"running","status_group":"running","tenant_id":"2c1cf409-966c-4c2d-bd57-f24c4c5a9393","workload":"server"},"value":[1762574206.765,"1"]},{"metric":{"__name__":"kubevirt_vm_info","cluster":"local-cluster","clusterID":"110b2032-b6b9-455a-abd5-9949482b61df","container":"virt-controller","endpoint":"metrics","instance":"10.129.0.198:8443","instance_type":"u1.medium","job":"kubevirt-prometheus-metrics","machine_type":"pc-q35-rhel9.6.0","name":"rhel-10-rose-damselfly-10","namespace":"acm-test","pod":"virt-controller-7744b59dd8-dk7tw","preference":"rhel.10","receive":"true","service":"kubevirt-prometheus-metrics","status":"running","status_group":"running","tenant_id":"2c1cf409-966c-4c2d-bd57-f24c4c5a9393"},"value":[1762574206.765,"1"]}],"analysis":{}}}

2. Create a configmap yml file, related to the custom alerting rule. Assume the configmap name is "thanos-ruler-custom-rules.yaml". The content can be as following.

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-ruler-custom-rules
  # need to ensure namespace is "open-cluster-management-observability"
  namespace: open-cluster-management-observability
  labels:
    # need this label to ensure custom alert rules are automatically mounted into Thanos Ruler pods
    thanos-ruler-rule: "true"
data:
  # name has to be custom_rules.yaml, custom alert rules are automatically mounted into Thanos Ruler pods successfully
  custom_rules.yaml: |
    groups:
    - name: VirtualMachineAlerts
      interval: 30s
      rules:
      - alert: VirtualMachineNotRunning
        expr: kubevirt_vm_info{status!="running"} > 0
        for: 2m
        annotations:
          summary: "VM {{ $labels.name }} is Not in running status on {{ $labels.cluster }}"
          description: "VM {{ $labels.name }} in namespace {{ $labels.namespace }} is not running."
        labels:
          severity: critical
          alert_type: vm_not_running

The essence is to leverage the existing Prometheus metrics "kubevirt_vm_info" and PromQL kubevirt_vm_info{status!="running"} > 0, which can fire an alert if detect any virtual machines are not in the Running status.

Attention a: To ensure the custom alert rules are automatically mounted into Thanos Ruler pods successfully, it need to meet the following 2 conditions:

• Ensure the configmap is labeled with: thanos-ruler-rule: "true"
• Ensure the "key" is set to "custom_rules.yaml"

Attention b: alert_type: vm_not_running has to match with the secret which gonna be created soon.

3. Apply the configmap by running "oc -n open-cluster-management-observability apply -f thanos-ruler-custom-rules.yaml". Then validate the output "/etc/thanos/rules/thanos-ruler-custom-rules from thanos-ruler-custom-rules", which means the custom rule has been mounted to the pod "observability-thanos-rule".

% oc describe pod observability-thanos-rule-0

Mounts:

  /etc/thanos/config/thanos-ruler-config from thanos-ruler-config (ro)

  /etc/thanos/configmaps/alertmanager-ca-bundle from alertmanager-ca-bundle (rw)

  /etc/thanos/rules/thanos-ruler-custom-rules from thanos-ruler-custom-rules (rw)

  /etc/thanos/rules/thanos-ruler-default-rules from thanos-ruler-default-rules (rw)

  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c55nd (ro)

  /var/thanos/rule from data (rw)

4. Create a secret yaml file named as alertmanager-config.yaml, which configures receiver, and slack webhook url, slack channel, etc.

Attention: ensure the alert_type is consistent with the one defined in the configmap which will be created later.

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  # namespace has to be open-cluster-management-observability
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
      # your slack webhook url
      slack_api_url: 'https://hooks.slack.com/services/\<replace_with_your_webhook\>'

    route:
      receiver: 'default-receiver'
      group_by: ['alertname', 'cluster', 'namespace', 'name']
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 12h

      # Specific sub-route for VM not running alerts
      routes:
        - match:
            # ensure the alert_type is consistent with the one defined in configmap
            alert_type: vm_not_running
          receiver: 'slack-vm-not-running'
          group_wait: 5s
          repeat_interval: 10m

    receivers:
      # default receiver (no-op)
      - name: 'default-receiver'

      # Slack receiver for VM not running alerts
      - name: 'slack-vm-not-running'
        slack_configs:
          # slack channel name, do NOT miss "#" before you type the channel name
          - channel: '#openshift-virt-observability-alerts'   # <-- adjust to your Slack channel
            title: '🚨 VM NOT RUNNING ALERT'
            text: |
              *Alert:* {{ .CommonLabels.alertname }}
              *VM Name:* {{ .CommonLabels.name }}
              *Namespace:* {{ .CommonLabels.namespace }}
              *Cluster:* {{ .CommonLabels.cluster }}
              *Severity:* {{ .CommonLabels.severity | toUpper }}
              *Status:* {{ .Status | toUpper }}

              {{ range .Alerts }}
              *Description:* {{ .Annotations.description }}
              {{ end }}
            send_resolved: true
            color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
            actions:
              - type: button
                text: 'View in Console'
                url: 'https://console-openshift-console.apps.{{ .CommonLabels.cluster }}/k8s/ns/{{ .CommonLabels.namespace }}/virtualmachines/{{ .CommonLabels.name }}'

5. Apply the secret by running: oc -n open-cluster-management-observability apply -f alertmanager-config.yaml

6. Stop a virtual machine by running: virtctl stop <vm-name>, and wait for the vm stopped

7. Check slack channel, and see if there could be corresponding alert like following generated

You can verify the generated alert through alert manager console too. To open the alert manager console, you can run: oc -n open-cluster-management-observability port-forward pod/observability-alertmanager-0 9093:9093, then access http://localhost:9093/#/alerts, to check if the alert is generated there. An example is as following, which shows "slack-vm-not-running" alert is generated.

8. Start a virtual machine by running: virtctl start <vm-name>, and wait for the vm started.

Once vm get started back, check the slack channel which will tip the alert is resolved, as following example shows.

Additional monitoring features

RHACM can be used to monitor essential metrics like CPU, memory, network, storage usage for a VM, pod, cluster node, or an entire cluster. Most of the same metrics can be monitored with the base observability included in OpenShift, but RHACM allows the user to monitor these metrics across multiple managed clusters.

RHACM adds preconfigured Grafana dashboards designed to monitor VM specific metrics. For instance, within the "ACM / OpenShift Virtualization" dashboards folder, the "Executive dashboards / Single Virtual Machine View" dashboard displays CPU Usage, Memory Usage, Network Usage (Transmit/Receive, Packets Dropped), Storage Usage (IOPS/Traffic), and Filesystem Usage for an individual VM.

For a brief assessment of the metrics displayed in every preconfigured dashboard provided through RHACM as well as the base OpenShift observability, refer to this chart:

Adding/Customizing Dashboards

New dashboards can not be added with the default grafana instance, but new dashboards can be created and customized by first creating a grafana-dev instance. This can be done by following the instructions outlined in the RHACM Observability Documentation for setting up the grafana dev instance using the scripts found in https://github.com/open-cluster-management/multicluster-observability-operator.

If you do not have access to the kube:admin user (which applies to Red Hat OpenShift on IBM Cloud), create a dashboard through a series of terminal commands instead. In the terminal after running the ./setup-grafana-dev.sh --deploy command in the preceding instructions, run the following:

1. Set the environment variable used in the upcoming commands to the observability namespace:

   export OBS_NS=open-cluster-management-observability
   

2. Run the following to create a JSON definition for the ConfigMap:

   cat <<'EOF' > full-dashboard.json
   {
     "id": null,
     "uid": "60EDAD96-69C2-4047-88AC-541C29BB74BA",
     "title": "VM Metrics Dashboard",
     "tags": ["control-plane", "cluster-metrics"],
     "timezone": "browser",
     "schemaVersion": 30,
     "version": 1,
     "panels": [
       {
         "type": "graph",
         "title": "Top 50 API Server Latency",
         "targets": [{"expr": "apiserver_request_latency_seconds_bucket"}]
       },
       {
         "type": "graph",
         "title": "etcd Latency",
         "targets": [{"expr": "etcd_disk_wal_fsync_duration_seconds_bucket"}]
       },
       {
         "type": "graph",
         "title": "Node CPU Usage",
         "targets": [{"expr": "node_cpu_seconds_total"}]
       },
       {
         "type": "graph",
         "title": "Node Memory Usage",
         "targets": [{"expr": "node_memory_MemAvailable_bytes"}]
       }
     ]
   }
   EOF
   

3. Create the ConfigMap object using the YAML definition:

   cat <<EOF > full-dashboard.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: full-dashboard
     namespace: $OBS_NS
     labels:
       grafana-custom-dashboard: "true"
     annotations:
       observability.open-cluster-management.io/dashboard-folder: Custom
   data:
     full-dashboard.json: |-
   $(sed 's/^/    /' full-dashboard.json)
   EOF
   

4. Apply the new ConfigMap:

   oc apply -f full-dashboard.yaml
   

After applying the ConfigMap, there should be a new customer Grafana dashboard created that can be further modified.

New dashboards can be created using PromQL queries from a wide selection of KubeVirt Components Metrics and other Kubernetes Metrics.

Custom metrics can be exported from either Platform or User Workload sources as described in the Advanced Observability Configuration section. A list of available platform or user workload metrics targets can be viewed by navigating to the Observe at Targets page and filtering the Source by Platform or User, respectively.

If you do not have access to the kube:admin user (as in the case for Red Hat OpenShift on IBM Cloud), it might be necessary to run the following commands to create a set of custom metrics. The following are some example ConfigMaps that can be created:

For some basic metrics on Node Memory and API Server:

1. Create a ConfigMap YAML file:

   cat <<EOF > observability-platform-metrics.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: observability-metrics-custom-allowlist
     namespace: open-cluster-management-observability
   data:
     metrics_list.yaml: |
       names:
         - node_memory_MemTotal_bytes
         - node_memory_MemAvailable_bytes
         - kubelet_runtime_operations_duration_seconds
       recording_rules:
         - record: apiserver_request_duration_seconds:histogram_quantile_90
           expr: histogram_quantile(0.90,sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver", verb!="WATCH"}[5m])) by (verb,le))
         - record: etcd_disk_wal_fsync_duration_seconds:histogram_quantile_90
           expr: histogram_quantile(0.90,sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance,le))
   EOF
   

2. Apply the newly created ConfigMap:

   oc apply -f observability-platform-metrics.yaml
   

For some user workload metrics:

1. Create a ConfigMap YAML file:

   cat <<EOF > observability-uwl-metrics.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: observability-metrics-custom-allowlist
     namespace: open-cluster-management-observability
   data:
     uwl_metrics_list.yaml: |
       names:
         - node_memory_MemTotal_bytes
         - node_memory_MemAvailable_bytes
   EOF
   

2. Apply the newly created ConfigMap:

   oc apply -f observability-uwl-metrics.yaml
   

RightSizing Recommendation Dashboard

RHACM also provides a dashboard showing recommendations for optimizing CPU and Memory allocations based on resource usage, if the Right-Sizing feature is enabled in the MultiClusterObservability custom resource on the hub cluster.

Viewing Metrics and Dashboards

To view the observability metrics and dashboards included in the base install of OpenShift:

1. From the left panel of the OpenShift console, select Observability.
2. Select Metrics to generate a graph visualization from a PromQL query.
3. Select Dashboards and choose from a number of pre-configured dashboards in the Dashboard drop down list.

To view the observability metrics and dashboards with RHACM installed:

1. From the OpenShift console, select All Clusters from the drop down menu at the top.

   1. 

2. From the Clusters page, select the Grafana link on the right.

   1. 

3. This will bring up the grafana dashboards page with a list of pre-configured dashboards that come with the RHACM installation.

General introduction

Two logging options are provided – LokiStack and IBM Cloud Logs – because each serves different needs. LokiStack offers in-cluster, customizable log collection and retention, while IBM Cloud Logs provides a managed, cloud-native, ease-of-use logging experience. You can choose one or use both depending on resources, scale, and visibility needs. The following sections outline the requirements for each option and detail the capabilities that are included so you can decide which solution, or combination, fits your environment best.

Things to note:

• IBM Cloud Logs supports easy integration when the Red Hat® OpenShift® Kubernetes Service instance is managed.
• LokiStack logging works for both managed and RYO configurations and provides native, in-console integration within the OpenShift environment.
• OpenShift RHACM is a tool for centrally managing multiple OpenShift clusters, but LokiStack itself functions as an in-cluster log aggregation system within a single OpenShift cluster.
• It is not required to have OpenShift Advanced Cluster Management (RHACM) installed to use LokiStack for logging.

Installation and configuration

To provision IBM Cloud Logs and connect it to your OpenShift cluster, perform the following steps:

1. Create IBM Cloud Logs instance
   1. ref: Provision instance
   2. This can be done through UI and CLI
2. Create a Storage Data Bucket
   1. ref: Configure Bucket
   2. This is required if you want long term data retention or search. the standard Cloud Logs instance has 7 days minimum and 90 days maximum priority log retention
3. Enable logging with Red Hat OpenShift Kubernetes Service cluster
   1. ref: Connect ICL with Red Hat OpenShift Kubernetes Service

Logging Rollback/Deletion - IBM Cloud Logs

To remove the IBM Cloud Logs integration and delete associated resources, perform the following steps:

1. Delete the Cloud Logs instance
   1. ref: Delete ICL instance
2. Delete the Storage Data Bucket
   1. ref: Delete Bucket

Logging UI - IBM Cloud Logs

Comparison with VMware Aria Logs (vRealize Log Insight vs IBM Cloud Logs)

Component Overview

Operator: LokiOperator (LokiStack)
version: 6.3.1
provider: Red Hat
notes: Forwards logs to the logging supported combination of Loki and web proxy with OpenShift Container Platform authentication integration. LokiStack's proxy uses OpenShift Container Platform authentication to enforce multi-tenancy
documentation: Loki Sizing

Operator: Red Hat OpenShift Logging (ClusterLogForwarder (CLF))
version: 6.3.1
provider: Red Hat
notes: The ClusterLogForwarder (CLF) enables flexible forwarding of log data to various destinations by allowing users to select log messages from multiple sources, process them through pipelines, and send them to one or more outputs. It supports inputs for selecting logs, outputs for forwarding to external systems, filters for transforming or dropping log messages, and pipelines that link inputs, filters, and outputs into a complete log-forwarding workflow.
documentation: Configuring Log Forwarder

Operator: Cluster Observability Operator (COO)
version: 1.3.0
provider: Red Hat
notes: The Cluster Observability Operator (COO) is an optional OpenShift component that lets administrators deploy custom, highly configurable monitoring stacks, providing deeper namespace-level observability than the default system. It can manage Prometheus for high-availability metrics and remote-write support, Thanos Querier for centralized cross-cluster querying, Alertmanager for flexible alerting, UI plugins for enhanced monitoring/logging/tracing interfaces, and Incident Detection to group related alerts into incidents and help identify root causes.
document: COO Function

Centralized Logging Architecture

Installation and configuration (Short with Refs)

To quickly configure centralized in-cluster logging using OpenShift Logging and LokiStack, perform the following steps:

1. Install the LokiOperator, Red Hat OpenShift Logging Operator, and Cluster Observability Operator (COO) from the software catalog.
   1. ref: How to Install Operators
2. Create and configure service account as required by Cluster Logging
   1. ref: Service account settings
3. Create a LokiStack instance
   1. ref: Create LokiStack
4. Create the UI for centralized logging
   1. ref: UI creation
5. Create the ClusterLogForwarder (CLF) instance
   1. ref: Creating a Log Forwarder

Installation and configuration (Detailed, custom)

To deploy a custom LokiStack logging configuration with detailed service accounts and resource specifications, perform the following steps:

1. Install the LokiOperator, Red Hat OpenShift Logging Operator, and Cluster Observability Operator (COO) from the software catalog.
   1. ref: How to Install Operators
2. Create and configure service account as required by Cluster Logging
   1. ref: Service account settings
   2. Commands to configure service account:

oc create sa loki-collector-sa -n openshift-logging

oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z loki-collector-sa  -n openshift-logging

oc adm policy add-cluster-role-to-user collect-application-logs -z loki-collector-sa -n openshift-logging

oc adm policy add-cluster-role-to-user collect-audit-logs -z loki-collector-sa -n openshift-logging

oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z loki-collector-sa -n openshift-logging

• These will provide the account that is created to have the correct permissions to the various types of logs

3. Create a LokiStack custom res using yaml input
   1. ref: Create LokiStack
   2. Custom yaml provides the best example for a working instance with optimizations in place:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  managementState: Managed
  size: 1x.medium

  storage:
    schemas:
      - effectiveDate: "2024-11-11"
        version: v13
    secret:
      name: logging-loki-secret
      type: s3
  storageClassName: ocs-storagecluster-ceph-rbd
  tenants:
    mode: openshift-logging
  template:
    compactor:
      replicas: 1
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
        requests:
          cpu: "250m"
          memory: "1Gi"

    distributor:
      replicas: 1

    gateway:
      replicas: 1

    indexGateway:
      replicas: 1

    ingester:
      replicas: 2

    querier:
      replicas: 2

    queryFrontend:
      replicas: 2

    ruler:
      replicas: 1
  lokiConfig:
    query_range:
      align_queries_with_step: true
      max_retries: 5
      parallelise_shardable_queries: true
      cache_results: true
      results_cache:
        cache_validity: 10m
        background_writeback: true

    frontend_worker:
      frontend_address: query-frontend-http.logging-loki.svc.cluster.local:9095
      grpc_client_config:
        max_send_msg_size: 104857600
        max_recv_msg_size: 104857600

    querier:
      query_ingesters_within: 12h
      engine:
        timeout: 5m
        max_look_back_period: 672h

    compactor:
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      working_directory: /var/loki/compactor
      compaction_window: 24h
      retention_stream:
        default: 30d

• Once PVC storage is created it can be easily increased but that's not the same for decreasing. To decrease, you must redeploy instance with a new set of minimums in order to achieve that.
  • Storage can also be a s3 bucket.
• Even if using local Ceph/PVC, a functioning S3 secret is required.

4. Create the UI for centralized logging
   1. ref: UI creation
   2. Custom yaml used to create the UI from the previously installed COO:

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: logging
spec:
  type: Logging
  logging:
    lokiStack:
      name: logging-loki
    timeout: 30s
    schema: otel

5. Create the Log forwarding custom resource (CR) using yaml method
   1. ref: Creating a Log Forwarder
   2. This example of a running log fowarder for lokistack:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: loki-forwarder
  namespace: openshift-logging
spec:
  collector:
    resources:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        cpu: 6
        memory: 6Gi
  serviceAccount:
    name: loki-collector-sa
  outputs:
    - name: forward-to-lokistack
      type: lokiStack
      lokiStack:
        authentication:
          token:
            from: serviceAccount
        target:
          name: logging-loki
          namespace: openshift-logging
        tuning:
          deliveryMode: AtLeastOnce
          compression: none
          maxWrite: 10Mi
          minRetryDuration: 2
          maxRetryDuration: 30
  pipelines:
    - name: default-pipeline
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - forward-to-lokistack

• Resource minimums should be set prior to deployment to prevent a failing collector on each Node
• There is only 1 collector daemonset per Node

Important Notes

Loki Sizing:

ref: Loki sizing document

• On deploy these are the general resource maximums that you are allowed per-sizing
• Once the disks sizing are set, you cannot decrease them. you can only increase. must delete your deployment and redeploy stack with new minimum requests.
  • This destroys logs unless backed up externally
• Ceph PVC is not required, can use external or other S3 storage config

Log forwarder collector

• Forwarding is hardware intensive and memory will need to be sized to align with scale of environment

Supported third party log forwarding solutions:

• azureMonitor
  • Forwards logs to Azure Monitor.
• cloudwatch
  • Forwards logs to AWS CloudWatch.
• elasticsearch
  • Forwards logs to an external Elasticsearch instance.
• googleCloudLogging
  • Forwards logs to Google Cloud Logging.
• http
  • Forwards logs to a generic HTTP endpoint.
• kafka
  • Forwards logs to a Kafka broker.
• otlp
  • Forwards logs using the OpenTelemetry Protocol.
• splunk
  • Forwards logs to Splunk.
• syslog
  • Forwards logs to an external syslog server.

ref: Log Forwarding Outputs

Logging UI - LokiStack

Comparison with VMware Aria Logs (vRealize Log Insight vs LokiStack)

Logging Rollback/Deletion - LokiStack

To uninstall LokiStack and remove in-cluster logging components, perform the following steps:

1. Navigate to each operator and delete each of the created instances. This will delete all associated components used to run the services pods.
   1. Ref: Uninstalling Logging

Aria Suite

Openshift is able to replicate the functionality of the VMware Aria Suite though the base install and a few additional operators. For Aria Log Insight, OpenShift Logging will be necessary while much of the functionality of Aria Operations is available at the start. RHACM Observability adds further features to make it more in line with the larger Aria Operations components.

IBM Cloud Monitoring/Sysdig