Module 2: Logging with LokiStack

Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus. Unlike traditional logging systems that index the full text of log messages, Loki only indexes metadata (labels) and stores log content separately.

Key differences from traditional logging:

LokiStack components:

In OpenShift, logs flow through this pipeline:

You’ll configure the ClusterLogForwarder to ensure application logs reach Loki.

Check that your logging infrastructure is operational:

What you learned: OpenShift Logging uses a collector (based on Vector) to gather logs from all pods and forward them to LokiStack according to ClusterLogForwarder rules.

Issue: LokiStack status shows "Degraded" or "Pending"

Solution: . Check LokiStack pods for errors: oc get pods -n openshift-logging | grep loki . Describe the LokiStack: oc describe lokistack logging-loki -n openshift-logging . If reason is MissingObjectStorageSecret, check sync Job status:

+

Issue: No collector pods running

Solution: . Check DaemonSet status: oc get daemonset -n openshift-logging . View collector logs: oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=100 . Verify ClusterLogForwarder resource exists: oc get clusterlogforwarder -n openshift-logging

Issue: ClusterLogForwarder not found

Solution: The logging stack may not be fully configured. Check that the GitOps deployment completed successfully.

Check that logs are being generated:

What you learned: Applications write logs to stdout/stderr, which OpenShift captures and forwards to your logging backend. Multiple services can run in the same namespace.

Click on the "show query" button to open the LogQL query editor

This shows note creation and note update calls from the frontend route.

Check that you can query logs effectively:

What you learned: LogQL uses labels (like namespace, pod name) to select log streams, then filters content with operators like |= (contains) and != (does not contain). Parsing stages (|json, | regexp) extract structured fields, and | unwrap unlocks metric-style aggregations over those fields.

Issue: No logs appear in query results

Solution: . Wait 1-2 minutes for logs to be ingested by Loki . Verify pods are running: oc get pods -n %OPENSHIFT_USERNAME%-observability-demo . Check collector is forwarding logs: oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=100 . Verify time range in query interface (expand to last 1 hour)

Issue: Query returns "Error: parse error"

Solution: . Check LogQL syntax (labels in braces, filters with pipes) . Ensure label names are correct: kubernetes_namespace_name not namespace . Use quotes around label values: {kubernetes_namespace_name="%OPENSHIFT_USERNAME%-observability-demo"}

Issue: Rate queries return no data

Solution: . Ensure enough matching logs exist (generate more traffic) . Check time range: [5m] requires data in last 5 minutes . Verify filter matches log content: |= "404" requires "404" in log lines

Navigate to Workloads → Deployments → In the top right, click the "dotted kube" icon → "Signal Correlation"

Check that you can correlate metrics and logs:

What you learned: Metrics tell you when and where problems occur. Logs provide the detailed context explaining why. Together, they reduce mean time to resolution through unified analysis.

Issue: Metric and log timestamps don’t align

Solution: . Check timezone settings in console (both views should use same timezone) . Verify system clocks are synchronized across cluster nodes . Allow 30-60 seconds for metric scraping and log ingestion delays

Issue: Cannot extract structured data with regexp

Solution: . Verify log format matches regex pattern . View raw logs first: {kubernetes_namespace_name="%OPENSHIFT_USERNAME%-observability-demo"} . Test regex pattern against actual log samples . Use simpler filters first, then add complexity

The Loki ruler evaluates AlertingRule resources (a CRD provided by the Loki Operator). When an expression matches, the ruler sends the alert to Alertmanager, which routes it according to its normal configuration (email, PagerDuty, Slack receivers, etc.).

For the application tenant, AlertingRule resources must be created in the application namespace and that namespace must carry the label openshift.io/loki-rules: "true". The workshop pre-provisions this label on your demo namespace.

A complete AlertingRule that fires when the 404 error rate exceeds 0.5 per second for 2 minutes looks like this:

The key fields map directly to Prometheus alerting semantics:

Once created, the alert follows the same lifecycle as any Prometheus alert:

Navigate to Observe → Alerting → Alerting rules in the OpenShift console to see log-based alerts alongside metric-based ones.

What you learned: LokiStack’s ruler component evaluates LogQL expressions and fires alerts through the same Alertmanager pipeline used by Prometheus. This means log patterns — error bursts, missing heartbeats, security events — can trigger the same notifications and on-call workflows as metric alerts, with no additional tooling required.