Streamlining Log Management with OTTL Rules

This document outlines a process to enhance log management by parsing unformatted logs using OTTL rules in Groundcover. The steps leverage an AI agent to automatically generate parsing rules, facilitating efficient log search, dashboard creation, and query execution.

Step 1

Start by identifying unformatted logs, which typically appear as raw strings of text.

Step 2

Leverage structured logs to enhance searchability by key value, dashboard building, and executing efficient queries. Groundcover addresses this by parsing logs using OTTL rules, with automatic rule generation facilitated by an AI agent.

Step 3

Begin the process by filtering out unformatted logs using the "format unknown" filter.

Step 4

Further refine the search by focusing on a specific workload. In this example, we will concentrate on the task generator workload, where there is a significant number of logs with an unknown format.

Step 5

Identify patterns within the logs. Use the "Actions" menu to select "Generate Parsing Rules." The AI agent will then sample logs that match the current search criteria.

Step 6

The AI agent organizes logs by structure and generates OTTL rules for each distinct pattern. After the rules are generated, you can review them in the result section. Clicking on a log provides a side-by-side comparison of the log before and after parsing.

Step 7

Evaluate the effectiveness of the rules. For instance, by selecting a rule, you can see extracted attributes that were initially not extracted. If satisfied, proceed to create the rules.

Step 8

Arrange and prioritize the newly created rules since order affects the pipeline processing. Save the pipeline, and once applied, the rules are hot reloaded across all system sensors with no need for restarts.