Introduction
For platform and observability teams, noisy logs are a persistent challenge. Health check messages, forgotten DEBUG statements, and verbose INFO logs from seldom-used services can quickly inflate your logging bill without delivering value. The traditional solution—toiling through infrastructure changes to suppress logs at the source—is slow and painful. Grafana Cloud now offers a simpler approach with Adaptive Logs drop rules, currently in public preview. This feature lets you define custom rules to drop low-value logs before they are written to Grafana Cloud Logs, instantly reducing noise and costs.
How Drop Rules Work
Each drop rule uses a combination of log labels, detected log levels, or line content to decide which logs to discard. You can set a drop percentage to sample repetitive logs rather than eliminate them entirely. Rules are evaluated in priority order, and the first matching rule applies its drop rate. This flexible logic allows you to target specific services, log levels, or even text patterns with precision.
Example Use Cases
- Drop logs by level: Instantly eliminate noisy DEBUG logs that dominate your logging budget and provide little insight.
- Sample chatty, repetitive logs: Instead of discarding all logs from a noisy source, use a drop percentage (e.g., 90%) to keep a representative sample while cutting costs.
- Target a specific noisy producer: When a service suddenly starts emitting high-volume, low-value logs, combine a label selector with a log level or text string to filter them out.
Integration with Adaptive Logs System
Drop rules are just one component of a complete log cost management system within Adaptive Logs. When a log line arrives in Grafana Cloud, it is evaluated in this sequence:
- Exemptions: Protected logs pass through untouched. If a log matches an exemption, no sampling is applied.
- Drop rules: Evaluated in priority order. The first matching rule applies its drop rate.
- Patterns: Optimization recommendations can be applied to remaining logs that were not exempted or dropped.
Exemptions, Drop Rules, and Patterns: A Complete System
Each mechanism serves a distinct purpose in managing log volume:
- Drop rules eliminate known noise: For instance, your platform team knows that health check logs need not be stored in Grafana Cloud. A single rule with a 100% drop rate enforces that standard across every service, without requiring individual teams to change their logging configuration.
- Drop rules apply sampling to specific workloads: A batch processing job generating repetitive log output can be targeted with a stream selector and a 90% drop rate, keeping a representative sample while drastically reducing costs.
Benefits and Best Practices
By using drop rules, centralized teams can quickly and easily prevent unwanted logs from being ingested, bypassing the cumbersome infrastructure change management process. This feature complements the intelligent optimization recommendations already available in Adaptive Metrics and Adaptive Traces. For best results, start by identifying the most costly and least useful log sources, then create targeted drop rules. Regularly review your drop rules to ensure they still align with your observability goals.
Start using drop rules today to take control of your logging costs and reduce noise. For detailed instructions, refer to the official documentation.