How to effectively store and analyze petabytes of log data

Let’s face it.

Logging is hard.

Sure, it’s relatively easy to throw your logs into Amazon Cloudwatch or Google Cloud Logging and call it a day. In fact, many common deployment platforms on public clouds do this for you (see AWS Lambda or Google Cloud Run).

But logs aren’t meant to be stored and forgotten about, they’re meant to be useful to your business. Logs sitting in storage accomplish nothing: it’s only when they are decoded, analyzed, and measured that you can gain insights and make decisions based on the data they contain.

There are 3 key areas to consider when logging:

1. Structured Logging
2. Log Storage
3. Log Querying

1. Structured Logging

Logs should be well-structured. By adding tags* Key-value pairs describing the source of the log. Usually static over the lifetime of a service or machine. , metadata* Key-value pairs describing the log entry. May change often. , and timestamps* The time the log was generated. , you can make your logs more readable and easier to parse programmatically.

Most storage engines make extensive use of metadata and tags to enable efficient querying. In fact, some storage engines, such as Grafana Loki, only allow querying by metadata or tags unless you want to traverse through every log (and similar to a full table scan in a DB, you don’t want to do this).

When you properly structure logs, you allow alerting on anomalous metadata, calculating metrics based on log timestamps, and retrospective log exploration during incident post-mortems.

2. Log Storage

There are a wide variety of logging storage solutions, ranging from SaaS vendors to self-hosted solutions. When choosing where to store your logs, there are several factors to consider.

Do your logs need to be retained for a long time?

If so, you should choose lower-cost object storage such as Amazon S3, which can be used by Grafana Loki and similar solutions.

Do you need instant access to your logs?

Error logs or security logs that need to be analyzed immediately often use real-time storage that can be accessed in milliseconds, while audit logs or archival logs can use a slower storage medium such as low-cost archival storage.

How much data are you generating?

If you’re generating a lot of logs, you’ll need a platform that can ingest logs quickly, not just store them. Managed platforms are designed for this, but they can get expensive quickly.

What is your budget?

Self-hosting logging solutions can be cheaper, but also cost your business valuable setup and maintenance time. Managed platforms are often more expensive but are entirely managed and operated for you.

3. Log Querying

Assuming you’ve decided where to store and ingest your logs, you need to put them to use. Logs should be checked often for errors, especially ones that are client-facing. Most logging solutions allow you to set up automated alerts when errors are found, but be sure not to overdo it (read our other post to discover why this can impact your business and team).

Metrics and logs are often tightly coupled, too. Because logs often contain timestamps* The time the log was generated. , you can calculate how long requests take and how quick—or slow—your application is behaving.

Logs are also a useful tool for measuring SLAs* Service Level Agreements (SLAs): a defined percentage of time you legally assure your customer that product is online and working. and SLOs.* Service Level Objective (SLO): The same as an SLA but not contractually assured. Collecting logs that measure SLAs and SLOs can result in lower SLA payouts* Money you pay your customer when you violate your SLA. , as they can be used to show your product was online and functioning when the customer might believe it was not.

Finally, nothing is more important during incident post-mortems than being able to look back and see what went wrong. When conducting retrospectives, being able to quickly and accurately filter logs for errors, anomalies, and problems is crucial.

Conclusion

Properly collecting, storing, and querying logs is essential to ensuring your product is operating at it’s best. We hope these tips help your business stay online, active, and working properly.

If you’ve got a logging solution already, then great! Otherwise, you should sign up for a free LogAI account to easily ingest massive amounts of logs, analyze them using cutting-edge AI tooling, and report and alert on any anomalies or warnings hidden within your data. Plus, for a limited time, get a free ebook with advice on how you can streamline your log gathering, analysis, and reporting—when using LogAI or otherwise.