In the past month, we have engaged in conversations with over 50 engineers, engineering managers, and SREs to gather feedback on the products we are developing at Keep. Here is a summary of what we have learned.
TL;DR: Creating and maintaining effective alerts, avoiding alert fatigue, and promoting a strong alerting culture can be difficult tasks. Keep addresses these challenges by treating alerts as code, integrating with observability tools, and using LLMs. Want to learn more? talk with me at shahar@keephq.dev, join Keep Slack or just start play with Keep.
Why alerting?
Now, let’s discuss why alerting is crucial.
With the increasing reliance on digital systems, monitoring, and alerting have become more critical than ever. Downtime or slow website performance can lead to significant financial losses and drive customers away to other competitors.
To meet the growing demand for observability, there has been a significant proliferation of observability tools, with companies like Datadog, Grafana, New Relic, Elasticsearch, and Splunk dominating the market. In addition, many other tools like Sentry, Coralogix, Sumo Logic, and BugSnag have also gained widespread adoption.
According to Grafana Labs Observability Survey 2023, 52% of companies use 6 or more observability tools (!) highlighting the significance of the problem.
Tools that fire alerts
So what’s the problem, if any?
There are several current problems around alerting that hinder companies from getting the most out of their monitoring systems. Let’s review them.
1. Alerts fatigue
One of the significant problems with alerting is alert fatigue. When you receive too many alerts, it can be challenging to determine which ones are critical and which ones can be ignored. This can lead to a lack of attention to alerts, which can ultimately result in missing critical issues.
2. Monitoring your monitoring
It’s essential to ensure that your monitoring tools are working correctly and providing accurate and timely alerts. However, it can be challenging to keep track of all the monitoring tools that you’re using, and it can be even more challenging to keep them all in sync.
3. Alerts maintenance
As systems change, alerts may need to be updated to reflect these changes. However, it can be challenging to keep track of all the alerts that need to be updated, leading to outdated alerts being triggered (or not triggered at all).
4. Lack of developer experience
Many companies rely on developers to set up and maintain their alerting systems. However, not all developers have the necessary experience to create effective alerting systems.
5. Too many tools
Finally, many companies have too many alerting tools, making it challenging to keep track of alerts, leading to confusion and missed alerts.
In conclusion, monitoring and alerting are critical for companies to ensure their systems run efficiently and effectively. However, several current problems with alerting can hinder companies from getting the most out of their monitoring systems. By understanding these problems, companies can work to overcome them and ensure that their monitoring systems provide accurate and timely alerts.
How does keep solving that?
Keep High-Level Architecture
Keep takes a holistic approach to solving all these problems with alerting. By treating alerts as code and integrating them with existing observability tools, along with leveraging AI, Keep can achieve the following objectives:
- Measure engagement, reduce noise, add context, and fine-tune the alerts.
- Single pane of glass — by integrating with all of your observability tools, Keep decoupling and deduplicating alerts so if your database is down, you’ll know that the alerts from the frontend are because of that.
- Using Keep’s CI/CD integration, you can maintain your alerts as easily as adding a new step in your GitHub Action.
- Decouple what you want to alert from the actual tool —using Keep’s semantic layer, a developer can just write “Using Datadog, alert me when service X is down for than Y minutes”.
Summary
In today’s digital age, monitoring and alerting are critical for ensuring the smooth functioning of a company’s systems. However, several problems with alerting are hindering companies from getting the most out of their monitoring systems. These problems include alert fatigue, difficulty in monitoring your monitoring, alert maintenance, lack of developer experience, and too many alerting tools. Keep, a platform that treats alerts as code and integrates them with existing observability tools, provides a holistic approach to solving these issues. By leveraging AI and using a semantic layer, Keep can measure engagement, reduce noise, add context, and fine-tune alerts, providing a single pane of glass for all observability tools, and allowing for easy maintenance of alerts.
