Member blog post originally published on the Logz.io blog by Jake O’Donnell
Observability isn’t new. But organizations are struggling to adopt mature observability practices, and the impact on business is palpable.
Organizations are seeing the value of observability for their applications and infrastructure—the results of our 2024 Observability Pulse survey of 500 global IT professionals reflects that across the board. But respondents are challenged by the notion of scaling tools and processes, ensuring the security of their data, managing huge data volumes and their associated cost, and a lack of knowledge on teams about how to deal with it all.
For the first time, we asked respondents about where they reside in their observability journey today. In total, 89% of respondents have observability on their radar—from those at least planning to start their observability journey through to those using full observability now.
Of those, some 36% of respondents said they feel they’ve partially started their journey, with plans to add more capabilities, followed by 20% saying they’re planning to start. And yet, just 10% said they’ve achieved full observability.
We also asked how organizations are currently practicing observability, and the two most common responses were using metrics to gain insight into systems or applications (64%) and consolidated analysis of logs and metrics (61%). It’s fair to say these are fairly basic cornerstones of observability, but it’s enlightening to see organizations taking advantage of these disciplines.
Some areas that have gotten attention in the wider observability market appear to have slower uptake. These include unified analysis of Kubernetes infrastructure (27%), combining security data with other telemetry (23%) and applying pipeline analytics to optimized related data (18%).
Teams certainly are seeing benefits from increased observability, with 60% saying they’re improving and accelerating troubleshooting, and 56% reporting they’re gaining central visibility into apps and infrastructure. But significant challenges remain.
In another sign of nascent observability maturity, lack of knowledge among teams was cited as the biggest challenge to gaining observability into cloud-native environments, clocking in at 48%. This is up from 30% in 2023.
In the face of a global engineering talent gap, even those organizations actively engaged in observability would appear to lack critical experience and insights needed to move forward.
Despite widespread adoption of observability tooling and significant focus on integration of related practices and data, most organizations have not yet achieved their vision of observability practices. This issue is underlined by general lack of expertise on how to address this pervasive issue.
Also, MTTR—likely the most critical measure of an observability practice’s effectiveness—has trended in the wrong direction for the third consecutive year, despite the prevalence of numerous analysis and troubleshooting tools, and the growing recognition of observability’s related importance.
How to Add Maturity Your Observability Practice
Going it alone with your observability practice—especially if you’re a smaller organization with limits on technical resources—can be a deeply challenging exercise. But through the adoption of both tooling and frameworks, you can start to move in a better and more mature direction.
Some of the results from the survey show things trending more positively for some. For the first time, we asked Pulse survey respondents about their use of OpenTelemetry (OTEL), the open source project providing an observability framework that assists in generating and capturing telemetry data from cloud-native software. Over three-quarters (76%) said adoption of OTEL or OTEL-centric tooling was at least somewhat important to their overall observability strategy.
As environments become more complex and costly to gain observability into, respondents to Pulse appear to be increasingly less reliant on siloed open source tools to reach their goals.
While open source tools for observability maintain broad adoption, organizations appear to be relying on them less than in previous years. Organizations reporting that less than half of their observability tools are open source grew from 46% in 2023 to 68% in 2024.
On top of frameworks, organizations should rely on trusted platforms to help provide the tooling necessary to reach your observability goals and grow a mature practice. Logz.io Open 360™ is a cloud-native observability platform that gives you the tools to monitor, troubleshoot and remediate issues that show up in your telemetry data—logs, metrics and traces—as well as forward-thinking ways to monitor your critical applications and infrastructure.
With Open 360, you’ll get access to our following offerings:
- Log Management: Quickly explore your data with intuitive and high performance search filters, or accelerate troubleshooting with Log Patterns to scan through your log data in seconds. Visualize spikes, dips, and other trends in your logs with prebuilt and customizable monitoring dashboards. We offer log management experiences based on OpenSearch Dashboards as well as Explore, our fully-managed service built on OpenSearch.
- Infrastructure Monitoring: Slice and dice your metric data to quickly gain insights into any component within ephemeral infrastructure. Track tens of thousands of metrics out-of-the-box with open source-based integrations like Prometheus or Logz.io’s Telemetry Collector, while easily filtering out unneeded data to reduce costs.
- Distributed Tracing: Quickly surface and debug issues in microservices architectures. Start with an overview of application performance across your services, and quickly drill into individual transactions to diagnose the root cause of issues. You’ll easily visualize the source of latency across application requests within microservices architectures.
- Kubernetes 360: Combine Kubernetes logs, metrics, and traces for unified analysis, troubleshooting and provides automatic contextualization and organizes relevant telemetry data by node or deployment. The solution adds context to monitoring dashboards by annotating them with production events which simplifies the process of correlating changes in production with service health and performance.
- App 360: Unify distributed tracing, service topology and services analysis to provide full visibility into application health and performance, formalizing an application observability-based alternative to traditional APM. App 360 is exceptionally easy to use with Logz.io’s Telemetry Collector enabling automatic service discovery, instrumentation and data collection for logs, metrics, and traces.
Additionally, Logz.io has developed a Well-Architected Observability Framework to help organizations like yours plan, prioritize, and measure progress of the many elements that make up an observability initiative. The end goal of this effort is to help you achieve desired observability outcomes—including faster MTTR, lower costs, simplified observability ops, and data security. Reach out to us to learn more.
You can also learn more here about Observability IQ, our new cutting-edge AI-based feature, that facilitates automated querying and interaction with the platform – in conversational terms. Now, you’re not just viewing data but engaging in a dynamic conversation with it. Observability IQ specifically invites you to dive deeper into your metrics, enabling analysis that unveils comprehensive insights and foster an unparalleled understanding of your systems.
In general, as much progress is being made on the observability spectrum, at large, attainment of organizational maturity and process efficiency clearly remains in the nascent stage.
Organizations must find a way to overcome their current technical and process challenges to engage practical observability that meets the vision for unified, cross-stack visibility and advanced troubleshooting that improves user experience and drives down MTTR.
Increasing maturity to best enable available talent while more closely controlling costs will continue to be critical requirements, based on the feedback of practitioners responding to this research.
If you want to see how Logz.io Open 360 can help you reach your observability maturity goals, sign up for a free trial today.