Tutorials
Unifying Observability: The Evolution of Distributed Tracing and Metrics with OpenTelemetry
In the dynamic landscape of cloud-native applications, effective observability has become paramount. For an extended period, the realm of distributed tracing was predominantly shaped by two distinct, yet complementary, solutions: OpenTracing and OpenCensus. These two projects, each with its unique approach to providing vendor-agnostic APIs for tracing, often presented developers with a challenging choice. However, a significant development occurred earlier this year when the teams behind these influential projects decided to consolidate their efforts, culminating in the inception of OpenTelemetry. This ambitious new initiative, now under the esteemed Cloud Native Computing Foundation (CNCF), is poised to revolutionize how organizations approach application performance monitoring.
While OpenTelemetry is currently undergoing active development, its foundational principles draw heavily from its predecessors. This article delves into the historical context provided by OpenTracing and OpenCensus, examines the current capabilities and architectural design of OpenTelemetry, and offers a glimpse into its promising future. To illustrate the practical implications of this merger, we will also explore comparative code examples that highlight the subtle yet impactful differences in their implementation.
A Look Back: The Foundations of OpenTelemetry
OpenTracing: Simplicity in Tracing
Launched in December 2016, OpenTracing quickly established itself as a leading CNCF project. Its primary focus was, as its name suggests, on distributed tracing, providing a streamlined and uncomplicated API for instrumenting applications. Supporting nine programming languages, OpenTracing's strength lay in its minimalist design, requiring only a few methods to satisfy its Tracer interface. This simplicity made it exceptionally easy for tracing systems, such as Jaeger, to integrate and offer client libraries that adhered to the OpenTracing standard. Both vendor-supported and community-driven tracers embraced this standard, facilitating a broad adoption across various ecosystems.
OpenCensus: Tracing, Metrics, and Google's Influence
Originating from Google's internal Census libraries and first released in January 2018, OpenCensus brought a more comprehensive approach to observability. Beyond distributed tracing, it also incorporated metrics collection, offering a more holistic view of application performance. With support for nine languages, sharing eight in common with OpenTracing, OpenCensus fostered a vibrant ecosystem of partners, including major cloud providers like Microsoft and numerous vendors in the tracing and metrics domains. Its design emphasized vendor neutrality, allowing for the implementation of exporters for various backend systems. While OpenCensus initially bundled some exporters with its libraries, a move towards separate repositories for better maintenance marked its evolution. This approach meant that application developers would typically import and initialize their desired exporter, a straightforward process for internal software. However, in complex cloud-native environments involving open-source microservices, managing different tracing backends (e.g., Jaeger and Zipkin) presented challenges, a problem also faced by OpenTracing. OpenCensus, however, offered an innovative solution.
The Innovation of OpenCensus Service
The OpenCensus Service emerged as a powerful solution to the complexities of data collection and routing. This collection of independent processes could receive observability data in its native OpenCensus format, or in formats like Zipkin, Jaeger, or Prometheus. Crucially, it could export this data to a multitude of monitoring backends, many of which were already supported by OpenCensus. Deployable as a collector near an application or as an agent within the same host (or as a sidecar container), the OpenCensus Service offered several compelling advantages. It decoupled exporter configuration from application code, enabling dynamic changes to backends without redeployment. It facilitated format conversion, for instance, from Zipkin to Jaeger, without requiring code modifications, a significant benefit for operators managing third-party software. Furthermore, it eliminated the need for applications to include and maintain exporter libraries, and centralized exporter implementation in a single language (Go) for OpenCensus maintainers. This comprehensive approach, encompassing both tracing and metrics, positioned OpenCensus as a more feature-rich solution compared to OpenTracing.
OpenTelemetry: The Synthesis of Strengths
OpenTelemetry represents a strategic consolidation, meticulously integrating the most advantageous features from both OpenTracing and OpenCensus. Upon its initial release, it is designed to support both distributed tracing and metrics, mirroring the capabilities of OpenCensus, with logging support slated for future integration. A key component inherited from OpenCensus is the OpenCensus Service, now re-envisioned as the OpenTelemetry Collector. This collector maintains its versatility, functioning identically as both an agent and a standalone collector, leveraging a unified binary and Docker image for these diverse use cases.
For existing users of OpenTracing or OpenCensus, the transition to OpenTelemetry promises to be smooth. The new standard is being developed with backward compatibility in mind, ensuring that migration to the new APIs can be undertaken at a measured pace as OpenTelemetry matures. This unified approach also brings substantial benefits to the broader CNCF ecosystem. The inclusion of metrics support in OpenTelemetry means that projects like Prometheus, another CNCF initiative, will have native support for metrics export from its inception. Similarly, Jaeger, which currently manages its own agent and collector implementations, is actively collaborating with OpenTelemetry to explore phasing out its proprietary components in favor of the new standard, as detailed in the discussions surrounding "Jaeger and OpenTelemetry."
Current Trajectory and Future Outlook
At present, all sub-projects within OpenTelemetry, including its specifications, client libraries, and the collector, are in their alpha development phase. Consequently, many of the libraries have not yet reached a versioned release status. Developers can anticipate high-quality preview releases of language SDKs throughout 2019, which will include support for the OpenTelemetry Collector. The ambitious plan for the second quarter of 2020 involves the graceful deprecation of both the OpenTracing and OpenCensus projects, marking the full transition to OpenTelemetry.
It is crucial to note that OpenTelemetry is not yet deemed production-ready. Its underlying protocol and API specifications are still subject to refinement and change. The current documentation for the OpenTelemetry Collector explicitly recommends the continued use of the OpenCensus Service until a stable release of OpenTelemetry becomes available. This cautious approach underscores the commitment to delivering a robust and reliable standard for the community.
Illustrative Code Comparison
To provide a clearer understanding of the practical differences across these projects, let's examine a real-world scenario: creating a span with an annotation using the JavaScript SDKs of OpenTracing, OpenCensus, and OpenTelemetry, all configured to use Jaeger as the backend.
OpenTracing Implementation
const jaeger = require('jaeger-client');const tracer = jaeger.initTracer({ /* Jaeger options */ });const span = tracer.startSpan('my_span');span.setTag({ 'key': 'value' });// Code to instrument goes here...span.finish();A notable aspect of OpenTracing is that direct importation of an OpenTracing-specific library might not always be necessary, as client libraries often implement its public API directly. However, the OpenTracing library itself provides useful extensions, such as a mock tracer and predefined constants for common tag keys.
OpenCensus Implementation
const tracing = require('@opencensus/nodejs');const { JaegerTraceExporter } = require('@opencensus/exporter-jaeger');const tracer = tracing.start({ samplingRate: 1 }).tracer;tracer.registerSpanEventListener(new JaegerTraceExporter({ /* Jaeger options */}));tracer.startRootSpan({ name: 'my_span' }, rootSpan => { rootSpan.addAttribute('key', 'value') // Code to instrument goes here... rootSpan.end();});In OpenCensus, the core library is central, and Jaeger functions as an exporter that requires explicit registration. Despite API variations, the general workflow for setup, span creation, and attribute management remains largely consistent with OpenTracing.
OpenTelemetry Implementation
const opentelemetry = require('@opentelemetry/core');const { BasicTracer, SimpleSpanProcessor } = require('@opentelemetry/tracing');const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');const tracer = new BasicTracer();tracer.addSpanProcessor(new SimpleSpanProcessor(new JaegerExporter({ /* Jaeger options */})));opentelemetry.initGlobalTracer(tracer);const span = opentelemetry.getTracer().startSpan('my_span');span.setAttribute('key', 'value');// Code to instrument goes here...span.end();OpenTelemetry’s setup might appear slightly more intricate, a direct consequence of its modular design. Similar to OpenCensus, a core library is foundational. However, OpenTelemetry introduces a greater degree of customization through its diverse range of exporters and span processors, including batch processors and console processors for diagnostic output. This modularity unlocks expanded possibilities for tailoring observability solutions.
It is important to reiterate that this OpenTelemetry example utilizes an alpha version, and its API is subject to potential changes.
A Unified Future for Observability
The convergence of OpenTracing and OpenCensus into OpenTelemetry marks a pivotal moment for the cloud-native community. Both antecedent projects boasted robust and active communities, making their decision to unite a testament to a shared vision for simplified and standardized observability. This merger delivers benefits across the board:
- Developers: The burden of choosing between competing standards is lifted, allowing them to focus on crafting code with truly portable telemetry capabilities.
- Operators: They gain a standardized and efficient mechanism for configuring, collecting, interpreting, and exporting tracing data, thanks to the OpenTelemetry Collector.
- Vendors: Providers of application performance monitoring solutions can streamline their efforts, needing to support and test against a single, unified standard, thereby reducing the need to develop proprietary agents or collectors.
For those embarking on new projects today, continuing with OpenTracing or OpenCensus might be the pragmatic choice, especially where stability is a critical requirement. However, with the forthcoming stable releases of OpenTelemetry, developers can leverage compatibility shims, providing a seamless upgrade path to the new API without immediate refactoring, and unlocking access to its advanced features and improvements.
The Promise of OpenTelemetry: A Reporter's Perspective
From a reporter's viewpoint, the emergence of OpenTelemetry is more than just a technical merger; it signifies a monumental step towards maturity and interoperability within the cloud-native ecosystem. For years, the fragmentation caused by competing observability standards created unnecessary complexity and friction for developers and operators alike. This unification effort, driven by the Cloud Native Computing Foundation, demonstrates a collective commitment to simplifying the arduous task of understanding distributed systems. It's a clear signal that the industry is moving past competitive silos towards collaborative solutions that prioritize developer experience and operational efficiency.
The vision of a single, vendor-agnostic standard for tracing, metrics, and eventually logging, is incredibly powerful. It promises to democratize advanced observability, making it accessible and manageable for a broader range of organizations, regardless of their chosen cloud providers or monitoring solutions. The OpenTelemetry Collector, in particular, stands out as a pragmatic solution to real-world data collection and routing challenges, offering flexibility and reducing the operational overhead of managing multiple agents and exporters. While the alpha stage cautions against immediate production use, the trajectory is clear: OpenTelemetry is set to become the definitive standard. Its success will not only foster greater innovation in monitoring tools but also empower engineering teams to build more resilient and performant applications, ultimately accelerating the pace of digital transformation across industries. This is a story of consolidation, collaboration, and a brighter, more transparent future for cloud-native softwar