Best of Observability — September 2024

  1. 1
    Article
    Avatar of last9Last9·2y

    Golang Logging: A Comprehensive Guide for Developers

    Logging in Go is essential for debugging and maintaining application performance. While the standard log package offers basic functionality, third-party libraries like Zerolog and Zap provide advanced features like structured logging and configurable log levels. Implementing best practices, such as avoiding sensitive data in logs and using context-rich messages, can significantly enhance log analysis and troubleshooting. Integrating with observability platforms like ELK can further improve monitoring capabilities in production environments.

  2. 2
    Article
    Avatar of netflixNetflix TechBlog·2y

    Noisy Neighbor Detection with eBPF

    Netflix's Compute and Performance Engineering teams are tackling the 'noisy neighbor' problem in their multi-tenant environment by using eBPF for continuous, low-overhead Linux scheduler instrumentation. This approach enables effective self-serve monitoring of resource-heavy containers causing performance issues. By instrumenting run queue latency and leveraging eBPF hooks like sched_wakeup and sched_switch, they efficiently track performance degradation. The gathered data helps refine CPU isolation strategies and enhance infrastructure observability. This initiative has also inspired the development of tools like bpftop for optimizing eBPF code.

  3. 3
    Article
    Avatar of last9Last9·2y

    Optimizing Prometheus Remote Write Performance: Guide

    Prometheus remote write is pivotal for storing and querying long-term metrics as infrastructure scales. Common performance bottlenecks include high CPU and memory usage, network bandwidth consumption, and delayed metric availability. Optimization strategies focus on queue configuration, reducing cardinality, effective relabeling, network optimization, and choosing appropriate remote storage. Best practices include starting with conservative settings, continuous monitoring, and adjusting configurations based on observed performance.

  4. 4
    Article
    Avatar of last9Last9·2y

    OpenTelemetry Collector: The Complete Guide

    OpenTelemetry provides a vendor-agnostic observability framework suitable for distributed systems. The OpenTelemetry Collector, or otelcol, allows for the receipt, processing, and export of telemetry data in diverse formats. It supports multiple protocols and can replace the need for multiple monitoring agents. Key benefits include vendor neutrality, data transformation, scalability, and cost reduction. Use cases range from microservices monitoring to legacy system integration. Advanced topics cover custom processors, high availability setups, and performance tuning. The guide also compares the Collector with Jaeger and emphasizes best practices for effective deployment.

  5. 5
    Article
    Avatar of devopsDevOps.com·2y

    OpenTelemetry Isn’t the Hero We Need: Here’s Why it’s Failing our Stack

    OpenTelemetry was intended to be a unified standard for collecting and correlating traces, logs, and metrics from distributed systems. However, it has become bloated and inefficient due to corporate influence and feature creep. In contrast, eBPF, operating at the kernel level, provides real-time, low-overhead observability and granular insights without the same drawbacks. While OpenTelemetry offers a big-picture view useful for distributed tracing, eBPF excels in detailed system performance insights. Combining both could provide comprehensive observability.

  6. 6
    Article
    Avatar of last9Last9·2y

    OpenTelemetry Filelog Receiver: Collecting Logs from Kubernetes

    The OpenTelemetry filelog receiver is a valuable tool for log collection in Kubernetes environments. It reads log files and converts them into the OpenTelemetry log format. Setting it up involves deploying the OpenTelemetry Collector in the Kubernetes cluster, configuring it via a YAML file, and applying the configuration. Optimization tips include using batch processors and memory limiters, and troubleshooting advice covers common issues like missing logs and parsing errors. The receiver integrates well with the OpenTelemetry ecosystem and can handle various log sources, including syslog. Advanced parsing techniques and integration with backends like Last9 Levitate are also discussed.

  7. 7
    Article
    Avatar of itnextITNEXT·2y

    An Introduction to the OpenTelemetry Collector

    OpenTelemetry provides open standards for interoperable tools handling telemetry data. The OpenTelemetry Collector, a flexible and extensible deployable binary, acts as a universal translator and pipeline for gathering, processing, and forwarding metrics, traces, and logs. It supports various plugins and can be tailored to different environments, including custom distributions for specific use cases. Deployable on Kubernetes, it can gather telemetry cluster-wide, with specific plugins for Kubernetes entities. The post promises a hands-on tutorial on integrating Kubernetes Cluster Logging with ClickHouse and Grafana using OpenTelemetry.

  8. 8
    Article
    Avatar of last9Last9·2y

    Developer's Guide to Installing OpenTelemetry Collector

    Learn how to install and configure the OpenTelemetry Collector to enhance your observability setup. This guide covers various installation methods including Docker, Kubernetes, binary installation for Linux, and package installation for Linux systems. It also explains the basic configuration needed to set up receivers, processors, exporters, and pipelines, along with troubleshooting tips and advanced topics.

  9. 9
    Article
    Avatar of grafanaGrafana Labs·2y

    OpenTelemetry and vendor neutrality: how to build an observability strategy with maximum flexibility

    OpenTelemetry offers vendor neutrality, allowing you to switch between different observability tools without being locked in by a single vendor. By focusing on decoupling telemetry collection from storage and analysis, OpenTelemetry provides flexibility in building observability strategies. Adopting OpenTelemetry in layers and ensuring reusable instrumentation can maximize your options and reduce migration difficulties.

  10. 10
    Article
    Avatar of communityCommunity Picks·2y

    4 Observability functionality in Spring AI (Milestone 2)

    Spring AI 1.0.0 M2 introduces essential observability features, including metrics and tracing, to help monitor AI-powered applications. Key components like ChatClient, ChatModel, and VectorStore now support these functionalities. Developers can integrate these features using dependencies like Spring Boot Actuator, Micrometer Tracing, and OpenTelemetry. The Digma plugin for IntelliJ aids in analyzing observability data by visualizing metrics and tracing within the development environment.

See all Observability archives