Logs and Telemetry


Book Description

Build cloud native observability pipelines with minimal footprints and high-performance throughput—all with Fluent Bit, Kubernetes, and your favorite visualization and analytics tools. Logs and Telemetry is an all-practical guide to monitoring both cloud-native and traditional environments with the Fluent Bit observability tool. It takes you from the basics of collecting app logs, all the way to filtering, routing, enriching and transforming logs, metrics, and traces. Inside Logs and Telemetry you’ll learn how to: • Deploy Fluent Bit for telemetry (log, metric, and trace) collection • Configure pipelines to filter, route, and transform data • Integrate Fluent Bit with containers and Kubernetes • Configure Fluent Bit to work with OpenTelemetry, Prometheus, and other open source tech • Monitor applications at scale with minimal footprint • Address challenges in Kubernetes-based ecosystems using Fluent Bit • Utilize Fluent Bit for real-time event analytics to derive new metrics and insights • Develop custom filters, inputs, and outputs for unique or reusable use cases Logs and Telemetry draws on both the input and support of key committers and founders of Fluent Bit, and author Phil Wilkins’ years of experience in DevOps. Inside, you’ll see how you can integrate Fluent Bit with Prometheus, OpenTelemetry, FluentD deployments, and more. Learn how Fluent Bit can not only meet all the demands of cloud-native use cases, but also more traditional deployments as well. Purchase of the print book includes a free eBook in PDF and ePub formats from Manning Publications. About the technology Fluent Bit is a super-fast lightweight observability tool that’s perfect for Kubernetes and containers, as well as traditional IT environments. Fluent Bit makes it a snap to extract meaning from the logs, traces, and other performance metrics generated by your applications and infrastructure. It’s also a great way to route telemetry to analysis tools like Prometheus and Grafana. About the book Logs and Telemetry shows you how to turn systems data into actionable insights using Fluent Bit. You’ll start by learning the pre-built plugins for common use cases and progress to integration with powerful tools like OpenTelemetry and real-time analytical event processing. You’ll use plugins to configure routing, filtering and processing, automate your observability with Lua scripts, and configure Fluent Bit to meet the demands of highly scalable environments. What's inside • Deploy Fluent Bit for telemetry collection • Configure pipelines to filter, route, and transform data • Integrate Fluent Bit with containers and Kubernetes • Monitor applications at scale About the reader For developers, DevOps engineers, and SREs working with observability. About the author Phil Wilkins has spent over 25 years in the software industry from multinationals to software startups. He is the author of Logging in Action. The technical editor on this book was Karthik Gaekwad. Table of Contents Part 1 1 Introduction to Fluent Bit 2 From zero to “Hello, World” Part 2 3 Capturing inputs 4 Getting inputs from containers and Kubernetes 5 Outputting events 6 Parsing to extract more meaning 7 Filtering and transforming events Part 3 8 Stream processors for time series calculations and filtering 9 Building processors and Fluent Bit extension options 10 Building plugins 11 Putting Fluent Bit into action: An enterprise use case Appendix A Installations Appendix B Useful resources Appendix C Comparing Fluent Bit and Fluentd




Software Telemetry


Book Description

Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. Summary In Software Telemetry you will learn how to: Manage toxic telemetry and confidential records Master multi-tenant techniques and transformation processes Update to improve the statistical validity of your metrics and dashboards Make software telemetry emissions easier to parse Build easily-auditable logging systems Prevent and handle accidental data leaks Maintain processes for legal compliance Justify increased spend on telemetry software Software Telemetry teaches you best practices for operating and updating telemetry systems. These vital systems trace, log, and monitor infrastructure by observing and analyzing the events generated by the system. This practical guide is filled with techniques you can apply to any size of organization, with troubleshooting techniques for every eventuality, and methods to ensure your compliance with standards like GDPR. About the technology Take advantage of the data generated by your IT infrastructure! Telemetry systems provide feedback on what’s happening inside your data center and applications, so you can efficiently monitor, maintain, and audit them. This practical book guides you through instrumenting your systems, setting up centralized logging, doing distributed tracing, and other invaluable telemetry techniques. About the book Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. Manage the pillars of observability—logs, metrics, and traces—in an end-to-end telemetry system that integrates with your existing infrastructure. You’ll discover how software telemetry benefits both small startups and legacy enterprises. And at a time when data audits are increasingly common, you’ll appreciate the thorough coverage of legal compliance processes, so there’s no reason to panic when a discovery request arrives. What's inside Multi-tenant techniques and transformation processes Toxic telemetry and confidential records Updates to improve the statistical validity of your metrics and dashboards Revisions that make software telemetry emissions easier to parse About the reader For software developers and infrastructure engineers supporting and building telemetry systems. About the author Jamie Riedesel is a staff engineer at Dropbox with over twenty years of experience in IT. Table of Contents 1 Introduction PART 1 TELEMETRY SYSTEM ARCHITECTURE 2 The Emitting stage: Creating and submitting telemetry 3 The Shipping stage: Moving and storing telemetry 4 The Shipping stage: Unifying diverse telemetry formats 5 The Presentation stage: Displaying telemetry 6 Marking up and enriching telemetry 7 Handling multitenancy PART 2 USE CASES REVISITED: APPLYING ARCHITECTURE CONCEPTS 8 Growing cloud-based startup 9 Nonsoftware business 10 Long-established business IT PART 3 TECHNIQUES FOR HANDLING TELEMETRY 11 Optimizing for regular expressions at scale 12 Standardized logging and event formats 13 Using more nonfile emitting techniques 14 Managing cardinality in telemetry 15 Ensuring telemetry integrity 16 Redacting and reprocessing telemetry 17 Building policies for telemetry retention and aggregation 18 Surviving legal processes




Site Reliability Engineering


Book Description

The overwhelming majority of a software system’s lifespan is spent in use, not in design or implementation. So, why does conventional wisdom insist that software engineers focus primarily on the design and development of large-scale computing systems? In this collection of essays and articles, key members of Google’s Site Reliability Team explain how and why their commitment to the entire lifecycle has enabled the company to successfully build, deploy, monitor, and maintain some of the largest software systems in the world. You’ll learn the principles and practices that enable Google engineers to make systems more scalable, reliable, and efficient—lessons directly applicable to your organization. This book is divided into four sections: Introduction—Learn what site reliability engineering is and why it differs from conventional IT industry practices Principles—Examine the patterns, behaviors, and areas of concern that influence the work of a site reliability engineer (SRE) Practices—Understand the theory and practice of an SRE’s day-to-day work: building and operating large distributed computing systems Management—Explore Google's best practices for training, communication, and meetings that your organization can use




Telemetry Techniques


Book Description




Cloud-Native Observability with OpenTelemetry


Book Description

Leverage OpenTelemetry's API, libraries, tools and the collector to produce and collect telemetry along with using open-source tools to analyze distributed traces, check metrics and logs, and gain insights into application health Key Features • Get to grips with OpenTelemetry, an open-source cloud-native software observability standard • Use vendor-neutral tools to instrument applications to produce better telemetry and improve observability • Understand how telemetry data can be correlated and interpreted to understand distributed systems Book Description Cloud-Native Observability with OpenTelemetry is a guide to helping you look for answers to questions about your applications. This book teaches you how to produce telemetry from your applications using an open standard to retain control of data. OpenTelemetry provides the tools necessary for you to gain visibility into the performance of your services. It allows you to instrument your application code through vendor-neutral APIs, libraries and tools. By reading Cloud-Native Observability with OpenTelemetry, you'll learn about the concepts and signals of OpenTelemetry - traces, metrics, and logs. You'll practice producing telemetry for these signals by configuring and instrumenting a distributed cloud-native application using the OpenTelemetry API. The book also guides you through deploying the collector, as well as telemetry backends necessary to help you understand what to do with the data once it's emitted. You'll look at various examples of how to identify application performance issues through telemetry. By analyzing telemetry, you'll also be able to better understand how an observable application can improve the software development life cycle. By the end of this book, you'll be well-versed with OpenTelemetry, be able to instrument services using the OpenTelemetry API to produce distributed traces, metrics and logs, and more. What you will learn • Understand the core concepts of OpenTelemetry • Explore concepts in distributed tracing, metrics, and logging • Discover the APIs and SDKs necessary to instrument an application using OpenTelemetry • Explore what auto-instrumentation is and how it can help accelerate application instrumentation • Configure and deploy the OpenTelemetry Collector • Get to grips with how different open-source backends can be used to analyze telemetry data • Understand how to correlate telemetry in common scenarios to get to the root cause of a problem Who this book is for This book is for software engineers, library authors, and systems operators looking to better understand their infrastructure, services and applications by leveraging telemetry data like never before. Working knowledge of Python programming is assumed for the example applications that you'll be building and instrumenting using the OpenTelemetry API and SDK. Some familiarity with Go programming, Linux, and Docker is preferable to help you set up additional components in various examples throughout the book.







Modern Distributed Tracing in .NET


Book Description

Instrument .NET apps using OpenTelemetry and explore logs and .NET diagnostic tools to debug, monitor, and analyze the performance of complex systems in the cloud Purchase of the print or Kindle book includes a free PDF eBook Key Features Get a clear understanding of complex systems using .NET and OpenTelemetry Adopt a systematic approach toward performance analysis and debugging Explore instrumentation techniques for common distributed patterns Book Description As distributed systems become more complex and dynamic, their observability needs to grow to aid the development of holistic solutions for performance or usage analysis and debugging. Distributed tracing brings structure, correlation, causation, and consistency to your telemetry, thus allowing you to answer arbitrary questions about your system and creating a foundation for observability vendors to build visualizations and analytics. Modern Distributed Tracing in .NET is your comprehensive guide to observability that focuses on tracing and performance analysis using a combination of telemetry signals and diagnostic tools. You'll begin by learning how to instrument your apps automatically as well as manually in a vendor-neutral way. Next, you'll explore how to produce useful traces and metrics for typical cloud patterns and get insights into your system and investigate functional, configurational, and performance issues. The book is filled with instrumentation examples that help you grasp how to enrich auto-generated telemetry or produce your own to get the level of detail your system needs, along with controlling your costs with sampling, aggregation, and verbosity. By the end of this book, you'll be ready to adopt and leverage tracing and other observability signals and tools and tailor them to your needs as your system evolves. What you will learn Understand the core concepts of distributed tracing and observability Auto-instrument .NET applications with OpenTelemetry Manually instrument common scenarios with traces and metrics Systematically debug issues and analyze the performance Keep performance overhead and telemetry volume under control Adopt and evolve observability in your organization Who this book is for This book is for software developers, architects, and systems operators running .NET services who want to use modern observability tools and standards and take a holistic approach to performance analysis and end-to-end debugging. Software testers and support engineers will also find this book useful. Basic knowledge of the C# programming language and .NET platform is assumed to grasp the examples of manual instrumentation, but it is not necessary.




Learning OpenTelemetry


Book Description

OpenTelemetry is a revolution in observability data. Instead of running multiple uncoordinated pipelines, OpenTelemetry provides users with a single integrated stream of data, providing multiple sources of high-quality telemetry data: tracing, metrics, logs, RUM, eBPF, and more. This practical guide shows you how to set up, operate, and troubleshoot the OpenTelemetry observability system. Authors Austin Parker, head of developer relations at Lightstep and OpenTelemetry Community Maintainer, and Ted Young, cofounder of the OpenTelemetry project, cover every OpenTelemetry component, as well as observability best practices for many popular cloud, platform, and data services such as Kubernetes and AWS Lambda. You'll learn how OpenTelemetry enables OSS libraries and services to provide their own native instrumentation—a first in the industry. Ideal for application developers, OSS maintainers, operators and infrastructure teams, and managers and team leaders, this book guides you through: The principles of modern observability All OpenTelemetry components—and how they fit together A practical approach to instrumenting platforms and applications Methods for installing, operating, and troubleshooting an OpenTelemetry-based observability solution Ways to roll out and maintain end-to-end observability across a large organization How to write and maintain consistent, high-quality instrumentation without a lot of work




Mastering OpenTelemetry and Observability


Book Description

Discover the power of open source observability for your enterprise environment In Mastering Observability and OpenTelemetry: Enhancing Application and Infrastructure Performance and Avoiding Outages, accomplished engineering leader and open source contributor Steve Flanders unlocks the secrets of enterprise application observability with a comprehensive guide to OpenTelemetry (OTel). Explore how OTel transforms observability, providing a robust toolkit for capturing and analyzing telemetry data across your environment. You will learn how OTel delivers unmatched flexibility, extensibility, and vendor neutrality, freeing you from vendor lock-in and enabling data sovereignty and portability. You will also discover: Comprehensive coverage of observability issues and technology: Dive deep into the world of observability and gain a comprehensive understanding of observability fundamentals with practical insights and real-world use cases. Practical guidance: From instrumentation techniques to advanced tracing strategies, gain the skills needed to create highly observable systems. Learn how to deploy and configure OTel, even in challenging brownfield environments, with step-by-step instructions and hands-on exercises. An opportunity for community contributions and communication: Join the OTel community, including end-users, vendors, and cloud providers, and shape the future of observability while connecting with experts and peers. Whether you are a novice or a seasoned professional, Mastering Observability and OpenTelemetry is your roadmap to troubleshooting availability and performance problems by learning to detect anomalies, interpret data, and proactively optimize performance in your enterprise environment. Embark on your journey to observability mastery today!