› › Apache Iceberg The Definitive Guide

Apache Iceberg: The Definitive Guide

Author : Tomer Shiran

Publisher : "O'Reilly Media, Inc."

Page : 352 pages

File Size : 20,45 MB

Release : 2024-05-02

Category : Computers

ISBN : 1098148584

Get eBook

Book Description

Traditional data architecture patterns are severely limited. To use these patterns, you have to ETL data into each tool—a cost-prohibitive process for making warehouse features available to all of your data. The lack of flexibility with these patterns requires you to lock into a set of priority tools and formats, which creates data silos and data drift. This practical book shows you a better way. Apache Iceberg provides the capabilities, performance, scalability, and savings that fulfill the promise of an open data lakehouse. By following the lessons in this book, you'll be able to achieve interactive, batch, machine learning, and streaming analytics with this high-performance open source format. Authors Tomer Shiran, Jason Hughes, and Alex Merced from Dremio show you how to get started with Iceberg. With this book, you'll learn: The architecture of Apache Iceberg tables What happens under the hood when you perform operations on Iceberg tables How to further optimize Apache Iceberg tables for maximum performance How to use Iceberg with popular data engines such as Apache Spark, Apache Flink, and Dremio How Apache Iceberg can be used in streaming and batch ingestion Discover why Apache Iceberg is a foundational technology for implementing an open data lakehouse.



Apache Iceberg: The Definitive Guide

Author : Tomer Shiran

Publisher : O'Reilly Media

Page : 0 pages

File Size : 15,39 MB

Release : 2024-02-29

Category :

ISBN : 9781098148621

Get eBook

Book Description

Traditional data architecture patterns are severely limited. To use these patterns, you have to ETL data into each tool--a cost-prohibitive process for making warehouse features available to all of your data. This lack of flexibility forces you to adjust your workflow to the tool your data is locked in, which creates data silos and data drift. This book shows you a better way. Apache Iceberg provides the capabilities, performance, scalability, and savings that fulfill the promise of an open data lakehouse. By following the lessons in this book, you'll be able to achieve interactive, batch, machine learning, and streaming analytics with this lakehouse. Authors Tomer Shiran, Jason Hughes, Alex Merced, and Dipankar Mazumdar from Dremio guide you through the process. With this book, you'll learn: The architecture of Apache Iceberg tables What happens under the hood when you perform operations on Iceberg tables How to further optimize Apache Iceberg tables for maximum performance How to use Apache Iceberg with popular data engines such as Apache Spark, Apache Flink, and Dremio Sonar How Apache Iceberg can be used in streaming and batch ingestion Discover why Apache Iceberg is a foundational technology for implementing an open data lakehouse.



The Definitive Guide to Data Integration

Author : Pierre-Yves BONNEFOY

Publisher : Packt Publishing Ltd

Page : 490 pages

File Size : 25,93 MB

Release : 2024-03-29

Category : Computers

ISBN : 1837634777

Get eBook

Book Description

Learn the essentials of data integration with this comprehensive guide, covering everything from sources to solutions, and discover the key to making the most of your data stack Key Features Learn how to leverage modern data stack tools and technologies for effective data integration Design and implement data integration solutions with practical advice and best practices Focus on modern technologies such as cloud-based architectures, real-time data processing, and open-source tools and technologies Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionThe Definitive Guide to Data Integration is an indispensable resource for navigating the complexities of modern data integration. Focusing on the latest tools, techniques, and best practices, this guide helps you master data integration and unleash the full potential of your data. This comprehensive guide begins by examining the challenges and key concepts of data integration, such as managing huge volumes of data and dealing with the different data types. You’ll gain a deep understanding of the modern data stack and its architecture, as well as the pivotal role of open-source technologies in shaping the data landscape. Delving into the layers of the modern data stack, you’ll cover data sources, types, storage, integration techniques, transformation, and processing. The book also offers insights into data exposition and APIs, ingestion and storage strategies, data preparation and analysis, workflow management, monitoring, data quality, and governance. Packed with practical use cases, real-world examples, and a glimpse into the future of data integration, The Definitive Guide to Data Integration is an essential resource for data eclectics. By the end of this book, you’ll have the gained the knowledge and skills needed to optimize your data usage and excel in the ever-evolving world of data.What you will learn Discover the evolving architecture and technologies shaping data integration Process large data volumes efficiently with data warehousing Tackle the complexities of integrating large datasets from diverse sources Harness the power of data warehousing for efficient data storage and processing Design and optimize effective data integration solutions Explore data governance principles and compliance requirements Who this book is for This book is perfect for data engineers, data architects, data analysts, and IT professionals looking to gain a comprehensive understanding of data integration in the modern era. Whether you’re a beginner or an experienced professional enhancing your knowledge of the modern data stack, this definitive guide will help you navigate the data integration landscape.



Delta Lake: The Definitive Guide

Author : Denny Lee

Publisher : "O'Reilly Media, Inc."

Page : 383 pages

File Size : 25,84 MB

Release : 2024-10-30

Category : Computers

ISBN : 1098151917

Get eBook

Book Description

Ready to simplify the process of building data lakehouses and data pipelines at scale? In this practical guide, learn how Delta Lake is helping data engineers, data scientists, and data analysts overcome key data reliability challenges with modern data engineering and management techniques. Authors Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu (with contributions from Delta Lake maintainer R. Tyler Croy) share expert insights on all things Delta Lake--including how to run batch and streaming jobs concurrently and accelerate the usability of your data. You'll also uncover how ACID transactions bring reliability to data lakehouses at scale. This book helps you: Understand key data reliability challenges and how Delta Lake solves them Explain the critical role of Delta transaction logs as a single source of truth Learn the Delta Lake ecosystem with technologies like Apache Flink, Kafka, and Trino Architect data lakehouses with the medallion architecture Optimize Delta Lake performance with features like deletion vectors and liquid clustering



Apache Iceberg: The Definitive Guide

Author : Tomer Shiran

Publisher : "O'Reilly Media, Inc."

Page : 344 pages

File Size : 41,21 MB

Release : 2024-05-02

Category : Computers

ISBN : 1098148592

Get eBook

Book Description

Traditional data architecture patterns are severely limited. To use these patterns, you have to ETL data into each tool—a cost-prohibitive process for making warehouse features available to all of your data. The lack of flexibility with these patterns requires you to lock into a set of priority tools and formats, which creates data silos and data drift. This practical book shows you a better way. Apache Iceberg provides the capabilities, performance, scalability, and savings that fulfill the promise of an open data lakehouse. By following the lessons in this book, you'll be able to achieve interactive, batch, machine learning, and streaming analytics with this high-performance open source format. Authors Tomer Shiran, Jason Hughes, and Alex Merced from Dremio show you how to get started with Iceberg. With this book, you'll learn: The architecture of Apache Iceberg tables What happens under the hood when you perform operations on Iceberg tables How to further optimize Iceberg tables for maximum performance How to use Iceberg with popular data engines such as Apache Spark, Apache Flink, and Dremio Discover why Apache Iceberg is a foundational technology for implementing an open data lakehouse.



Snowflake: The Definitive Guide

Author : Joyce Kay Avila

Publisher : "O'Reilly Media, Inc."

Page : 468 pages

File Size : 19,8 MB

Release : 2022-08-11

Category : Computers

ISBN : 1098103793

Get eBook

Book Description

Snowflake's ability to eliminate data silos and run workloads from a single platform creates opportunities to democratize data analytics, allowing users at all levels within an organization to make data-driven decisions. Whether you're an IT professional working in data warehousing or data science, a business analyst or technical manager, or an aspiring data professional wanting to get more hands-on experience with the Snowflake platform, this book is for you. You'll learn how Snowflake users can build modern integrated data applications and develop new revenue streams based on data. Using hands-on SQL examples, you'll also discover how the Snowflake Data Cloud helps you accelerate data science by avoiding replatforming or migrating data unnecessarily. You'll be able to: Efficiently capture, store, and process large amounts of data at an amazing speed Ingest and transform real-time data feeds in both structured and semistructured formats and deliver meaningful data insights within minutes Use Snowflake Time Travel and zero-copy cloning to produce a sensible data recovery strategy that balances system resilience with ongoing storage costs Securely share data and reduce or eliminate data integration costs by accessing ready-to-query datasets available in the Snowflake Marketplace



Spark: The Definitive Guide

Author : Bill Chambers

Publisher : "O'Reilly Media, Inc."

Page : 594 pages

File Size : 31,61 MB

Release : 2018-02-08

Category : Computers

ISBN : 1491912294

Get eBook

Book Description

Learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the open-source cluster-computing framework. With an emphasis on improvements and new features in Spark 2.0, authors Bill Chambers and Matei Zaharia break down Spark topics into distinct sections, each with unique goals. Youâ??ll explore the basic operations and common functions of Sparkâ??s structured APIs, as well as Structured Streaming, a new high-level API for building end-to-end streaming applications. Developers and system administrators will learn the fundamentals of monitoring, tuning, and debugging Spark, and explore machine learning techniques and scenarios for employing MLlib, Sparkâ??s scalable machine-learning library. Get a gentle overview of big data and Spark Learn about DataFrames, SQL, and Datasetsâ??Sparkâ??s core APIsâ??through worked examples Dive into Sparkâ??s low-level APIs, RDDs, and execution of SQL and DataFrames Understand how Spark runs on a cluster Debug, monitor, and tune Spark clusters and applications Learn the power of Structured Streaming, Sparkâ??s stream-processing engine Learn how you can apply MLlib to a variety of problems, including classification or recommendation



Trino: The Definitive Guide

Author : Matt Fuller

Publisher : "O'Reilly Media, Inc."

Page : 322 pages

File Size : 44,78 MB

Release : 2022-10-03

Category : Computers

ISBN : 1098137205

Get eBook

Book Description

Perform fast interactive analytics against different data sources using the Trino high-performance distributed SQL query engine. In the second edition of this practical guide, you'll learn how to conduct analytics on data where it lives, whether it's a data lake using Hive, a modern lakehouse with Iceberg or Delta Lake, a different system like Cassandra, Kafka, or SingleStore, or a relational database like PostgreSQL or Oracle. Analysts, software engineers, and production engineers learn how to manage, use, and even develop with Trino and make it a critical part of their data platform. Authors Matt Fuller, Manfred Moser, and Martin Traverso show you how a single Trino query can combine data from multiple sources to allow for analytics across your entire organization. Explore Trino's use cases, and learn about tools that help you connect to Trino for querying and processing huge amounts of data Learn Trino's internal workings, including how to connect to and query data sources with support for SQL statements, operators, functions, and more Deploy and secure Trino at scale, monitor workloads, tune queries, and connect more applications Learn how other organizations apply Trino successfully



Amazon DynamoDB - The Definitive Guide

Author : Aman Dhingra

Publisher : Packt Publishing Ltd

Page : 415 pages

File Size : 44,34 MB

Release : 2024-08-30

Category : Computers

ISBN : 1803248327

Get eBook

Book Description

Harness the potential and scalability of DynamoDB to effortlessly construct resilient, low-latency databases Key Features Discover how DynamoDB works behind the scenes to make the most of its features Learn how to keep latency and costs minimal even when scaling up Integrate DynamoDB with other AWS services to create a full data analytics system Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionThis book will help you master Amazon DynamoDB, the fully managed, serverless, NoSQL database service designed for high performance at any scale. Authored by Aman Dhingra, senior DynamoDB specialist solutions architect at AWS, and Mike Mackay, former senior NoSQL specialist solutions architect at AWS, this guide draws on their expertise to equip you with the knowledge and skills needed to harness DynamoDB's full potential. This book not only introduces you to DynamoDB's core features and real-world applications, but also provides in-depth guidance on transitioning from traditional relational databases to the NoSQL world. You'll learn essential data modeling techniques, such as vertical partitioning, and explore the nuances of DynamoDB's indexing capabilities, capacity modes, and consistency models. The chapters also help you gain a solid understanding of advanced topics such as enhanced analytical patterns, implementing caching with DynamoDB Accelerator (DAX), and integrating DynamoDB with other AWS services to optimize your data strategies. By the end of this book, you’ll be able to design, build, and deliver low-latency, high-throughput DynamoDB solutions, driving new levels of efficiency and performance for your applications.What you will learn Master key-value data modeling in DynamoDB for efficiency Transition from RDBMSs to NoSQL with optimized strategies Implement read consistency and ACID transactions effectively Explore vertical partitioning for specific data access patterns Optimize data retrieval using secondary indexes in DynamoDB Manage capacity modes, backup strategies, and core components Enhance DynamoDB with caching, analytics, and global tables Evaluate and design your DynamoDB migration strategy Who this book is for This book is for software architects designing scalable systems, developers optimizing performance with DynamoDB, and engineering managers guiding decision-making. Data engineers will learn to integrate DynamoDB into workflows, while product owners will explore its innovative capabilities. DBAs transitioning to NoSQL will find valuable insights on DynamoDB and RDBMS integration. Basic knowledge of software engineering, Python, and cloud computing is helpful. Hands-on AWS or DynamoDB experience is beneficial but not required.



Practical Lakehouse Architecture

Author : Gaurav Ashok Thalpati

Publisher : "O'Reilly Media, Inc."

Page : 286 pages

File Size : 50,16 MB

Release : 2024-07-24

Category : Computers

ISBN : 1098152980

Get eBook

Book Description

This concise yet comprehensive guide explains how to adopt a data lakehouse architecture to implement modern data platforms. It reviews the design considerations, challenges, and best practices for implementing a lakehouse and provides key insights into the ways that using a lakehouse can impact your data platform, from managing structured and unstructured data and supporting BI and AI/ML use cases to enabling more rigorous data governance and security measures. Practical Lakehouse Architecture shows you how to: Understand key lakehouse concepts and features like transaction support, time travel, and schema evolution Understand the differences between traditional and lakehouse data architectures Differentiate between various file formats and table formats Design lakehouse architecture layers for storage, compute, metadata management, and data consumption Implement data governance and data security within the platform Evaluate technologies and decide on the best technology stack to implement the lakehouse for your use case Make critical design decisions and address practical challenges to build a future-ready data platform Start your lakehouse implementation journey and migrate data from existing systems to the lakehouse