› › Generic Pipelines Using Docker

Generic Pipelines Using Docker

Author : Brandon Atkinson

Publisher : Apress

Page : 149 pages

File Size : 36,14 MB

Release : 2018-12-19

Category : Computers

ISBN : 1484236556

Get eBook

Book Description

Create generic pipelines to reduce your overall DevOps workload and allow your team to deliver faster. This book helps you get up to speed on the pros and cons of generic pipeline methodology, and learn to combine shell scripts and Docker to build generic pipelines. In today’s world of micro-services and agile practices, DevOps teams need to move as fast as feature teams. This can be extremely challenging if you’re creating multiple pipelines per application or tech stack. What if your feature teams could utilize a generic pipeline that could build, test, and deploy any application, regardless of tech stack? What if that pipeline was also cloud and platform agnostic? Too good to be true? Well think again! Generic Pipelines Using Docker explores the principles and implementations that allow you to do just that. You will learn from real-world examples and reusable code. After reading this book you will have the knowledge to build generic pipelines that any team can use. What You'll Learn Explore the pros and cons of generic pipeline methodology Combine shell scripts and Docker to build a generic pipeline Implement a pipeline across CI/CD platforms Build a pipeline that lends itself well to both centralized and federated DevOps teams Construct a modular pipeline with components that can be added, removed, or replaced as needed Who This Book Is For Professionals who use DevOps or are part of a DevOps team, and are seeking ways to streamline their pipelines and drive more deployments while using less code



Data Pipelines with Apache Airflow

Author : Julian de Ruiter

Publisher : Simon and Schuster

Page : 480 pages

File Size : 43,90 MB

Release : 2021-04-05

Category : Computers

ISBN : 1638356831

Get eBook

Book Description

"An Airflow bible. Useful for all kinds of users, from novice to expert." - Rambabu Posa, Sai Aashika Consultancy Data Pipelines with Apache Airflow teaches you how to build and maintain effective data pipelines. A successful pipeline moves data efficiently, minimizing pauses and blockages between tasks, keeping every process along the way operational. Apache Airflow provides a single customizable environment for building and managing data pipelines, eliminating the need for a hodgepodge collection of tools, snowflake code, and homegrown processes. Using real-world scenarios and examples, Data Pipelines with Apache Airflow teaches you how to simplify and automate data pipelines, reduce operational overhead, and smoothly integrate all the technologies in your stack. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology Data pipelines manage the flow of data from initial collection through consolidation, cleaning, analysis, visualization, and more. Apache Airflow provides a single platform you can use to design, implement, monitor, and maintain your pipelines. Its easy-to-use UI, plug-and-play options, and flexible Python scripting make Airflow perfect for any data management task. About the book Data Pipelines with Apache Airflow teaches you how to build and maintain effective data pipelines. You’ll explore the most common usage patterns, including aggregating multiple data sources, connecting to and from data lakes, and cloud deployment. Part reference and part tutorial, this practical guide covers every aspect of the directed acyclic graphs (DAGs) that power Airflow, and how to customize them for your pipeline’s needs. What's inside Build, test, and deploy Airflow pipelines as DAGs Automate moving and transforming data Analyze historical datasets using backfilling Develop custom components Set up Airflow in production environments About the reader For DevOps, data engineers, machine learning engineers, and sysadmins with intermediate Python skills. About the author Bas Harenslak and Julian de Ruiter are data engineers with extensive experience using Airflow to develop pipelines for major companies. Bas is also an Airflow committer. Table of Contents PART 1 - GETTING STARTED 1 Meet Apache Airflow 2 Anatomy of an Airflow DAG 3 Scheduling in Airflow 4 Templating tasks using the Airflow context 5 Defining dependencies between tasks PART 2 - BEYOND THE BASICS 6 Triggering workflows 7 Communicating with external systems 8 Building custom components 9 Testing 10 Running tasks in containers PART 3 - AIRFLOW IN PRACTICE 11 Best practices 12 Operating Airflow in production 13 Securing Airflow 14 Project: Finding the fastest way to get around NYC PART 4 - IN THE CLOUDS 15 Airflow in the clouds 16 Airflow on AWS 17 Airflow on Azure 18 Airflow in GCP



Docker in Action, Second Edition

Author : Jeffrey Nickoloff

Publisher : Simon and Schuster

Page : 481 pages

File Size : 28,40 MB

Release : 2019-10-28

Category : Computers

ISBN : 1638351740

Get eBook

Book Description

Summary Docker in Action, Second Edition teaches you the skills and knowledge you need to create, deploy, and manage applications hosted in Docker containers. This bestseller has been fully updated with new examples, best practices, and a number of entirely new chapters. About the technology The idea behind Docker is simple—package just your application and its dependencies into a lightweight, isolated virtual environment called a container. Applications running inside containers are easy to install, manage, and remove. This simple idea is used in everything from creating safe, portable development environments to streamlining deployment and scaling for microservices. In short, Docker is everywhere. About the book Docker in Action, Second Edition teaches you to create, deploy, and manage applications hosted in Docker containers running on Linux. Fully updated, with four new chapters and revised best practices and examples, this second edition begins with a clear explanation of the Docker model. Then, you go hands-on with packaging applications, testing, installing, running programs securely, and deploying them across a cluster of hosts. With examples showing how Docker benefits the whole dev lifecycle, you’ll discover techniques for everything from dev-and-test machines to full-scale cloud deployments. What's inside Running software in containers Packaging software for deployment Securing and distributing containerized applications About the reader Written for developers with experience working with Linux. About the author Jeff Nickoloff and Stephen Kuenzli have designed, built, deployed, and operated highly available, scalable software systems for nearly 20 years.



Continuous Delivery with Docker and Jenkins

Author : Rafal Leszko

Publisher : Packt Publishing Ltd

Page : 374 pages

File Size : 34,88 MB

Release : 2022-05-04

Category : Computers

ISBN : 1803245301

Get eBook

Book Description

Create a complete continuous delivery process using modern DevOps tools such as Docker, Jenkins, Kubernetes, Ansible, Terraform, and many more Key Features • Build reliable and secure applications using Docker containers • Create a highly available environment to scale Jenkins and your services using Kubernetes • Automate your release process end-to-end Book Description This updated third edition of Continuous Delivery with Docker and Jenkins will explain the advantages of combining Jenkins and Docker to improve the continuous integration and delivery process of app development. You'll start by setting up a Docker server and configuring Jenkins on it. Next, you'll discover steps for building applications and microservices on Dockerfiles and integrating them with Jenkins using continuous delivery processes such as continuous integration, automated acceptance testing, configuration management, and Infrastructure as Code. Moving ahead, you'll learn how to ensure quick application deployment with Docker containers, along with scaling Jenkins using Kubernetes. Later, you'll explore how to deploy applications using Docker images and test them with Jenkins. Toward the concluding chapters, the book will focus on missing parts of the CD pipeline, such as the environments and infrastructure, application versioning, and non-functional testing. By the end of this continuous integration and continuous delivery book, you'll have gained the skills you need to enhance the DevOps workflow by integrating the functionalities of Docker and Jenkins. What you will learn • Grasp Docker fundamentals and dockerize applications for the CD process • Understand how to use Jenkins on-premises and in the cloud • Scale a pool of Docker servers using Kubernetes • Write acceptance tests using Cucumber • Run tests in the Docker ecosystem using Jenkins • Provision your servers and infrastructure using Ansible and Terraform • Publish a built Docker image to a Docker registry • Deploy cycles of Jenkins pipelines using community best practices Who this book is for The book is for DevOps engineers, system administrators, Docker professionals, or anyone who wants to explore the power of working with Docker and Jenkins together. No prior knowledge of DevOps is required to get started.



CI/CD Pipeline with Docker and Jenkins

Author : Sandeep Rawat

Publisher : BPB Publications

Page : 232 pages

File Size : 44,80 MB

Release : 2023-01-12

Category : Computers

ISBN : 935551350X

Get eBook

Book Description

A Practical Guide to Mastering Modern-day Software Development KEY FEATURES ● Leverage the power of Docker for faster deployment and migrations. ● Learn how to create a Continuous Integration pipeline in Jenkins. ● Get familiar with different types of deployments in Continuous Delivery. DESCRIPTION ‘CI/CD Pipeline with Docker and Jenkins’ is the right mix of narrative, concepts, and real-life implementation. The book focuses on the CI/CD maturity journey of a team from a DevOps perspective. The book takes you on a CI/CD journey of a project, starting from identifying the challenges faced by the team and how the team uses CI as a first step to overcome the problems. Moving on, the book explains how the team matures via the Jenkins shared library and Dockerization. The later part of the book covers the next journey, where the DevOps team decides to introduce CD to mature the DevOps practices of the project further. Here, the book explains how the DevOps team learns about the various implementations of CD and applies them via Jenkins. Given that there are multiple technologies to cover, the journey starts from the simplest one and slowly goes on to higher concepts ensuring that you follow the right learning path. To keep things in perspective, a sample project has been included with the book, and all the concepts of CI/CD are implemented in it. You can work with it, test out different scenarios, and can refer to them for your projects. WHAT YOU WILL LEARN ● Design a CI implementation plan in terms of Pre and Post Deployment integration checks. ● Learn how to run your CI/CD jobs in Docker containers. ● Understand how CI and CD work together end to end to achieve modern software delivery goals. ● Design and implement a very comprehensive CI process of any stack on any platform. ● Assess and identify the CD requirements of a project and architect the right CD solution. WHO THIS BOOK IS FOR This book is for current and aspiring tech professionals, students, and anyone who wishes to build a career in DevOps. At the same time, it will also prove essential to existing engineers and project owners looking to streamline their DevOps practices with industry standards, specifically the CI/CD part. TABLE OF CONTENTS 1. Introduction 2. Continuous Integration 3. Introduction to Jenkins 4. CI with Jenkins 5. Introduction to Docker 6. CI with Jenkins and Docker 7. Continuous Deployment 8. Continuous Deployment Using Jenkins



Automating DevOps with GitLab CI/CD Pipelines

Author : Christopher Cowell

Publisher : Packt Publishing Ltd

Page : 348 pages

File Size : 28,79 MB

Release : 2023-02-24

Category : Computers

ISBN : 1803242930

Get eBook

Book Description

Use GitLab CI/CD pipelines for automating and deploying different steps of your software development lifecycle using best practices and troubleshooting methods. Key Features Reap the power of GitLab CI/CD pipelines at every stage of your software development lifecycle Learn how GitLab makes Git easier to use and more powerful when committing and reviewing code Cement your understanding using hands-on tutorials and extensive self-assessment exercises Purchase of the print or Kindle book includes a free eBook in the PDF format Book DescriptionDevelopers and release engineers understand the high stakes involved in building, packaging, and deploying code correctly. Ensuring that your code is functionally correct, fast, and secure is a time-consuming and complex task. Code implementation, development, and deployment can be conducted efficiently using GitLab CI/CD pipelines. Automating DevOps with GitLab CI/CD Pipelines begins with the basics of Git and GitLab, showing how to commit and review code. You’ll learn to set up GitLab Runners for executing and autoscaling CI/CD pipelines and creating and configuring pipelines for many software development lifecycle steps. You'll also discover where to find pipeline results in GitLab, and how to interpret those results. Through the course of the book, you’ll become well-equipped with deploying code to different environments, advancing CI/CD pipeline features such as connecting GitLab to a Kubernetes cluster and using GitLab with Terraform, triggering pipelines and improving pipeline performance and using best practices and troubleshooting tips for uncooperative pipelines. In-text examples, use cases, and self-assessments will reinforce the important CI/CD, GitLab, and Git concepts, and help you prepare for interviews and certification exams related to GitLab. By the end of this book, you'll be able to use GitLab to build CI/CD pipelines that automate all the DevOps steps needed to build and deploy high-quality, secure code.What you will learn Gain insights into the essentials of Git, GitLab, and DevOps Understand how to create, view, and run GitLab CI/CD pipelines Explore how to verify, secure, and deploy code with GitLab CI/CD pipelines Configure and use GitLab Runners to execute CI/CD pipelines Explore advanced GitLab CI/CD pipeline features like DAGs and conditional logic Follow best practices and troubleshooting methods of GitLab CI/CD pipelines Implement end-to-end software development lifecycle workflows using examples Who this book is for This book is for DevOps/DevSecOps engineers, application developers, release engineers, quality assurance engineers, security engineers, SREs, and sysadmins looking to implement fast, secure and automated software development lifecycle tasks using continuous integration and continuous delivery (CI/CD) pipelines in GitLab. Basic knowledge of major stages of the software development life cycle and DevOps processes will be helpful.



Implementing CI/CD Using Azure Pipelines

Author : Piti Champeethong

Publisher : Packt Publishing Ltd

Page : 326 pages

File Size : 50,1 MB

Release : 2023-12-28

Category : Computers

ISBN : 1804618209

Get eBook

Book Description

Leverage Azure Pipelines to build, test, monitor, and deploy CI/CD solutions on Azure, AWS, and Flutter mobile apps while integrating with tools like Jenkins and SonarQube using best practices Key Features Develop automated end-to-end CI/CD solutions with Azure Pipelines Learn how to implement and configure your pipeline using real-world examples and scenarios Gain the skills you need to efficiently develop and deploy your organization’s software Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionContinuous integration and continuous delivery (CI/CD) are ubiquitous concepts in modern development. Azure Pipelines is one of the most popular services that you can utilize for CI/CD, and this book shows you how it works by taking you through the process of building and automating CI/CD systems using Azure Pipelines and YAML, simplifying integration with Azure resources and reducing human error. You’ll begin by getting an overview of Azure Pipelines and why you should use it. Next, the book helps you get to grips with build and release pipelines, and then builds upon this by introducing the extensive power of YAML syntax, which you can use to implement and configure any task you can think of. As you advance, you’ll discover how to integrate Infrastructure as Code tools, such as Terraform, and perform code analysis with SonarQube. In the concluding chapters, you’ll delve into real-life scenarios and hands-on implementation tasks with Microsoft Azure services, AWS, and cross-mobile application with Flutter, Google Firebase, and more. By the end of this book, you’ll be able to design and build CI/CD systems using Azure Pipelines with consummate ease, write code using YAML, and configure any task that comes to mind.What you will learn Create multiple jobs, stages, and tasks on the Azure DevOps portal Use YAML syntax for Node.js, .NET, Docker, and SQL Server tasks Automate microservice applications on Azure Kubernetes Service (AKS) clusters Deploy Docker applications on AWS container services Use SonarQube and Jenkins for security and artifacts Implement CI/CD on Flutter-based mobile applications Utilize Azure Key Vault secrets in Azure Pipelines Build a Node.js application in Azure Container Instances Who this book is for This book is for DevOps engineers, release engineers, SREs, application developers, and sysadmins looking to manage CI/CD using Azure Pipelines with the help of real-world use cases. A clear understanding of cloud computing services on Azure and AWS, DevOps, and CI/CD concepts, along with knowledge of building and deploying web and mobile applications automatically on cloud is assumed.



Practical Applications of Computational Biology and Bioinformatics, 17th International Conference (PACBB 2023)

Author : Miguel Rocha

Publisher : Springer Nature

Page : 113 pages

File Size : 32,19 MB

Release : 2023-07-10

Category : Technology & Engineering

ISBN : 3031380797

Get eBook

Book Description

This book aims to promote the interaction among the scientific community to discuss applications of CS/AI with an interdisciplinary character, exploring the interactions between sub-areas of CS/AI, bioinformatics, chemoinformatics, and systems biology. The success of bioinformatics in recent years has been prompted by research in molecular biology and molecular medicine in several initiatives. This year’s technical program presents both high quality and diversity, with contributions in well-established and evolving areas of research. The PACBB’23 technical program has selected 9 full papers in the main track and, as in past editions, it will be special issues in ranked journals. This symposium is organized by the LASI and Centro Algoritmi of the University of Minho (Portugal). The authors would like to thank all the contributing authors, the members of the program committee, national associations (AEPIA, APPIA), and the sponsors (AIR Institute).



Jenkins Administrator's Guide

Author : Calvin Sangbin Park

Publisher : Packt Publishing Ltd

Page : 437 pages

File Size : 34,8 MB

Release : 2021-12-24

Category : Computers

ISBN : 183882829X

Get eBook

Book Description

Build and manage a production Jenkins instance, complete with CI/CD pipelines using GitHub and Docker Hub, Jenkins Configuration as Code, Shared Libraries, Script Security, and optimization guides Key FeaturesSet up production-grade Jenkins and CI/CD pipelines with GitHub and Docker Hub integrationsManage, protect, and upgrade a production Jenkins instance regardless of its size and the number of usersScale a Jenkins instance using advanced optimization tips, tricks, and best practicesBook Description Jenkins is a renowned name among build and release CI/CD DevOps engineers because of its usefulness in automating builds, releases, and even operations. Despite its capabilities and popularity, it's not easy to scale Jenkins in a production environment. Jenkins Administrator's Guide will not only teach you how to set up a production-grade Jenkins instance from scratch, but also cover management and scaling strategies. This book will guide you through the steps for setting up a Jenkins instance on AWS and inside a corporate firewall, while discussing design choices and configuration options, such as TLS termination points and security policies. You'll create CI/CD pipelines that are triggered through GitHub pull request events, and also understand the various Jenkinsfile syntax types to help you develop a build and release process unique to your requirements. For readers who are new to Amazon Web Services, the book has a dedicated chapter on AWS with screenshots. You'll also get to grips with Jenkins Configuration as Code, disaster recovery, upgrading plans, removing bottlenecks, and more to help you manage and scale your Jenkins instance. By the end of this book, you'll not only have a production-grade Jenkins instance with CI/CD pipelines in place, but also knowledge of best practices by industry experts. What you will learnSet up a production-grade Jenkins instance on AWS and on-premisesCreate continuous integration and continuous delivery (CI/CD) pipelines triggered by GitHub pull request eventsUse Jenkins Configuration as Code to codify a Jenkins setupBackup and restore configurations and plan for disaster recoveryPlan, communicate, execute, and roll back upgrade scenariosIdentify and remove common bottlenecks in scaling JenkinsUse Shared Libraries to develop helper functions and create new DSLsWho this book is for This book is for both new Jenkins administrators and advanced users who want to optimize and scale Jenkins. Jenkins beginners can follow the step-by-step directions, while advanced readers can join in-depth discussions on Script Security, removing bottlenecks, and other interesting topics. Build and release CI/CD DevOps engineers of all levels will also find new and useful information to help them run a production-grade Jenkins instance following industry best practices.



Kubeflow for Machine Learning

Author : Trevor Grant

Publisher : "O'Reilly Media, Inc."

Page : 264 pages

File Size : 18,1 MB

Release : 2020-10-13

Category : Computers

ISBN : 1492050075

Get eBook

Book Description

If you're training a machine learning model but aren't sure how to put it into production, this book will get you there. Kubeflow provides a collection of cloud native tools for different stages of a model's lifecycle, from data exploration, feature preparation, and model training to model serving. This guide helps data scientists build production-grade machine learning implementations with Kubeflow and shows data engineers how to make models scalable and reliable. Using examples throughout the book, authors Holden Karau, Trevor Grant, Ilan Filonenko, Richard Liu, and Boris Lublinsky explain how to use Kubeflow to train and serve your machine learning models on top of Kubernetes in the cloud or in a development environment on-premises. Understand Kubeflow's design, core components, and the problems it solves Understand the differences between Kubeflow on different cluster types Train models using Kubeflow with popular tools including Scikit-learn, TensorFlow, and Apache Spark Keep your model up to date with Kubeflow Pipelines Understand how to capture model training metadata Explore how to extend Kubeflow with additional open source tools Use hyperparameter tuning for training Learn how to serve your model in production