Back to Catalog

Building ETL and Data Pipelines with Bash, Airflow and Kafka

Learn on

edX logo
IntermediateCourse

Learn how to design and automate data pipelines and ETL processes to create and manage data workflows. Build data pipelines using Bash shell-scripting, Apache Airflow and Apache Kafka. Pre-requisite skills: datasets, SQL, relational databases, Bash shell scripts.

Language

  • English

Topic

  • Database

Skills You Will Learn

  • Data Engineering, Apache Airflow, Apache Kafka, Data Pipeline, Extract Transform And Load (ETL)

Offered By

  • IBMSkillsNetwork

Estimated Effort

  • 5 weeks

Platform

  • edX

Last Update

  • December 17, 2024
About this Course
Well-designed and automated data pipelines and ETL processes are the foundation of a successful Business Intelligence platform. Defining your data workflows, pipelines and processes early in the platform design ensures the right raw data is collected, transformed and loaded into desired storage layers and available for processing and analysis as and when required.


This course is designed to provide you the critical knowledge and skills needed by Data Engineers and Data Warehousing specialists to create and manage ETL, ELT, and data pipeline processes.


Upon completing this course you’ll gain a solid understanding of Extract, Transform, Load (ETL), and Extract, Load, and Transform (ELT) processes; practice extracting data, transforming data, and loading transformed data into a staging area; create an ETL data pipeline using Bash shell-scripting, build a batch ETL workflow using Apache Airflow and build a streaming data pipeline using Apache Kafka.


You’ll gain hands-on experience with practice labs throughout the course and work on a real-world inspired project to build data pipelines using several technologies that can be added to your portfolio and demonstrate your ability to perform as a Data Engineer.


This course pre-requisites that you have prior skills to work with datasets, SQL, relational databases, and Bash shell scripts.

Instructors

Rav Ahuja

Global Program Director, IBM Skills Network

Rav Ahuja is a Global Program Director at IBM. He leads growth strategy, curriculum creation, and partner programs for the IBM Skills Network. Rav co-founded Cognitive Class, an IBM led initiative to democratize skills for in demand technologies. He is based out of the IBM Canada Lab in Toronto and specializes in instructional solutions for AI, Data, Software Engineering and Cloud. Rav presents at events worldwide and has authored numerous papers, articles, books and courses on subjects in managing and analyzing data. Rav holds B. Eng. from McGill University and MBA from University of Western Ontario.

Read more