Orchestration with AWS Glue

Difficulty: Intermediate

Duration: 2 minutes and 8 seconds

Students: 5

Rating: 5/5

This lesson is going to explain the different ways you can orchestrate data pipelines that contain AWS Glue components.

Learning Objectives

By the end of this lesson, you will have a greater understanding of which orchestration services you can use for AWS Glue and when to select each option. The services we’ll discuss are:

AWS Glue Triggers
AWS Glue Workflows
AWS Step Functions
And Amazon Managed Workflows for Apache Airflow

Intended Audience

This lesson has been created for those who are interested in orchestrating their ETL workflows in AWS. This may include those in roles such as data engineers, software engineer, and more.

Prerequisites

To get the most out of this lesson, you should have a basic understanding of AWS Glue and common AWS services such as AWS Lambda, Amazon S3, Amazon CloudWatch, and Amazon SNS. It will also be helpful to have a general understanding of ETL processes.

Covered Topics