AWS Glue provides a robust environment for building and managing ETL pipelines, but many data engineers face the challenge of chaining or coordinating multiple workflows. This article explores practical approaches to relate two or more Glue workflows, covering both native features and complementary AWS services.

Why You Might Need Multiple Workflows

In many data engineering projects, you have distinct domains or stages:

• Domain A: Ingesting raw transactional data (e.g., from MySQL or PostgreSQL)
• Domain B: Transforming and enriching data into analytics-ready datasets

Instead of creating one large workflow, you may prefer to split them for modularity, error isolation, and ease of maintenance. The challenge is ensuring that Workflow B runs only after Workflow A finishes successfully.

Option 1: AWS Step Functions (Recommended)

AWS Step Functions is a fully managed service that lets you coordinate multiple AWS services into serverless workflows.

Advantages:

• Native integration with Glue Workflows and Glue Jobs
• Ability to wait, branch, and retry on errors
• Centralized orchestration for complex pipelines

Example Definition:

{
  "StartAt": "RunWorkflowA",
  "States": {
    "RunWorkflowA": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startWorkflowRun",
      "Parameters": {
        "Name": "WorkflowA"
      },
      "Next": "RunWorkflowB"
    },
    "RunWorkflowB": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startWorkflowRun",
      "Parameters": {
        "Name": "WorkflowB"
      },
      "End": true
    }
  }
}

This JSON definition runs WorkflowA, waits for its completion, and then triggers WorkflowB.

Option 2: Triggering a Workflow from a Glue Job

Within a Glue Job (written in Python or PySpark), you can use boto3 to start another workflow programmatically:

import boto3

glue = boto3.client('glue')
response = glue.start_workflow_run(Name='WorkflowB')
print("WorkflowB started:", response['RunId'])

This approach is useful if you need to pass runtime parameters from one workflow to the next.

Option 3: Amazon EventBridge (CloudWatch Events)

You can set up an EventBridge rule to listen for Glue Workflow state changes. For example:

• When WorkflowA transitions to SUCCEEDED
• Trigger a Lambda function that starts WorkflowB

This is a serverless and event-driven solution that decouples workflows.

Option 4: Single Workflow with Conditional Triggers

If your workflows are strongly related, you may not need two workflows at all. Instead, you can create one workflow with multiple triggers and define conditional dependencies.

Best Practices

• Use Step Functions for complex pipelines with multiple branches and error handling.
• Pass Parameters if downstream workflows depend on upstream outputs.
• Monitor with CloudWatch for observability and alerting.
• Version Control your Step Functions definitions and Glue scripts for reproducibility.

Conclusion

AWS Glue workflows are powerful, but their native features are limited when you need cross-workflow coordination. By leveraging Step Functions, EventBridge, or boto3, you can build robust, event-driven data pipelines that are maintainable and scalable.