Introducing Apache Airflow 3 on Amazon MWAA: New options and capabilities

As we speak, Amazon Net Providers (AWS) introduced the final availability of Apache Airflow 3 on Amazon Managed Workflows for Apache Airflow (Amazon MWAA). This launch transforms how organizations use Apache Airflow to orchestrate knowledge pipelines and enterprise processes within the cloud, bringing enhanced safety, improved efficiency, and trendy workflow orchestration capabilities to Amazon MWAA clients.

Amazon MWAA introduces Airflow 3 options that modernize workflow administration for AWS clients. Following the April 2025 launch of Airflow 3 by the Apache group, AWS has included these capabilities into Amazon MWAA. Airflow now incorporates a utterly redesigned, intuitive UI that simplifies workflow orchestration for customers throughout expertise ranges. With the Process Execution Interface (Process API), duties can run each inside Airflow and as standalone Python scripts, enhancing code portability and testing. Scheduler-managed Backfill strikes operations from the CLI to the scheduler, offering centralized management and visibility by way of the Airflow UI. CLI safety enhancements change direct database entry with API calls, sustaining constant safety throughout interfaces. Airflow now helps event-driven workflows, enabling triggers from AWS providers and exterior sources. Amazon MWAA additionally provides help for Python 3.12, bringing the most recent language capabilities to workflow improvement.

This publish explores the options of Airflow 3 on Amazon MWAA and descriptions enhancements that enhance your workflow orchestration capabilities. The service maintains the Amazon MWAA pay-as-you-go pricing mannequin with no upfront commitments. You’ll be able to start instantly by visiting the Amazon MWAA console, launching new Apache Airflow environments by way of the AWS Administration Console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, or AWS SDK inside minutes.

Architectural developments in Airflow 3 on Amazon MWAA

Airflow 3 on Amazon MWAA introduces vital architectural enhancements that improve safety, efficiency, and adaptability. These developments create a extra sturdy basis for workflow orchestration whereas sustaining backward compatibility with current workflows.

Enhanced safety

Amazon MWAA with Airflow 3 adjustments the safety mannequin by making element isolation a regular apply reasonably than elective. In Airflow 2, the DAG processor (the element that parses and processes DAG recordsdata) runs throughout the scheduler course of by default, however can optionally be separated into its personal course of for higher scalability and safety isolation. Airflow 3 makes this separation commonplace, sustaining constant safety practices throughout deployments.

API server and Process API

Constructing on this safety basis, a brand new API server element is launched in Amazon MWAA with Airflow 3, which serves as an middleman between job cases and the Airflow metadata database. This modification improves your workflows’ safety posture by minimizing direct entry to the Airflow metadata database from duties. Duties now function with least privilege database entry, decreasing the danger of 1 job affecting others and enhancing total system stability by way of fewer direct database connections.

The standardized communication by way of well-defined API endpoints creates a basis for safer, scalable, and versatile workflow orchestration. The Process Execution Interface (Process API) helps duties run each inside Airflow and as standalone Python scripts, enhancing code portability and testing capabilities.

From data-aware to event-driven scheduling

Airflow’s evolution towards event-driven scheduling started with the introduction of data-aware scheduling in Airflow 2.4, so DAGs could possibly be triggered based mostly on knowledge availability reasonably than time schedules alone. Amazon MWAA with Airflow 3 builds on this basis by way of a transition that features the renaming of datasets to belongings and introduces superior capabilities, together with asset partitions, exterior occasion integration, and asset-centric workflow design.

The transition from datasets to belongings represents greater than a easy rename. A knowledge asset is a group of logically associated knowledge that may symbolize numerous knowledge merchandise, together with database tables, continued ML fashions, embedded dashboards, or directories containing recordsdata.

Amazon MWAA with Airflow 3 introduces a brand new asset-centric syntax that represents an essential shift in how workflows may be designed. The @asset decorator helps builders put knowledge belongings on the heart of their workflow design, creating extra intuitive asset-driven pipelines.

The next code is an instance of asset-aware DAG scheduling:

from airflow.sdk import DAG, Asset
from airflow.suppliers.commonplace.operators.python import PythonOperator

# Outline the asset
customer_data_asset = Asset(identify="customer_data", uri="s3://my-bucket/customer-data.csv")

def process_customer_data():
    """Course of buyer knowledge..."""
    # Implementation right here

# Create the DAG and job
with DAG(dag_id="process_customer_data", schedule="@day by day"):
    PythonOperator(
        task_id="process_data", 
        shops=[customer_data_asset], 
        python_callable=process_customer_data
    )

The next code reveals an asset-centric strategy with the @asset decorator:

from airflow.sdk import asset

@asset(uri="s3://my-bucket/customer-data.csv", schedule="@day by day")
def customer_data():
    """Course of buyer knowledge..."""
    # Implementation right here

The @asset decorator routinely creates an asset with the operate identify, a DAG with the identical identifier, and a job that produces the asset. This reduces code complexity and facilitates computerized DAG creation, the place every asset turns into a self-contained workflow unit.

Exterior event-driven scheduling with Asset Watchers

A big development in Amazon MWAA with Airflow 3 is the introduction of Asset Watchers, which assist Airflow react to occasions taking place exterior of the Airflow system itself. Whereas earlier variations supported inner cross-DAG dependencies, Asset Watchers lengthen this functionality to exterior knowledge techniques and message queues by way of the AssetWatcher class.

Amazon MWAA with Airflow 3 consists of help for Amazon Easy Queue Service (Amazon SQS) by way of Asset Watchers. This enables your workflows to be triggered by exterior messages and facilitates extra event-driven scheduling. Airflow now helps event-driven workflows, enabling triggers from AWS providers and exterior sources. Asset Watchers monitor exterior techniques asynchronously and set off workflow execution when particular occasions happen, enabling workflows to answer enterprise occasions, knowledge updates, or system notifications with out the overhead of conventional sensor-based polling mechanisms.

Fashionable React-based UI

Amazon MWAA with Airflow 3 incorporates a utterly redesigned, intuitive UI constructed with React and FastAPI that simplifies workflow orchestration for customers throughout expertise ranges. The brand new interface gives extra intuitive navigation and workflow visualization, with an enhanced grid view that gives higher visibility into job standing and historical past. Customers will recognize the addition of darkish mode help, which reduces eye pressure throughout prolonged use, and the general sooner efficiency that’s particularly noticeable when working with giant DAGs.

The brand new UI maintains acquainted workflows whereas offering a extra trendy and environment friendly expertise for DAG administration and monitoring, making day by day operations extra productive for each builders and operators. The legacy UI has been utterly eliminated, providing a cleaner, extra constant expertise throughout the system. The inspiration for the brand new UI is constructed on REST APIs and a set of inner APIs for UI operations, each of which at the moment are based mostly on FastAPI, making a extra cohesive and safe structure for each programmatic entry and UI operations.

Scheduler optimizations

Amazon MWAA with Airflow 3’s enhanced scheduler delivers efficiency enhancements for job execution and workflow administration. The redesigned scheduling engine processes duties extra effectively, decreasing the time between job submissions and executions. This optimization advantages knowledge pipeline operations that require fast job processing and well timed workflow completion.

The scheduler now manages computing assets extra successfully, enabling secure efficiency whilst workloads scale. When operating a number of DAGs concurrently, the improved useful resource allocation system helps stop bottlenecks and maintains constant execution speeds. This development is especially helpful for organizations operating advanced workflows with various useful resource necessities. The brand new scheduler additionally handles concurrent operations with elevated precision, so groups can run a number of DAG cases concurrently whereas sustaining system stability and predictable efficiency.

Enhanced scheduler backfill operations

Scheduler-managed backfill (the method of operating DAGs for historic dates) strikes operations from the CLI to the scheduler, offering centralized management and visibility by way of the Airflow UI. Amazon MWAA with Airflow 3 delivers essential upgrades to the scheduler’s backfill capabilities, serving to knowledge groups course of historic knowledge extra effectively. The backfill course of has been optimized for higher efficiency, decreasing the database load throughout these operations and ensuring backfills may be accomplished extra rapidly, minimizing the affect on close to real-time workflow execution.

Amazon MWAA with Airflow 3 additionally improves the administration of backfill operations, with the scheduler offering higher isolation between backfill jobs and supporting extra environment friendly processing of historic datasets. Operators now have higher monitoring instruments to trace the progress and standing of their backfill jobs, leading to more practical administration of those vital knowledge processing duties.

Developer-focused enhancements

Airflow 3 on Amazon MWAA delivers a number of enhancements designed to enhance the developer expertise, from simplified job definition to raised workflow administration capabilities.

Process SDK

The Process SDK gives a extra intuitive technique to outline duties and DAGs:

# Instance utilizing the Process SDK
from airflow.sdk import dag, job
from datetime import datetime

@dag(
    start_date=datetime(2023, 1, 1),
    schedule="@day by day",
    catchup=False
)
def modern_etl_workflow():
    
    @job
    def extract():
        # Extract knowledge from supply
        return {"knowledge": [1, 2, 3, 4, 5]}
    
    @job
    def rework(input_data):
        # Rework the info
        return [x * 10 for x in input_data]
    
    @job
    def load(transformed_data):
        # Load knowledge to vacation spot
        print(f"Loading knowledge: {transformed_data}")
    
    # Outline the workflow
    extracted_data = extract()
    transformed_data = rework(extracted_data["data"])
    load(transformed_data)

# Instantiate the DAG
etl_dag = modern_etl_workflow()

This strategy provides extra intuitive knowledge move between duties, higher built-in improvement surroundings (IDE) help with improved kind hinting, and extra simple unit testing of job logic. The result’s cleaner, extra maintainable code that higher represents the precise knowledge move of your pipelines. Groups adopting this sample typically discover their DAGs change into extra readable and less complicated to take care of over time, particularly as workflows develop in complexity.

DAG versioning

Amazon MWAA with Airflow 3 consists of fundamental DAG versioning capabilities that come by default with Airflow 3. Every time a DAG is modified and deployed, Airflow serializes and shops the DAG definition to protect historical past. This computerized model monitoring minimizes the necessity for handbook record-keeping and ensures each modification is documented.

By means of the Airflow UI, groups can entry and assessment the historical past of their DAGs. This visible illustration reveals model numbers (v1, v2, v3, and many others.) and helps groups perceive how their workflows have developed over time.

The DAG versioning supported in Amazon MWAA gives the aptitude to see completely different DAG variations that had been run within the Airflow UI, providing improved workflow visibility and enhanced collaboration for knowledge engineering groups managing advanced, evolving knowledge pipelines.

Python 3.12 help

Amazon MWAA provides help for Python 3.12, bringing the most recent language capabilities to workflow improvement. This improve gives entry to the most recent Python language enhancements, efficiency enhancements, and library updates, retaining your knowledge pipelines trendy and environment friendly.

Options not at the moment supported in Amazon MWAA

Though we’re launching many of the Airflow 3 options on Amazon MWAA on this launch, some options should not supported right now:

• DAG versioning (AIP-63) – Superior versioning options past fundamental model monitoring
• Change Flask AppBuilder (AIP-79) – Full alternative capabilities
• Edge Executor and job isolations (AIP-69) – Distant execution capabilities
• Multi-language help (AIP-72) – Assist for languages apart from Python

We plan to help these options in subsequent variations of Airflow on Amazon MWAA.

Conclusion

Airflow 3 on Amazon MWAA delivers enhanced workflow automation capabilities. The architectural enhancements, enhanced safety mannequin, and developer-friendly options present a stable basis for constructing extra dependable and maintainable knowledge pipelines.The introduction of Asset Watchers adjustments how workflows can reply to exterior occasions, enabling actually event-driven scheduling. This functionality, mixed with the brand new asset-centric workflow design, makes Airflow 3 a extra highly effective and versatile orchestration service.

The scheduler optimizations ship efficiency enhancements for job execution and workflow administration, and the improved backfill capabilities make historic knowledge processing extra environment friendly. The DAG versioning system improves workflow stability and collaboration, and Python 3.12 help retains your knowledge pipelines trendy and environment friendly.

Organizations can now reap the benefits of these new options and enhancements in Airflow 3 on Amazon MWAA to boost their workflow orchestration capabilities. To get began, go to the Amazon MWAA product web page.

In regards to the authors

Anurag Srivastava works as a Senior Massive Information Cloud Engineer at Amazon Net Providers (AWS), specializing in Amazon MWAA. He’s enthusiastic about serving to clients construct scalable knowledge pipelines and workflow automation options on AWS.

Kamen Sharlandjiev is a Sr. Massive Information and ETL Options Architect, Amazon MWAA and AWS Glue ETL knowledgeable. He’s on a mission to make life simpler for patrons who’re dealing with advanced knowledge integration and orchestration challenges. His secret weapon? Absolutely managed AWS providers that may get the job performed with minimal effort. Observe Kamen on LinkedIn to maintain updated with the most recent Amazon MWAA and AWS Glue options and information!

Ankit Sahu brings over 18 years of experience in constructing revolutionary digital services. His numerous expertise spans product technique, go-to-market execution, and digital transformation initiatives. At present, Ankit serves as Senior Product Supervisor at Amazon Net Providers (AWS), the place he leads the Amazon MWAA service.

Mohammad Sabeel works as a Senior Cloud Assist Engineer at Amazon Net Providers (AWS), specializing in AWS Analytics providers together with AWS Glue, Amazon MWAA, and Amazon Athena. With over 14 years of IT expertise, he’s enthusiastic about serving to clients construct scalable knowledge processing pipelines and optimize their analytics options on AWS.

Satya Chikkala is a Options Architect at Amazon Net Providers. Based mostly in Melbourne, Australia, he works intently with enterprise clients to speed up their cloud journey. Past work, he’s very enthusiastic about nature and pictures.

Sriharsh Adari is a Senior Options Architect at Amazon Net Providers (AWS), the place he helps clients work backward from enterprise outcomes to develop revolutionary options on AWS. Over time, he has helped a number of clients on knowledge system transformations throughout trade verticals. His core space of experience embrace know-how technique, knowledge analytics, and knowledge science. In his spare time, he enjoys enjoying sports activities, binge-watching TV reveals, and enjoying Tabla.