Setting up Apache Airflow

Last updated: 05/22/2026

As mentioned in the beginning of this book, Amazon Managed Workflows for Apache Airflow, or Amazon MWAA, is used for this project. This cloud-based service has been chosen because, even though Airflow is supported by all major operating systems, installing it locally on a server can prove to be very difficult. Indeed, setting up Airflow is much more complicated that simply downloading and installing the software—in reality, it relies on several other services to function properly, most notably a database and a web server for the user interface. The advantage of using Amazon MWAA is that everything is setup, managed and secured by AWS. An overview of the architecture of each Airflow environment is presented here.

AWS is not the only cloud provider to offer Airflow as a Service—another popular provider that makes it available for its users is Microsoft Azure. ITS chose MWAA because it is already using AWS for most of its operations and systems.

This page explains how to configure and launch an Airflow environment using MWAA. It also gives details on how the environment for the proof of concept was set up and configured.

MWAA

As explained above, Amazon Managed Workflows for Apache Airflow is one of the services provided by AWS and is accessible via the dashboard.

For now, the URI Foundation does not have its own AWS accounts and shares the accounts used by ITS. To get access to these accounts, permission must be requested to Chi Shen (cshen@uri.edu) or Patrick Canole (pcanole@uri.edu). Once the right permissions have been granted, the AWS dashboard can be accessed by:

Logging into Microsoft using the URI SSO
Going to the Microsoft "My Apps" page
Clicking on "AWS Single Sign-On"
Choosing the appropriate account and permission

For the proof of concept:

The account we are using is "ITS shared non-prod", with the permission "URIPermSet_FullAdmin". This account is only used for testing.

Once on the AWS dashboard, Amazon Managed Workflows for Apache Airflow can be accessed by simply typing "MWAA" in the search bar on top of the page. It can also appear in the "Recently visited" section if it has already been accessed.

When clicking on "Amazon MWAA", a list of environments is displayed. An environment is simply an instance of Airflow isolated from the others that is running in one of the data centers of AWS. The configuration of an environment can be seen by clicking on its name, and the dashboard of Airflow can be accessed by clicking on "Open Airflow UI".

For the proof of concept:

The environment that was created for the proof of concept is called "foundation-etl-poc-airflow".

Creating a new Airflow environment

If a new environment needs to be created, AWS provides a detailed guide on how to proceed here. The AWS CLI (command line interface) can be used, but it is much simpler and more intuitive to do it using the console.

Creating a new S3 bucket

Before creating a new Airflow environment, an S3 bucket needs to be created. S3 (Simple Storage Service) is another service provided by AWS that is used to store objects in the cloud. As explained in this part of the guide, the S3 bucket will be used to store the requirements file and the DAGs of the Airflow environment. The structure of this bucket will be extremely simple.

S3 bucket
|___ requirements.txt
|___ dags/
		|___ first_dag.py
		|___ second_dag.py
		|___ ...

For the proof of concept:

The S3 bucket that was created for the "foundation-etl-poc-airflow" environment is called "foundation-etl-poc-bucket".

Creating a new VPC network

Technically, there should be no need to create a new VPC (Virtual Private Cloud) network. When configuring the new Airflow environment, the VPC called "MWAAEnvironment" can be selected.

Configuring the new Airflow environment

To configure the new Airflow environment, only its name, the version of Airflow (it is recommended to select the latest available version) and the paths to the requirements file as well as the DAGs folder need to be specified for the details. For the advanced setting, once the VPC called "MWAAEnvironment" has been selected, everything else can be left as is, unless some other settings need to be modified in order to meet the requirements for the environment. Once the configuration has been reviewed, the environment can be created—this process will take between 20 and 30 minutes to complete.

For the proof of concept:

In the advanced settings, under "Web server access", "Public Network (Internet Accessible)" was selected. This setting makes the Airflow UI of the environment easily accessible on web browsers via Internet.

As the environment created for the proof of concept only has to run a few simple DAGs occasionally, the smallest "Environment class" was selected (mw1.micro).

Updating an Airflow environment

The configuration of an Airflow environment can be edited and updated at any time by simply clicking on "Edit". Like the creation process, the environment will take between 20 and 30 minutes to update.

If the requirements file of an Airflow environment is modified (i.e. some Python libraries are added or removed), this environment needs to be updated. This can be done by clicking on "Edit", selecting the appropriate version of the requirements file and updating the environment. Once again, this process will take between 20 and 30 minutes. If the environment is not updated, the Python libraries that were added to the requirements file will not be installed automatically and they will not be available—it they are imported in a DAG, it will result in an error.

MWAA Local Runner

If the requirements file is regularly modified during the development phase of a DAG, having to wait up to 30 minutes every time the environment updates can massively slow down work. Luckily, AWS has a GitHub repository that provides "a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally."

The CLI builds a Docker container image locally that’s similar to a MWAA production image. This allows you to run a local Apache Airflow environment to develop and test DAGs, custom plugins, and dependencies before deploying to MWAA.

> GitHub - aws-mwaa-local-runner

As rebuilding the Docker image only takes a couple of minutes, this local Airflow environment updates way faster than Airflow environments running in the cloud.

The latest available Airflow version on the local runner is 2.10. AWS also provides Docker images for later versions but not through the local runner, and the installation process is slightly more complicated. If using the latest Airflow version on MWAA (currently 3.2.1), the user interface will be different on the local runner. The Docker images for Airflow versions above 2.10 can be found here.

This local runner can be installed on MacOS, Linux and Windows with WSL. The instructions can be found on the README of this GitHub repository.

When the requirements file has been modified, the environment can be updated by stopping the Docker container, building a new image and starting the container again.