Using Airflow on Windows
•5 min read
From version 2.3, Airflow (opens in a new tab) has been available in Apple Silicon.
Unfortunately, using Airflow in Windows operating systems is a different matter as it doesn't natively support the Microsoft's operating system.
However, it is not entirely true, as I learned that Airflow can be used with the Docker (opens in a new tab).
In a word, with Docker WSL 2 installed (by default), Windows users can use the nix-based commands easily. In other words, Airflow commands can be used in Windows as well.
Below, assuming that you already installed Docker, I will just show deploy Airflow with Docker and an example of Airflow CLIs.
Deploying Airflow with Docker
Get Ready
According to offical Airflow website (opens in a new tab), you can deploy using curl command as below.
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.4.3/docker-compose.yaml'However, in Windows, unfortunately, that command didn't work for me.
And since curl command is downloading a file, you can simply copy the content of the yaml file to create one in Docker local directory (C:\Users{user_name}.docker) with filename docker-compose.yaml.
Also, just in case it doesn't work, it is advised to create a file named .env with adding the following line.
AIRFLOW_UID=50000Now, you can deploy the Airflow as below.
- Note: Make you to start Docker Desktop before proceeding. Otherwise, docker commands are not recognized.
Initializing Airflow
docker-compose up airflow-initSimple, heh?
Running Airflow (Webserver)
docker-compose upFirst, let's if the process is running.
(works) C:\Users\minpark\.docker>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7f54c4de8703 apache/airflow:2.4.3 "/usr/bin/dumb-init …" 31 hours ago Up 2 seconds (health: starting) 8080/tcp docker-airflow-worker-1
dde3a7124f42 apache/airflow:2.4.3 "/usr/bin/dumb-init …" 31 hours ago Up 27 seconds (healthy) 8080/tcp docker-airflow-scheduler-1
36103c149a6d apache/airflow:2.4.3 "/usr/bin/dumb-init …" 31 hours ago Up 27 seconds (healthy) 8080/tcp docker-airflow-triggerer-1
d9d8ddb4fd60 apache/airflow:2.4.3 "/usr/bin/dumb-init …" 31 hours ago Up 27 seconds (healthy) 0.0.0.0:8080->8080/tcp docker-airflow-webserver-1
3db3f96e4dc9 redis:latest "docker-entrypoint.s…" 31 hours ago Up 27 seconds (healthy) 6379/tcp docker-redis-1
150e86e38855 postgres:13 "docker-entrypoint.s…" 31 hours ago Up 28 seconds (healthy) 5432/tcp docker-postgres-1Looks good, with this, you can now access Airflow Admin GUI with https://localhost:8080 (opens in a new tab).
Airflow CLIs (dags list)
Although you can access Admin GUI with the previous command, executing Airflow CLI commands is a little bit different.
Unlike installing Airflow locally, you cannot just run airflow dags list as Airflow runs under Docker.
To see a list of dags, you run the following command using the same Docker command.
docker-compose run airflow-worker airflow dags listAs you might have noticed, for the similar Airflow CLI commands, you need airflow-worker here.
Below, as you can see, we can use Airflow CLIs in Windows!
(works) C:\Users\minpark\.docker>docker-compose run airflow-worker airflow dags list
[+] Running 3/0
- Container docker-redis-1 Running 0.0s
- Container docker-postgres-1 Running 0.0s
- Container docker-airflow-init-1 Created 0.0s
[+] Running 3/3
- Container docker-redis-1 Healthy 0.5s
- Container docker-postgres-1 Healthy 0.5s
- Container docker-airflow-init-1 Started 1.2s
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
FutureWarning,
dag_id | filepath | owner | paused
=========================================+============================================================+=========+=======
dataset_consumes_1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
dataset_consumes_1_and_2 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
dataset_consumes_1_never_scheduled | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
dataset_consumes_unknown_never_scheduled | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
dataset_produces_1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
dataset_produces_2 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_datasets.py | |
demo | demo.py | airflow | True
example_bash_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_bash_operator.py | |
example_branch_datetime_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_datetime_operator.py | |
example_branch_datetime_operator_2 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_datetime_operator.py | |
example_branch_datetime_operator_3 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_datetime_operator.py | |
example_branch_dop_operator_v3 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_python_dop_operator_3.py | |
example_branch_labels | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_labels.py | |
example_branch_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_operator.py | |
example_branch_python_operator_decorator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_operator_decorator.py | |
example_complex | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_complex.py | |
example_dag_decorator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_dag_decorator.py | |
example_external_task_marker_child | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_external_task_marker_dag.py | |
example_external_task_marker_parent | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_external_task_marker_dag.py | |
example_kubernetes_executor | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_kubernetes_executor.py | |
example_local_kubernetes_executor | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_local_kubernetes_executor.py | |
example_nested_branch_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_nested_branch_dag.py | |
example_passing_params_via_test_command | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_passing_params_via_test_command.py | |
example_python_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_python_operator.py | |
example_short_circuit_decorator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_short_circuit_decorator.py | |
example_short_circuit_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_short_circuit_operator.py | |
example_skip_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_skip_dag.py | |
example_sla_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_sla_dag.py | |
example_subdag_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_subdag_operator.py | |
example_subdag_operator.section-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_subdag_operator.py | |
example_subdag_operator.section-2 | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_subdag_operator.py | |
example_task_group | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_task_group.py | |
example_task_group_decorator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_task_group_decorator.py | |
example_time_delta_sensor_async | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_time_delta_sensor_async.py | |
example_trigger_controller_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_trigger_controller_dag.py | |
example_trigger_target_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_trigger_target_dag.py | |
example_weekday_branch_operator | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_branch_day_of_week_operator.py | |
example_xcom | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_xcom.py | |
example_xcom_args | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_xcomargs.py | |
example_xcom_args_with_operators | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_xcomargs.py | |
latest_only | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_latest_only.py | |
latest_only_with_trigger | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/example_latest_only_with_trigger.py | |
tutorial | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/tutorial.py | |
tutorial_dag | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/tutorial_dag.py | |
tutorial_taskflow_api | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/tutorial_taskflow_api.py | |
tutorial_taskflow_api_virtualenv | /home/airflow/.local/lib/python3.7/site-packages/airflow/e | airflow | True
| xample_dags/tutorial_taskflow_api_virtualenv.py | |