Apache Airflow Cookbook

At some point, every data team hits the same roadblock. Each night, the script that used to run smoothly starts to go wrong at 2 a.m. No one knows what step has gone wrong, and reruns are double-counting the numbers that the finance team has already published. Apache Airflow is great because it’s been rebuilt from the ground up to suit the way work teams actually do things today.This cookbook provides you hundreds of solutions that are independent and will take you from the first installation to a platform that supports ETL, ELT, MLOps, AIOps and business operations all at the same time. Each recipe starts with a real problem, has short and easy-to-read code, and ends by showing how the fix works with real terminal output. We can learn to author DAGs with the Task SDK, schedule pipelines on data rather than on the clock, bind extractions so reruns repair instead of duplicates, and run any task inside its own container. We will be practising to write custom operators, packaging them for other teams, extending Airflow through plugins and secrets backends, orchestrating model training and promotion, provisioning infrastructure that tears itself down, and diagnosing stalls from their symptoms.To me, this book is best suited for every software engineer, backend developer and every such platform teams who keep on building, running and troubleshooting the workflows every day.Key LearningsConvert legacy operator DAGs into TaskFlow functions that infer dependencies from ordinary callsBound every extraction to its data interval so reruns repair rather than duplicateSchedule pipelines on asset updates instead of guessing when upstream work finishesFan tasks out dynamically over lists discovered at runtime using expand and partialRun any task inside a Kubernetes pod with its own image and resourcesWrite custom operators, hooks and sensors, then package them as installable providersExtend Airflow through plugins, macros, secrets backends and custom XCom storageOrchestrate model training, registry logging and promotion gates without a serving endpointProvision ephemeral infrastructure with setup and teardown pairs that always release resourcesDiagnose stuck queues, starved pools and zombie tasks from their distinct symptomsTable of ContentGetting Airflow 3 RunningAuthoring DAGs with Task SDKScheduling, Assets and Event-Driven PipelinesBuilding ETL PipelinesELT and Warehouse OrchestrationExtracting Insights from Batch ProcessesContainers and KubernetesCustom Operators, Hooks and SensorsPlugin Interface and Extending AirflowManaging ML PipelinesAIOps, Infrastructure and Business OperationsTesting, Monitoring and TroubleshootingTarget ReadersYou don’t need to be an Airflow pro or an expert to get started with this book. All you need is to be curious, have a Linux machine and be ready to run the commands instead of just reading them. If you’re committed to creating a strong, flexible and expandable workflow, you’re on the right track.