Human moderators who give final review and sign off.Security, consistency, and quality checking. ModerationĮvery version of each package undergoes a rigorous moderation process before it goes live that typically includes: When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.Īpache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.Welcome to the Chocolatey Community Package Repository! The packages found in this section of the site are provided, maintained, and moderated by the community. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. Can I use the Apache Airflow logo in my presentation?Īirflow works best with workflows that are mostly static and slowly changing.Base OS support for reference Airflow images.Support for Python and Kubernetes versions.The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. When the DAG structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include Luigi, Oozie and Azkaban.Īirflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's XCom feature). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work.Īirflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |