newfrontdocker/apache-airflow-spark repository overview

⁠Airflow 2.1.0 with some Friends

This image extends the official apache/airflow image for 2.1.0 and adds the Java 11 runtime (openjdk). This enables Airflow programs to dynamically compile using Java, but adds the additional weight of the full JDK.

⁠Apache Spark Providers

This image also ships with the airflow contrib spark libraries. This enables you to harness Spark directly.

---

Note: In order to make this flexible, you can volume mount your Spark distribution so that the Spark Connector can use multiple versions of spark. Given this is packaged with Java 11 - that unfortunately only means versions of Spark >= 3.0.0.

If you have questions regarding how to use this distribution, please feel free to reach out on twitter at https://twitter.com/newfront⁠