https://szeitlin.github.io/posts/airflow/ Recent content in Airflow on p < Whatever Hugo en-us Sat, 18 Nov 2017 00:00:00 +0000 https://szeitlin.github.io/posts/airflow/airflow-for-hands-off-etl/ Sat, 18 Nov 2017 00:00:00 +0000 https://szeitlin.github.io/posts/airflow/airflow-for-hands-off-etl/ <h1 id="airflow-for-hands-off-etl">Airflow for hands-off ETL</h1> <hr> <p>Almost exactly a year ago, I joined <a href="https://www.yahoo.com">Yahoo</a>, which more recently became <a href="https://www.oath.com">Oath</a>.</p> <p>The team I joined is called the Product Hackers, and we work with large amounts of data. By large amounts I meant, billions of rows of log data.</p> <p>Our team does both ad-hoc analyses and ongoing machine learning projects. In order to support those efforts, our team had initially written scripts to parse logs and run them with cron to load the data into Redshift on AWS. After a while, it made sense to move to <a href="http://pythonhosted.org/airflow/">Airflow</a>.</p>