I decided to embark on this comparison mostly out of curiosity. No tool is perfect for all use cases, that’s why we have forks, and spoons, and sometimes when we’re camping, sporks. Although Pachyderm claims to use a git-style approach for data and code versioning, there are aspects of the Pachyderm approach (like forking) that aren’t exactly like git. So one thing I wanted to know is, how well does this analogy to git work for DVC? And are there use cases where one should definitely use DVC or Pachyderm?

Coming from a background in bench science, or what we affectionately referred to as “wetlab”, I like to test everything I do, and I like my tests to be fast and representative of what I expect to find when I run things “for real”.

Most people I’ve met who are newer to data engineering find that it’s not immediately obvious how to write and run tests for data things. It’s different enough from writing unit tests for web apps that there are some pitfalls to be aware of. This post is about that.

Recently, a coworker asked me what the difference is between data science and data engineering. She seemed to think that data engineering was “just analytics”, which is a definition I’ve often heard applied to data science, as well.

After I got over my initial horror that she had been wondering this for months and hadn’t asked sooner, I started to explain:

Data engineering is getting data, cleaning data, reshaping data, validating data, and loading it into databases. Data science is all of that, plus analyzing the data and figuring out how to display it in a way that makes sense, and sometimes also building models and doing machine learning.

Recently, someone asked for help with setting up postgres in docker and connecting to it with python.

While helping this person, I realized this is something that should be fairly straightforward with a simple set of instructions, but there aren’t a lot good beginner tutorials out there. So I decided to write this up because I’m sure it’s something other people would also find useful. A few years ago I wouldn’t have been able to do this even with a lot of googling (this time I only had to google a few things!).

I frequently hear Python referred to as a ‘scripting’ language, because it’s not compiled. Unfortunately, for this reason, a lot of people seem to assume you can’t write ‘real’ programs with it. This post is about moving beyond using Python as a scripting language. I’m assuming you’re already comfortable with basic python data types and methods.

Note: Most of the content here is specific to Python 3. If you’re just learning Python now, don’t learn Python 2, it’s being deprecated and many current libraries already stopped supporting it.

If you do a lot of data pipelining, you’ve probably heard a lot about Airflow by now. I gave a talk about it a while back at a meetup, and wrote a blog post about it. The gist of my pitch for Airflow was essentially “Look, it’s so much better than cron.”

Fast-forward a year or two, and my team is using Pachyderm now. This post is about why I wanted to try Pachyderm, what I love about it, some things that can be improved about it, and some of the tricks you’ll need to know if you want to start using it.

To make a long, mostly whiny story short, as part of my current role, I’ve been doing a lot of fighting with AWS to help support my team.

Some of the things I’ve learned along the way are probably not obvious if you, like me, are relying mostly on AWS docs and other people’s advice, so I thought I’d collect some of them here.

Best practices for storing big data on S3

The scene:

I needed to process data from an s3 bucket using pyspark. The s3 bucket was owned by a different account. I had done this before. But this time, there was a twist: we needed to encrypt the data because of GDPR requirements. At the end of the processing, I needed to save the results to another s3 bucket for loading into Redshift.

Thus began a weeks-long saga of learning about AWS the hard way.

Why Spark?

Lately I have been working on a project that requires cleaning and analyzing a large volume of event-level data.

Originally, I did some exploratory data analysis on small samples of data (up to 15 million rows) using pandas, my usual data visualization tools, and multiprocessing. But then it was time to scale up.

Why Spark is good for this

Distributed processing means it’s very fast at very large scale, and we can scale it up with minimal adjustments (the same code still works, we just need a bigger cluster).

Things I learned while following a tutorial on how to build reusable models with scikit-learn.

1. When in doubt, go back to pandas.
2. When in doubt, write tests.
3. When in doubt, write helper methods to wrap existing objects, rather than creating new objects.

Ingesting “clean” data is easy, right?

Step 1 of this tutorial began with downloading data using requests, and saving that to a csv file. So I did that. I’ve used requests before, I had no reason to think it wouldn’t work. It looked like it worked.