Engineering on p < Whatever

DVC_vs_Pachyderm

Wed, 03 Mar 2021 13:13:57 -0800

I decided to embark on this comparison mostly out of curiosity. No tool is perfect for all use cases, that’s why we have forks, and spoons, and sometimes when we’re camping, sporks. Although Pachyderm claims to use a git-style approach for data and code versioning, there are aspects of the Pachyderm approach (like forking) that aren’t exactly like git. So one thing I wanted to know is, how well does this analogy to git work for DVC? And are there use cases where one should definitely use DVC or Pachyderm?

Test Patterns for Data Engineering

Fri, 26 Feb 2021 12:45:27 -0700

Coming from a background in bench science, or what we affectionately referred to as “wetlab”, I like to test everything I do, and I like my tests to be fast and representative of what I expect to find when I run things “for real”.

Most people I’ve met who are newer to data engineering find that it’s not immediately obvious how to write and run tests for data things. It’s different enough from writing unit tests for web apps that there are some pitfalls to be aware of. This post is about that.

Data Engineering Is Software Engineering

Thu, 15 Oct 2020 09:47:10 -0700

Recently, a coworker asked me what the difference is between data science and data engineering. She seemed to think that data engineering was “just analytics”, which is a definition I’ve often heard applied to data science, as well.

After I got over my initial horror that she had been wondering this for months and hadn’t asked sooner, I started to explain:

Data engineering is getting data, cleaning data, reshaping data, validating data, and loading it into databases. Data science is all of that, plus analyzing the data and figuring out how to display it in a way that makes sense, and sometimes also building models and doing machine learning.

Postgres With Docker

Tue, 08 Oct 2019 15:03:46 -0700

Recently, someone asked for help with setting up postgres in docker and connecting to it with python.

While helping this person, I realized this is something that should be fairly straightforward with a simple set of instructions, but there aren’t a lot good beginner tutorials out there. So I decided to write this up because I’m sure it’s something other people would also find useful. A few years ago I wouldn’t have been able to do this even with a lot of googling (this time I only had to google a few things!).

Python OOP

Fri, 24 May 2019 13:29:53 -0700

I frequently hear Python referred to as a ‘scripting’ language, because it’s not compiled. Unfortunately, for this reason, a lot of people seem to assume you can’t write ‘real’ programs with it. This post is about moving beyond using Python as a scripting language. I’m assuming you’re already comfortable with basic python data types and methods.

Note: Most of the content here is specific to Python 3. If you’re just learning Python now, don’t learn Python 2, it’s being deprecated and many current libraries already stopped supporting it.

Pachyderm vs Airflow

Sun, 02 Dec 2018 17:10:01 -0800

If you do a lot of data pipelining, you’ve probably heard a lot about Airflow by now. I gave a talk about it a while back at a meetup, and wrote a blog post about it. The gist of my pitch for Airflow was essentially “Look, it’s so much better than cron.”

Fast-forward a year or two, and my team is using Pachyderm now. This post is about why I wanted to try Pachyderm, what I love about it, some things that can be improved about it, and some of the tricks you’ll need to know if you want to start using it.

More AWS things I learned the hard way: S3 best practices and VPCs

Sun, 02 Dec 2018 16:29:47 -0800

To make a long, mostly whiny story short, as part of my current role, I’ve been doing a lot of fighting with AWS to help support my team.

Some of the things I’ve learned along the way are probably not obvious if you, like me, are relying mostly on AWS docs and other people’s advice, so I thought I’d collect some of them here.

Best practices for storing big data on S3

Cross-account access with AWS

Sat, 04 Aug 2018 00:00:00 +0000

The scene:

I needed to process data from an s3 bucket using pyspark. The s3 bucket was owned by a different account. I had done this before. But this time, there was a twist: we needed to encrypt the data because of GDPR requirements. At the end of the processing, I needed to save the results to another s3 bucket for loading into Redshift.

Thus began a weeks-long saga of learning about AWS the hard way.

Things I learned about Pyspark the hard way

Wed, 28 Mar 2018 00:00:00 +0000

Why Spark?

Lately I have been working on a project that requires cleaning and analyzing a large volume of event-level data.

Originally, I did some exploratory data analysis on small samples of data (up to 15 million rows) using pandas, my usual data visualization tools, and multiprocessing. But then it was time to scale up.

Why Spark is good for this

Distributed processing means it’s very fast at very large scale, and we can scale it up with minimal adjustments (the same code still works, we just need a bigger cluster).

A tutorial within a tutorial on building reusable models with scikit-learn

Mon, 12 Sep 2016 00:00:00 +0000

Things I learned while following a tutorial on how to build reusable models with scikit-learn.

When in doubt, go back to pandas.
When in doubt, write tests.
When in doubt, write helper methods to wrap existing objects, rather than creating new objects.

Ingesting “clean” data is easy, right?

Step 1 of this tutorial began with downloading data using requests, and saving that to a csv file. So I did that. I’ve used requests before, I had no reason to think it wouldn’t work. It looked like it worked.

Shuffling the deck: an interview experience

Thu, 16 Jun 2016 00:00:00 +0000

Here is a story about an interesting interview question and how I approached it.

The company in question wasn’t interested in actually looking at my code, since I apparently tried to answer the wrong question.

Given a deck of n unique cards, cut the deck c cards from the top and perform a perfect shuffle. A perfect shuffle is where you put down the bottom card from the top portion of the deck followed by the bottom card from the bottom portion of the deck. This is repeated until one portion is used up. The remaining cards go on top.

Test-driven data pipelining

Mon, 08 Feb 2016 00:00:00 +0000

When to test, and why:

• Write a test for every method.

• Write a test any time you find a bug! Then make sure the test passes after you fix the bug.

• Think of tests as showing how your code should be used, and write them accordingly. The next person who’s going to edit your code, or even just use your code, should be able to refer to your tests to see what’s happening.

Data pipelining with pandas

Sat, 02 Jan 2016 00:00:00 +0000

For better or worse, when you’re dealing with data pipelines of varying shapes and sizes, sometimes you need to combine objects that don’t match up evenly.

For example, if you want to apply a condition via lookup, sometimes it makes sense to just do a merge. This creates a new column in your data table, and then you can use that for reference.

This is an extremely simple example to show what I mean:

Things I learned about zip files

Thu, 15 Jan 2015 00:00:00 +0000

In an effort to advance my python skills, I spent some time slowly pecking away at the puzzles on pythonchallenge. I got stuck on most of the challenges, and either had to search for a hint, or ask for help from a friend, or both. This latest one was particularly instructive, and it had to do with zipfiles.

I thought I knew what zip files were. I have used them since grad school, for transferring folders via email, and for compression. I used various utilities and command-line tools to deal with zipping and unzipping. But I never needed to know how they worked.

Automating user-friendly documentation with Selenium

Mon, 01 Jan 0001 00:00:00 +0000

Once upon a time, a friend recruited me to do some technical writing for the company where he works now. Basically, they needed someone to quickly revise and update the documentation for their software.

Most modern user-friendly software documentation isn’t just writing, though. It’s screenshots. A LOT of screenshots. So you don’t just write “click on the blue box”, you also show a picture of it, like this. See the blue box?

Fun with text file encodings

Mon, 01 Jan 0001 00:00:00 +0000

This post is about a couple of painful problems I’ve had with reading in text files.

They’re just text files, right? What could be hard about that?

1. Fun with encoding

The problem:

A friend sent me a dump of data from iTunes, as a table.

Just a text file.

She’s on a Mac. I’m on a Mac. But when I went to read in the file, Python choked in a way I hadn’t seen before.

Quick and dirty: plot your data on a map with python

Mon, 01 Jan 0001 00:00:00 +0000

Once upon a time, I looked at a couple of data sets that involved geographical data. I wanted to actually plot the data on a map, so I had to do some shopping around for easy ways to do this quickly with python.

What I wanted to do

One of the data sets provided zipcodes, which I was able to convert to latitude and longitude (more on this below). At a minimum, I knew I wanted to be able to plot: a) location, b) a number value and text label, c) multiple colors designating groups of data points, e.g. ‘high’ or ’low’ values for a particular variable, which ideally could be toggled on and off.

Recursion excursion

Mon, 01 Jan 0001 00:00:00 +0000

More than once, and probably not for the last time, I have done a technical interview for which I was underprepared. I feel like no matter how much I try to prepare, I am always underprepared for technical interviews.

I’m going to tell you about a time I was underprepared for a few reasons, including:

a) It was the first interview where I was asked to write more than a couple lines of recursive code