I’ve been doing a lot of work around grief lately. Both of my parents died in 2023. Two other friends also died in 2023 - both former coworkers. One was in his mid-thirties and had a rare brain tumor; the other died from COVID. My dad’s death last June came after several months in the ICU. He endured repeated rounds of dialysis, which he hated, and ventilator-associated pneumonia, which I hadn’t heard of before.

Life is growth. If we stop growing, technically and spiritually, we are as good as dead. —- Morihei Ueshiba I started studying aikido about 12 years ago, shortly after I moved to San Francisco. I had no idea how much I would enjoy it, or how much it would change my life. I didn’t know what it was when I first started, but the translation I like the best is “Path of Life Harmony.

This is one of those posts that I’m writing mostly because I’m still frequently hearing people say they’re struggling with the question of how to interview data people. I’ve blogged and spoken previously about the misery of being an interviewee, so suffice it to say I have a ton of empathy for how awful it is to go through a bad interview process, and how disappointing it is to go through a long, grueling interview and not get an offer.

I decided to embark on this comparison mostly out of curiosity. No tool is perfect for all use cases, that’s why we have forks, and spoons, and sometimes when we’re camping, sporks. Although Pachyderm claims to use a git-style approach for data and code versioning, there are aspects of the Pachyderm approach (like forking) that aren’t exactly like git. So one thing I wanted to know is, how well does this analogy to git work for DVC?

Coming from a background in bench science, or what we affectionately referred to as “wetlab”, I like to test everything I do, and I like my tests to be fast and representative of what I expect to find when I run things “for real”. Most people I’ve met who are newer to data engineering find that it’s not immediately obvious how to write and run tests for data things. It’s different enough from writing unit tests for web apps that there are some pitfalls to be aware of.

Recently, a coworker asked me what the difference is between data science and data engineering. She seemed to think that data engineering was “just analytics”, which is a definition I’ve often heard applied to data science, as well. After I got over my initial horror that she had been wondering this for months and hadn’t asked sooner, I started to explain: Data engineering is getting data, cleaning data, reshaping data, validating data, and loading it into databases.

Question: How come we don’t have more diversity at our company? This is one of those posts, I’m writing it because I keep hearing people asking the same questions over and over. I hear these questions at meetups and in Slack groups every week, and I hate seeing people trying to reinvent the wheel when it’s really not necessary. Answer: You haven’t tried that hard. For the purposes of this blog post, let’s go with the hypothesis that there are plenty of nontraditional candidates out there like me.

Recently, someone asked for help with setting up postgres in docker and connecting to it with python. While helping this person, I realized this is something that should be fairly straightforward with a simple set of instructions, but there aren’t a lot good beginner tutorials out there. So I decided to write this up because I’m sure it’s something other people would also find useful. A few years ago I wouldn’t have been able to do this even with a lot of googling (this time I only had to google a few things!

I was chatting with some friends recently, and this question came up: What does a good working relationship with your manager look like? This is the list I came up with. I’m not saying I’ve been perfect at all of these myself as a manager, but it’s what I strive for, and it’s what I look for in a good manager. They trust you They ask for your input when appropriate They promote your work to others in the company where appropriate You trust them No surprises You feel safe asking them for help when you need it They discuss your career goals with you, and help steer you toward them This post is just an elaboration on what I mean by each of the things on that list.

I frequently hear Python referred to as a ‘scripting’ language, because it’s not compiled. Unfortunately, for this reason, a lot of people seem to assume you can’t write ‘real’ programs with it. This post is about moving beyond using Python as a scripting language. I’m assuming you’re already comfortable with basic python data types and methods. Note: Most of the content here is specific to Python 3. If you’re just learning Python now, don’t learn Python 2, it’s being deprecated and many current libraries already stopped supporting it.