Things I learned while following a tutorial on how to build reusable models with scikit-learn.

1. When in doubt, go back to pandas.
2. When in doubt, write tests.
3. When in doubt, write helper methods to wrap existing objects, rather than creating new objects.

Ingesting “clean” data is easy, right?

Step 1 of this tutorial began with downloading data using requests, and saving that to a csv file. So I did that. I’ve used requests before, I had no reason to think it wouldn’t work. It looked like it worked.

Here is a story about an interesting interview question and how I approached it.

The company in question wasn’t interested in actually looking at my code, since I apparently tried to answer the wrong question.

Given a deck of n unique cards, cut the deck c cards from the top and perform a perfect shuffle. A perfect shuffle is where you put down the bottom card from the top portion of the deck followed by the bottom card from the bottom portion of the deck. This is repeated until one portion is used up. The remaining cards go on top.

I don’t believe truth is a finite value. Truth is what we know right now. Every ten years or so, a major discovery gets overturned. Scientists are just people, and we’re wrong a lot.

So one of the scariest things about doing research, or predictions, is trying to convince yourself, and other people, that what you think you’ve discovered is ‘real’.

Or at least real enough, right now, to be believable. Whenever I do a project, I hope my findings will stand the test of time, at least long enough to be useful.

When to test, and why:

• Write a test for every method.

• Write a test any time you find a bug! Then make sure the test passes after you fix the bug.

• Think of tests as showing how your code should be used, and write them accordingly. The next person who’s going to edit your code, or even just use your code, should be able to refer to your tests to see what’s happening.

For better or worse, when you’re dealing with data pipelines of varying shapes and sizes, sometimes you need to combine objects that don’t match up evenly.

For example, if you want to apply a condition via lookup, sometimes it makes sense to just do a merge. This creates a new column in your data table, and then you can use that for reference.

This is an extremely simple example to show what I mean:

Am I getting slower every day?

If you’ve ever been a bike commuter, you’ve probably asked yourself this question. Thanks to these little devices we can now attach to ourselves or our bicycles, we can now use our own actual ride data to investigate these kinds of questions, as well as questions like these:

• If I’m going to work from home one day a week, which day would maximize my recovery?
• Do I tend to ride faster in the morning or the evening?

Last year, I wrote a few posts about learning how to parse a set of Garmin XML data from 2013 and analyze it using pandas, matplotlib, and seaborn. This year I redid the same analyses, with a new installment of data from 2014.

In continuing my series on investigating bike data, I ran into some interesting aspects of working with device data.

I have some experience with devices, thanks to my many years of working in research labs. This post is about the fun of hunting down what’s working and what’s not.

Things to consider when working with devices

1. Are you using the device yourself?
2. Are you interacting with the user(s) (directly or indirectly)? Or not at all?
3. What is the device designed to do? Are you using it for its intended purpose?
4. How well does the device actually work? Generic measurables might include: sensitivity, specificity, accuracy, precision, battery life
5. What else is being measured?
6. Measured how?
7. How are data stored? How much data can it store? How does it connect to other devices/data stores?

In the case of a bike computer, I have been looking at:

So I have some bike data that I parsed out of XML and put into a pandas dataframe. Most of the questions I wanted to ask required that the timestamp of each ride segment, or lap, be used as the index along the x-axis of a plot.

Non-obvious nuances of pandas datetime objects and indexes.

You have to sort the dataframe by timestamps, before you can convert the timestamps to use as an index.

One thing I wanted to do with this data set was experiment with plotting methods. I had already done some exploratory plotting with regular matplotlib, so I had some vague ideas about what I wanted to do.

First I had to select out subsets of data to compare. I knew that there were two types of rides: shorter trips in the city, and longer trips in the suburbs. I was feeling lazy, so I just did a quick threshold with SQL.

One of the main reasons this project turned out to be interesting is that time series data has all kinds of gotchas. I never had to deal with a lot of this before, because the sorts of time series I did in my scientific life didn’t care about real-life things like time zones. We mostly just cared about calculating time elapsed.

…tick…tick…tick

Anyway one thing I wondered about with the bike data was, can we compare average speeds in the morning vs. the afternoon? But to do that, I first had to parse the datetime objects and put them in the right time zone.