Statistics on p < Whatever

Probability binning: simple and fast

Fri, 04 Nov 2016 00:00:00 +0000

Over the years, I’ve done a few data science coding challenges for job interviews. My favorite ones included a data set and asked me to address both specific and open-ended questions about that data set.

One of the first things I usually do is make a bunch of histograms. Histograms are great because it’s an easy way to look at the distribution of data without having to plot every single point, or get distracted by a lot of noise.

Validating Results

Fri, 15 Apr 2016 00:00:00 +0000

I don’t believe truth is a finite value. Truth is what we know right now. Every ten years or so, a major discovery gets overturned. Scientists are just people, and we’re wrong a lot.

So one of the scariest things about doing research, or predictions, is trying to convince yourself, and other people, that what you think you’ve discovered is ‘real’.

Or at least real enough, right now, to be believable. Whenever I do a project, I hope my findings will stand the test of time, at least long enough to be useful.

Probability binning part 2: speeding up machine learning

Mon, 01 Jan 0001 00:00:00 +0000

In a previous post, I wrote about using probability binning to create categories out of numerical data. This simplifying approach works well on various kinds of distributions.

In this post, I wanted to show an example of how well it can work.