How to Get Started with a Data Science Project

TLDR? go straight to the list of 99 data science project ideas!

Doing projects is one of the best and most satisfying ways of improving at machine learning or data science.

But how do you come up with a good project idea? How do you then actually execute on that idea to get something working at the end? In this guide we’ll walk through what you need to think about and do to get off the ground.

First of all, you need to decide, why do you want to do a project? Common reasons include to learn a new skill, to practice a skill, to increase your chances of getting a job, or just for pleasure.

How you should approach a project depends on what your goal for it is. In this guide we will assume that the goal is to learn or practice a specific skill, like cleaning and exploring data, or training models, or presenting findings. In addition to those specific skills, doing a project also means that at the end of the day, you want to have an actual end product, something that you can show to others.

Creating a project successfully is a balancing act, you have to gather and connect the different pieces together into a coherent whole. This in itself, the ability to deliver, is one of the most important skills you can learn in any field.

Before going any further, write down why you want to do a project.

I want to do a project because I want to _____________________
______________________________________________________________

Now that you know why you are doing this project, let’s see what the parts of it will be.

Anatomy of a Data Science Project #

A data science or machine learning project is all about taking some data, extracting the value from it, and then presenting (or acting on) that value. So you need to decide on the value you will provide, and then find some data to do it.

A project has the following parts:

1. Data
2. Method applied to the data to extract the value
3. Presentation of the value

While we list data first, that’s not where you start. You start with the most important thing, which is the value you want to provide. What form does that value take? It could be a prediction or an inference. A prediction provides information about whether or in what way something will happen. Inference provides information about what is happening or why something is happening. While presenting the value in the best way often requires a lot of thought and iteration, for now we are not going to worry too much about it.

To make the value you can provide more concrete, here are a few examples of predictions:

1. Predict the outcome of an election, or sports event.

2. Predict some condition that is useful: whether a train will be late, whether there will be traffic congestion, whether it will rain.

3. Predict what a user might like, based on their consumption patterns, like a book, movie, or song recommendation site.

Here are some examples of using inference to provide value:

1. Making sense of data, like a dashboard showing pace and heart rate from a running watch.

2. Making data easily explorable, like a dashboard that visualizes crime reports and other data for a city.

3. A report that explains why a party won the latest election, or why there is more crime in one city, or, why there is more traffic congestion in some places.

When you think of what value you want to provide, also keep in mind the goal we wrote down at the top. A project does not have to be unique. It does not have to be the best. It does not have to be interesting (ok, it does need to be interesting to you).

Concrete Steps to Get Started #

The best way to make a good project and have a good time is to take shallow passes across all of the parts, ensuring that you can keep everything together. Otherwise you might invest a lot of time building a beautiful site, only to later realize that the data or method chosen can’t actually provide the value the site claims.

It’s a little bit tricky, because you have to do a quick pass over each of the parts to see if they will work together.

So far we have not talked about the concrete steps that you need to take. They are the following. To see if they all fit together, you need to take shallow passes over each one, so that you can get to the end. This is the end-to-end iteration. It helps to also make quick decisions about some parts and then “hold them constant”. In this loop, you can start by not worrying too much about how to present the value. Once you have steps 1-3 down, with good data and good value being extracted, you can dive into how to present it in the best way.

1. Deciding on what value we want to provide.
2. Finding some data.
3. Extracting the value.
4. Presenting the value.

Deciding on the value you will provide #

First, you need to decide what value you want to provide. Use these questions to narrow it down.

1. Is it going to be through prediction or inference?
2. What will it be concretely? Try to write this out as precisely as you can.
3. Think about what parts of your value statement can be flexible.

Now write out a value statement:

My project will ____________________________
____________________________________________

Once you have filled yours out, have a look at this example:

Finding Data #

Once you have written out your value statement, it’s time to look for data. To deliver any value, you must have at least two variables: your outcome variable that you want to predict or understand, and at least one predictor. If you are doing a time series you only need the outcome, because you use earlier observations of the same variable as predictors.

So start by doing a web search for data on your topic. Keep in mind whether the data you find is for your outcome, predictors, or both.

For the example above, I did the following web searches:

If you can’t find any data on your specific topic, try to make it more general. For example, if we couldn’t find data on your topic for your country, you could look for the same data for another country. Try both by removing and adding search terms.

Notice how changing the search terms, and, looking for slightly different data, is actually also changing your value statement.

Hopefully by going through this cycle a few times you will have a good value statement and a list of datasets you can use. Make note of both. That should be enough to get you started.

Extracting value #

Once you have locked in your project goal, the way you will provide value, and, the data you will use, actually extracting the value is relatively straightforward. It’s still a lot of work, but, it requires applying well-defined techniques that there are lots of ways to learn.

If you are going to provide value through prediction, then, extracting the value is all about fitting models and minimizing your prediction error. If you are going to provide value through inference, you have more choices to make, but, it still a relatively straightforward process of choosing the right inference method. There are already lots of excellent resources on how to do this out there.

If you don’t know where to get started, I recommend the following:

• For prediction: An Introduction to Statistical Learning
• For inference: The Effect: An Introduction to Research Design and Causality

You don’t have to learn everything in those books before continuing with your project, just enough that you can try applying one of the methods described to your data and value statement.

Presenting the value #

Once you have a way of extracting some value from your data, in a way that fits with your value statement, you need to think about how to present it.

To start, I recommend keeping this part simple. Use those books above as examples, they have many plots and tables both explaining how the methods work and presenting the value they extract. Start with a few of those. Here is an example from an event study in the effect book.

In upcoming posts we will go through how to apply all these steps at a very concrete level.