↓Skip to main content

What is the most important skill for a data scientist?

·4 mins
Learning
Author
Alex
Table of Contents

The beauty of our field is that the opportunity to learn is endless. But this also means that it can be overwhelming when you are just starting out.

What is most important? What should I start with? Where do I start?

Coding? SQL? Machine Learning? Data Structures and Algorithms? Statistics and probability?

Sometimes the necessary skills for a data scientist are described as a combination of programming, statistics, and business knowledge. One famous Venn diagram illustrates Data Science as the intersection of “Hacking Skills”, " Math and Statistics Knowledge" and “Substantive Expertise”, with Machine Learning and Traditional Research off to the side.

Any effective data scientist must have some of each of those three skills. But that doesn’t tell you what to prioritize or where to start.

The defining feature of a (good) data scientist is right there in the definition: a data scientist is a scientist, somebody who applies the scientific method.

Learning and applying the scientific method is the single most powerful way to learn and develop as a data scientist, especially because if you do it right it will also lead you naturally to what else you need to learn! Not only is it helpful by itself, it will also teach you to prioritize what you need to learn next.

Scientific Method #

So what is it?

In general, the scientific method consists of applying the following steps:

  1. Define the problem or question.

  2. Gather information on the problem, question and topic, informing the next step.

  3. Define a hypothesis, a statement that if you knew it was either true or false would help you understand the problem or answer the question.

  4. Collect evidence for and against the hypothesis, through for example an experiment, analysis, backtest or simulation.

  5. Draw conclusions about the hypothesis using this evidence.

  6. Restate the problem or question in light of this conclusion.

The process is iterative, so that once you reach the conclusion and can say something useful about the problem or question, new problems and questions arise. Or, you might find that you need to revise the problem or question as you go through the steps.

Now that you know what it is, how do you actually apply it?

How to do it #

There are lots of ways to learn the scientific method. You can, for example, take a course, or read a book.

But the best is to, once you have a basic understanding of the steps, practice it by applying it to your projects. You can do this by asking a series of questions. Ask these of yourself and of your stakeholders and colleagues. Take careful notes, review, and then in a separate session try to condense and hone down the answers to each. Circulate this as a project brief with the group and get feedback and buy-in.

  1. What is the question we are trying to answer? What is the problem we are trying to solve? The answer to this becomes the problem statement.

  2. What do we think are the answers and solutions? The answers to this becomes the hypotheses.

  3. How can we measure if these answer the question or solve the problem? The answers to this becomes the metrics that the hypotheses are formulated in terms of.

These questions are best asked of and with stakeholders, the remaining ones are best answered by you and your colleagues. Sometimes stakeholders also know about measurement. Get feedback and buy-in on your answers to the ones above before proceeding.

  1. How can we collect evidence for each of the hypotheses? This is where you decide what method to use, whether it is an analysis of existing data, a simulation, a backtest, an experiment, some other causal inference approach. Choose a method that, within the constraints you have, allows you to collect the most and best evidence for your hypotheses.

  2. What does the analysis (using whatever method you decide) say? How confident or uncertain are we about these results?

  3. What are the conclusions and what do they mean for the initial question or problem?

Challenge #

Now that you have a basic understanding of the scientific method, you can apply it to your own learning path. Ask yourself, what problem am I trying to solve? Once you have a clear definition of the problem you have, formulate hypotheses about how to solve it.

For example, “my problem is that I take too long accessing and assembling the data I need for my analyses.”, which would lead to the hypothesis “learning more SQL will make me faster at getting data”, which you can then test: does picking up some more SQL actually help you? and if it does, what is the next problem you are trying to solve?