What Do You Need to Know to Be a Data Scientist?
Table of Contents
How do I become a data scientist?
Python, pandas, RAG, AUROC, cross-validation, trees and ensembles? Where do I even begin?
The scientific method is the most powerful tool for defining and solving problems we humans (and our robot servants) know of. But in an actual data scientist job you cannot succeed by scientific method alone, you also need some practical knowledge and skills to actually do your job. This guide lists everything you need to know at a minimum to be a data scientist.
Use this list as a guide to see where you currently stand, and, as a first goal to work towards. It will give you enough to be able to provide value as a data scientist, whether it’s in your own projects, at school or at work.
An effective data scientist relies on their knowledge and skill in several areas when applying the scientific method: programming, math, statistics, causal inference, machine learning, and project and process methods.
Who this is for #
This guide is for anyone who wants to know what a data scientist needs to know. If you want to become a data scientist, use this as a roadmap. If you already are a data scientist, use this to identify gaps and be inspired about what next to learn.
How to use this guide #
This guide lists everything you will need, but, it does not by itself teach it to you. There are many excellent resources out there. Choose ones that fit your style, whether it be textbooks, tutorials, videos or courses.
I recommend a very hands-on approach when learning the material. First, read the list and check your own status against it. What do you already know, and, what are your gaps? Once you’ve identified those gaps, you know what to focus on. You can then tackle those areas, going relatively deep, knowing that this will be productive time spent.
The best way I have found for learning this type of material is to alternate between focused learning (either through a book, course, tutorial, video, etc) and then doing your own hands-on projects where you apply those skills. For each section I’ve added a short instruction on how I think the material can be best learned.
Some of these areas are free-standing, some depend on each other. The dependencies are marked, and when you see them keep in mind that you should check if you know the prerequisite parts first. Also note that the table rows below vary a lot in scope: don’t allocate the same amount of time to each.
Most importantly of all, this guide presents the bare minimum of what you must know: don’t treat this as the end, treat it as the beginning.
99 Project Ideas
Get the full list of 99 data science and ML project ideas, filterable by difficulty and domain. We won't send you spam. Unsubscribe at any time.
Programming #
Why do data scientists need to know programming and some software engineering? There are three main reasons. First, we need to know SQL to be able to extract and manipulate data, in an industry setting data often lives in databases that use SQL. Second, once you have that data extracted, you need to explore, visualize, clean it, and then train and evaluate models on it. For this we use a general purpose data science language, almost always R or Python. These languages have libraries that make these data science tasks easy. The third reason is that programming is how we make our work actually powerful: it’s how we can make our reports, models, findings etc durable and automated, using the true power of software. For this step we need a mix of different languages and techniques. Lastly, knowing some software engineering allows us to write and use code more efficiently and effectively. We can for example wrap up functions for common tasks in a library that we can reuse, saving time and effort.
| Topic | So you can | Concretely |
|---|---|---|
| SQL | Extract and manipulate data | Joins, window functions, aggregations/group by, CTEs/subqueries |
| Python/R | Explore, clean, manipulate data; train and evaluate models | pandas/tidyverse, matplotlib/ggplot, scikit-learn/tidymodels |
| Software engineering | Write reusable and correct code | Version control, TDD, encapsulation and modularity, time complexity, algorithms and data structures, CI/CD |
| Glue | Deploy a model to production in industry (using existing infrastructure) | Docker, infrastructure as code, monitoring, basic bash/linux |
These programming topics can be learned by reading about the methods, then trying them in exercises, and then applying them in more realistic projects.
Math #
Why do data scientists need to know math? And what math do they need to know? While the most basic way to understand data science is the application of the scientific method, the way data scientists do apply that method is through quantitative methods, statistics and machine learning, that are all built using math. So we need to know the math underlying these methods so that we can understand the methods and when to use them. Sometimes, data scientists also do simple mathematical modeling, and math is also helpful there. Lastly, data scientists deal in data, and data is often summarized and understood using simple mathematical methods, understanding these and being fluent in them helps dealing with data.
| Topic | So you can | Concretely |
|---|---|---|
| Numeracy | Do quick sanity checks and communicate results clearly | Percentages vs. percentage points, orders of magnitude, mental math, back-of-envelope estimation |
| Set theory | Reason precisely about the groups and events that probability is built on | Unions, intersections, complements, basic notation |
| Probability | Reason about uncertainty and quantify it | Axioms, conditional probability, Bayes’ rule, common distributions |
| Logarithms | Understand growth rates and interpret log-transformed variables/coefficients | Log rules, log-scale plots, interpreting log-odds |
| Calculus | Understand how models are fit and optimized | Derivatives, partial derivatives/gradients, chain rule |
| Linear algebra | Understand how data and models are represented and manipulated | Vectors, dot products, matrices, matrix multiplication, eigenvalues/eigenvectors (PCA) |
This table is a rough outline, in practice when learning these topics you might find that you have to alternate, or take two passes, at some of these topics. More advanced probability, like maximum likelihood estimation, requires some calculus, and, understanding multivariate calculus’s gradient requires some basic linear algebra (vectors, dot products) first.
The math topics almost certainly require hands-on practice, i.e. doing problems and exercises by hand. Many methods can then also be implemented or explored using simple programs.
Statistics and Causal Inference #
There is a reason for the joke “data science is statistics, but on a Mac”. The core of data science is applying statistics and machine learning to large amounts of data in an applied setting. You must know the math in the preceding section so that you can learn statistics, causal inference and machine learning. And, you must learn programming so that you can effectively and efficiently apply those methods. Statistics helps you quantify uncertainty. Causal inference is that part of statistics that is concerned with making causal claims, saying that one thing caused another. In some ways it is the most valuable, but also the hardest thing to do.
| Topic | So you can | Concretely |
|---|---|---|
| Distributions | Describe and summarize how values are spread | Normal, binomial, Poisson, uniform; mean, variance, skew |
| Confidence intervals | Quantify the uncertainty around an estimate | Standard error, central limit theorem, interval construction/interpretation |
| Hypothesis testing | Decide whether an observed effect is likely real or just noise | p-values, t-tests, chi-square tests, significance levels |
| Bayesian inference | Apply a flexible and powerful alternative to traditional statistics | Priors, likelihood, posterior, credible intervals |
| Experimental design / power analysis | Design an experiment that can actually detect the effect you care about | Sample size calculation, minimum detectable effect, Type I/II errors |
| Linear regression | Quantify the relationship between a continuous outcome and one or more predictors | OLS, coefficients, standard errors/p-values/CIs on coefficients, R² |
| Logistic regression | Quantify the relationship between a binary outcome and one or more predictors | Log-odds, odds ratios, interpreting coefficients |
| Time series analysis | Model and forecast data that changes over time | Trend/seasonality decomposition, autocorrelation, ARIMA, stationarity |
| Randomized controlled trials & experimental analysis | Measure a causal effect with the strongest evidence available | A/B tests, randomization, analyzing experiment results |
| Observational causal methods | Estimate a causal effect when you can’t run an experiment | Panel data / fixed effects, difference-in-differences, regression discontinuity, instrumental variables |
Bayesian statistics is a flexible alternative to traditional (sometimes called frequentist) statistics. If you prefer, you can take a Bayesian approach to all the rows in the table. These topics are most easily learned by reading about them, sometimes implementing them by hand or from scratch, then applying them in simple situations and then expanding their use to more realistic data and situations.
99 Project Ideas
Get the full list of 99 data science and ML project ideas, filterable by difficulty and domain. We won't send you spam. Unsubscribe at any time.
Machine Learning #
Machine learning has many similarities with statistics, but the emphasis is more on making models good at predicting the outcome, as opposed to explaining the outcome, like with causal inference. Models are often larger (more variables) and trained on more data. Because these models can be very flexible and good at fitting the data, how well they generalize to unseen data is a key concern. So this section starts with learning how to measure performance reliably. Then follow different classes of models, by increasing complexity: regression, tree-based methods, and neural networks. Sometimes when doing machine learning the outcome is not defined. These cases call for unsupervised learning, which can be difficult because there is no clear answer on when the model is right or wrong. Lastly, successful machine learning requires knowing when a model is good enough, and, what to do to improve if it does need improvement.
| Topic | So you can | Concretely |
|---|---|---|
| Cross-validation and overfitting | Reliably estimate how a model will perform on new data | Train/test/validation split, k-fold cross-validation, bias-variance tradeoff |
| Regression with regularization | Build a model that generalizes well when you have many or correlated predictors | Ridge, Lasso, Elastic Net |
| Tree-based methods and ensembles | Model non-linear relationships and interactions with little feature engineering | Decision trees, random forests, gradient boosting (XGBoost/LightGBM) |
| Neural networks | Model complex patterns in unstructured data (text, images) or very large datasets | Layers and activation functions, embeddings, when to reach for one vs. classical ML |
| LLMs and generative AI | Apply pretrained foundation models instead of training from scratch | Prompting, fine-tuning vs. RAG, embeddings for retrieval, evaluating generated output |
| Unsupervised learning | Find structure or groups in data without labels | k-means, hierarchical clustering, choosing the number of clusters |
| Evaluation and what to do next | Know whether a model is good enough and what to improve | Accuracy/precision/recall/AUC/RMSE, error analysis, iteration |
Machine learning can be learned by reading about the methods, then sometimes implementing them from scratch, then applying them to simple and toy problems, and then applying them to more realistic data and situations.
Project and process methods #
Some definitions of or guides to data science emphasize domain knowledge. In my view, domain knowledge should not be a study goal in itself. First, it varies from industry to industry and even company to company, so it’s not broadly useful. Second, one reason data science is so powerful is that it can reduce a broad range of what seems like very specific and complicated business problem into a few standard problems, such as inference, prediction or optimization. There are of course exceptions, where the domain is highly specialized and requires extensive study.
Instead of domain knowledge, there are specific techniques and methods that are useful for data science work that are more about how to work with others, how to work in an organization, and, how to organize the work to become more efficient and effective.
These methods and principles are often very easy to learn, but require deliberate practice to apply successfully and consistently.
| Topic | So you can | Concretely |
|---|---|---|
| Clear communication | Make sure your work is understood, used, and trusted by others | Writing clearly, presenting results, tailoring to your audience |
| Limit work in progress and work in small batches | Reduce risk and get feedback faster | WIP limits, small experiments/PRs, incremental delivery |
| Agile and Extreme Programming | Iterate quickly and adapt to changing requirements | Sprints/iterations, standups, pair programming, continuous integration |
| Lean | Apply the scientific method to how you work, not just to your analysis | One-piece flow, kanban, pull-based systems, elimination of waste, continuous improvement (kaizen) |
| Working backwards | Start from the customer/end outcome and design backward from there | PR-FAQ (press release + FAQ before building anything) |
| Critical path | Identify the minimum sequence of steps that determines how fast a project can finish | Identifying blocking dependencies, sequencing, prioritization |
| RACI & DACI | Clarify who owns what, and who decides what | RACI (Responsible/Accountable/Consulted/Informed) for ownership and handoffs; DACI (Driver/Approver/Contributor/Informed) for decisions |
| Blameless retrospectives | Learn from failures and incidents without triggering defensiveness, so real fixes happen | Postmortems, 5-whys, root cause over blame — requires psychological safety to actually work |
| Checklists | Catch preventable mistakes before they cause damage | Pre-deployment checklists, data leakage checks, train/test contamination checks |
These topics can be learned by reading about them and then applying them in your daily work.
Now what #
The beauty of our discipline, data science, is that there is infinite room to learn and develop. You are free to take it in whatever direction you are most interested in. Once you know everything in this list, congratulations! But don’t stop here. Consider this reaching the first rung of the ladder. From here you can reach for the stars.