When to use Databricks

If you like this post, you should subscribe. Or, send me an email directly.

I just learned about Databricks, we should start using it.

My company has signed an enterprise deal with Databricks and now we are all learning it.

Most of us have likely heard about Databricks.

But when should you really use Databricks? Only when you absolutely have to.

You should use Databricks when you are sure that you need distributed processing, because your data is too large to easily handle on one machine.

When developing data science applications or just doing data science analysis, the highest value compute is the one you are doing yourself as you are iterating on the model or analysis. This iteration speed is directly bottlenecked by how quickly you can try an idea and get feedback, i.e. how quickly you can run your analysis.

Therefore you should always try to work “in memory”, i.e. in an interactive R or Python session on a single machine, because this gives you the fastest feedback. On a modern machine you can easily work with a few million rows in memory. Starting a Databricks cluster alone can take minutes.

In many cases you can still work in memory, even if the total data available is very large. Just sample (just make sure you sample correctly). It’s only really when you need exact counts, like reporting, that you can’t sample. Otherwise both model building and analysis can be developed on a sample.

Just make sure to periodically run your code on the full data. For this you can and should use Databricks.

But even if you do have Databricks available, my recommendation is to be deliberate about when to use it. Otherwise you will spend a lot of time waiting for cluster upsizing etc., which means that your most valuable compute, yourself, will have less time to try fewer ideas and fewer refinements.

Always work locally when you can. That said, if Databricks speeds up your development feedback loop, your most valuable compute, then you should absolutely use it.