this time, I remembered that Databricks offered a certification for this topic, so I figured, why not take it? Since I was already waist-deep in Apache Spark topics before studying, the exam material was straightforward. This was not an exam I intended to take, but I’m happy that I went through with it.
The most important exam sections are Apache Spark Architecture, Spark SQL, and using the DataFrame API. For understanding Spark SQL and the DataFrame API, it comes down to practice. It's best to perform different operations with SQL and PySpark to understand how to work with data. The best path is getting your hands on a good enough dataset to practice these operations. The Spark API Reference is worth glancing at to understand how different functions work.
Questions about Spark’s architecture came up many times in the exam. You should be familiar with the basic facts about the architecture and build your knowledge from there. However, it’s not enough to know the general ideas about how Spark architecture works. You should try to read through different scenarios that require you to understand Spark’s architecture, recognize potential issues, and make optimizations when necessary. I find that the answers to these questions to be simple in isolation but context-dependent.
Out of all the Databricks exams, this one had the most immediate effect on how I approach my work. Knowing key details of Spark optimizations matters when you’re doing data operations at scale. These are details that can save time and money.
There are many resources out there to help dig deep into all things Apache Spark. I would recommend some courses on Udemy because they’re usually comprehensive. I linked the ones I used below.