I recently passed the Databricks Certified Engineer Professional certification exam. It took me a couple of months of studying alongside my daily job duties. While I had passed the associate exam a year earlier, I can't fully explain the gap between earning the two certifications. I suspect I wasn't as ambitious or curious when I first completed the associate exam. Looking back, I would have studied for both the associate and professional certifications simultaneously—just as I did with the Databricks Machine Learning Associate and Professional certifications (more on that in separate blogs).
Exam Information
The exam measures your ability to use Databricks to perform basic to advanced data engineering tasks, primarily on the Databricks platform. The exam website guide provides a clear summary of what the exam assesses and the exam format. I strongly recommend downloading the PDF version of the exam guide and studying it thoroughly. One particularly useful feature of the guide is its detailed outline of required knowledge, complete with percentage breakdowns of topic coverage. This helped me prioritize what topics I should focus on while studying.
The exam covers the following topics:
- Databricks Tooling — 20%
- Data Processing — 30%
- Data Modeling — 20%
- Security and Governance — 10%
- Monitoring and Logging — 10%
- Testing and Deployment — 10%
Study Process
For learning material, my main sources were the Databricks Learning Academy, Databricks Documentation, Udemy and some reference material. I have access to the Learning Academy through my job. If you don’t have access, you can supplement that through YouTube and Udemy. The Udemy course I used was solid and offered more technical examples. The material in the Databricks Learning Academy course was sometimes a little too abstract and didn’t offer more technical/coding examples. The Udemy course helped offset this limitation.
My main learning resources were the Databricks Learning Academy, Databricks Documentation, Udemy, and various reference materials. While I have access to the Learning Academy through my work, those without access can find alternatives through YouTube and Udemy. The Udemy course I took was particularly valuable, offering concrete technical examples that complemented the Learning Academy's more theoretical approach. This was helpful since the Learning Academy's material, while comprehensive, sometimes lacked hands-on coding examples.
I strongly recommend working through each bullet point in the exam outline. For any sections where you feel uncertain, take extra time to study and fill in knowledge gaps. Make sure to complete all available hands-on labs. After covering the learning material and exam outline thoroughly, focus on practice exam questions. The practice exam course on Udemy provides excellent examples that closely mirror the actual exam questions. The similarity between the practice material and the real exam was striking—I got a little frustrated during the test when I realized how closely they matched, especially since I hadn't even finished the practice course! Fortunately, I still passed.
Topic Breakdown
Databricks Tooling
This section covers the underlying logic of Delta Lake tables, optimizations, and partitioning strategies. You should understand the mechanics of Delta Lake transactions and available optimization techniques. While you can find most of this information in both the Databricks documentation and Delta documentation, I found the most valuable resource to be Delta Lake: Up and Running By Bennie Haelen and Dan Davis. This book offers clear explanations of Delta Lake features with practical coding examples.
Data Processing
This section covers optimization, partitioning, and various structured streaming concepts. Take extra time with this section as it encompasses many different topics. Both Structured Streaming and Delta Lake are crucial components to master. You should understand Change Data Capture (CDC), Change Data Feed (CDF), and deduplication—particularly how they function within Structured Streaming. It's essential to grasp why data partitioning can improve performance. You should also understand the small file problem and its solutions. For in-depth coverage of these topics, consult the Delta Lake Up and Running book. The Databricks documentation also provides numerous examples of Structured Streaming and partitioning.
Data Modeling
This section focuses on data transformation across different layers (bronze, silver, gold), including how to propagate updates and deletes, maintain data quality, and handle incremental processing—all within the Databricks Lakehouse framework. I strongly recommend mastering Slowly Changing Dimensions (SCD) tables and their implementation. While I initially found SCD concepts challenging, working through coding examples helped me grasp the core principles.
Security and Governance
This section is relatively simple. Focus on understanding how to create dynamic views in SQL and manage access controls for different groups or users.
Monitoring and Logging
This section is straightforward but has a few tricky points. You should be familiar with the Spark UI and understand what information can be found in both the Ganglia UI and Cluster UI. These tools help you debug failing applications. I saw one or two questions about this topic on the exam, so pay attention to these topics.
Testing and Deployment
This is another straightforward section. It covers topics related to installing Python dependencies on clusters, Python wheels, creating multi-task jobs, and working with both the Databricks CLI and REST API.
Personal Study Guide
I created a personal study guide for this exam. You can access it here. It contains notes that directly correlate to the exam guide and includes coding examples from various sources.
Summary
After completing the study material, you'll be well-prepared for the exam. I cannot emphasize this enough: taking practice exams before the official test will significantly improve your chances of success!
Certifications should not be viewed as an automatic ticket to employment. This is common advice for anyone pursuing technology certifications. While the value of certifications varies depending on the exam and how it's administered, it's unfair to dismiss them as worthless. It's best to view certifications as benchmarks for measuring your knowledge of the material.
In my pursuit of this certification, I learned extensively about basic and advanced data engineering tasks on the Databricks platform. The knowledge I gained has made me a better software engineer overall and boosted my confidence in my abilities. However, hands-on experience solving complex real-world problems will always be more valuable than any certification. In other words, it's better to be the engineer who delivers results rather than one who simply collects certifications.