In June 2024, Databricks held their annual Data & AI Summit, during which they announced a host of new features and upgrades. One of the biggest announcements of the weekend was the general availability of Serverless Compute for Notebooks, Workflows and Delta Live Tables. Serverless became GA (general availability) in July 2024 (with some regions yet to be updated).
What is Serverless?
Serverless offers almost instantaneous compute power for workloads across the Databricks platform, including notebooks, jobs, and delta live tables. Serverless offers an alternative to the classic cluster compute method. With Serverless, compute resources run within your Databricks account in the same region as your workspace.
Why Use Serverless?
Speed: Almost-instant compute via Databricks pools means no more waiting for (or being charged for) clusters to start up.
Simplicity: Automatic scaling to workloads and spark versioning means less complexity in choosing cluster configs
Reliability: Automatic instance type failover and a ‘warm pool’ of instances offer shields from availability shortages and cloud outages
Elastic Costs: Elastic billing modal and autoscaling to required capacity means no costs for idle time or over-provisioning.
What Should I Consider Before Transitioning to Serverless?
Costs: Serverless currently has a higher DBU-Cost per hour than classic compute methods. That being said, Databricks have expressed that lower serverless costs are a future goal. There is also the potential for future networking charges.
Unity Catalog: Serverless requires Unity Catalog (UC). If not currently enabled, the workspace will require upgrading. However, UC has many benefits of its own and the upgrade is cost-free.
Impact on Codebase: Currently, Serverless supports both SQL and Python but there is no support for R or Scala. Even in Python and SQL, some codebase will require changes given a few limitations of serverless.
Existing Setup: Any current prepurchase agreements or reserved instances. For those not yet ready to transition, classic compute is still available, offering Databricks customers both options.
Should I Transition to Serverless?
Serverless compute is a great option for improving speed and simplicity, but it might not be the right move for everyone just yet. To help you decide, Coeo offers a Databricks Health Check, providing a detailed review of your setup, costs, and performance.
Coeo’s Databricks Health Check assesses your organisation’s readiness to transition to serverless. Our evaluation covers your current setup, the roadmap to serverless, and the immediate benefits of switching. You’ll receive a comprehensive report and a workshop to map out your path to serverless, ensuring your data platform supports your long-term goals effectively.
Ready to assess your transition? Get in touch with our team today to schedule your Databricks Health Check or find out more about it by downloading the quick guide.