The world’s leading publication for data science, AI, and ML professionals.

How to Increase Azure Databricks Cluster vCPU Cores Limits

The solution of the warning: "This account may not have enough CPU cores to satisfy this request" for Azure Databricks

Photo by Alexandre Debiève on Unsplash
Photo by Alexandre Debiève on Unsplash

It is quite common to be asked by my customers:

Why my Azure Databricks cluster is limited?

Specifically, for a default PAYG Azure Subscription, when you want to create a cluster in Azure Databricks, you may be bothered with such an issue:

This account may not have enough CPU cores to satisfy this request Estimated available: 10, requested: **

That is as shown in the screenshot below.

Even if you click the link "Learn more about CPU quota", there is no straightforward answers in that documentation. However, if you patiently drill down in the official document, you’ll find out that Microsoft deliberately set these quota limits by default for all the PAYG subscriptions to avoid any shock billings for you.

Nevertheless, it is very inconvenient for Azure Databricks clusters. The number of vCPU cores is limited to 10, which also limited the ability of Azure Databricks. In most cases, the cluster usually requires more than one node, and each node may have at least 4 cores to run (the recommended worker VM is DS3_v2 which has 4 vCores).

Consequently, the cluster will always be limited to have only 1 node running, which become useless as Spark needs its distributed features to expedite the calculations. Please note that the driver node will need another 4 cores if we choose "Same as worker". Therefore, the following scenarios happened:

  • Driver node (4 cores) +1 * Worker node (4 cores) = 8 cores < 10 cores limitation
  • Driver node (4 cores) +2 * Worker node (8 cores) = 12 cores > 10 cores limitation (not allowed)

We have to increase the limitation before Databricks can be operated as expected.

The solution for that is to increase the quota is not tricky, but I found that it is just not directly specified on the official documentation page. Now, I’ll demonstrate how to increase the quota and eliminate the constraints.

Steps to Increase the Quota

  1. Go to Azure Portal **** > Clik in Search Resource Field> Type in "help" > Choose "Help + support"
  1. In the navigation on the left, select "New support request". Then, on the right pane, in the "basics" tab,
  • Issue type: choose "Services and subscription limits (quotas)"
  • Subscription: choose your subscription
  • Quota type: select "Compute-VM (cores-vCPUs) subscription limit increases"

After selecting all the required fields, click "Next: Solutions >>". Please note that there won’t be any "solutions" for this because it requires a support guy to review your request and manually approve it. So, the "Solution" tab will be skipped and directly jump to "Details".

  1. In the "Details" tab, click the link "Provide details" to bring the "Quota details" blade window to the right. Then, in the window:
  • Deployment model: Select "Resource Manager".
  • Location: Select your location(s). Please note that you can request quota increases for multiple locations at one time.
  • Types: Select "Standard".
  • Standard: Select the VM series that you want to increase the quota. Please note that you can request quota increases for multiple VM series at one time
  • New vCPU Limit: Input the new quota that you want to increase up to

After filling in all the required information, click "Save and continue".

  1. Now, the requested quota increases are displayed in the "Request Summary". There is still some other required information to be filled in. Just input based on your preferences, and then click "Next: Review + create >>".
  1. After review everything input, click "Create" to create the ticket.

Response from Azure Support

Depending on the "severity" you selected, a staff member will contact you in a timely manner. Usually, the quota increases will be approved without any issues, and followed by an email from the support team just for confirming before closing the tickets.

Summary & Tips

Photo by Tadas Sar on Unsplash
Photo by Tadas Sar on Unsplash

In fact, Microsoft Azure sets enormous quota limits to various services. You can find the details here:

Azure subscription limits and quotas – Azure Resource Manager

Of course, using the method I have demonstrated in this article, you can increase not only the quota for vCPU Cores but also other services quota that is listed in the official document above.

Now, your Azure Databricks Cluster has no limitations, but think twice before you go, do not produce a shock bill later on 🙂

Join Medium with my referral link – Christopher Tao

If you feel my articles are helpful, please consider joining Medium Membership to support me and thousands of other writers! (Click the link above)


Related Articles

Some areas of this page may shift around if you resize the browser window. Be sure to check heading and document order.