Welcome to our tutorial on configuring Application Default Credentials for Google Cloud and Python. In this article, we will cover how to properly set GOOGLE_APPLICATION_CREDENTIALS in Python.
In order to be able to programmatically interact with Google Cloud Platform services, such as Google BigQuery, you first need to properly authenticate the application and grant all the required permissions. This is achieved by defining Application Default Credentials to point to a file with the required credentials.
A commonly reported error when missing this step is the following
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.Subscribe to Data Pipeline, a newsletter dedicated to Data Engineering
How Application Default Credentials work in Google Cloud
The Application Default Credentials (ADC) is the strategy used on Google Cloud in order to infer credentials based on the application environment. This means that the application code can run in different environments without requiring to change the way your code authenticates to GCP services or Application Programming Interfaces (APIs).
For local development, there are typically two different ways to provide credentials to ADC:
- User Credentials
- Service Account keys
Creating the credentials JSON file
In order to create the JSON file containing the required credentials, you first need to ensure you have gcloud CLI installed on your host machine.
Now for local development, the best option you have is to use user credentials which are associated to you personal Google Cloud account. __ To do so, you’ll have to run the following command, that will display a login prompt on your (default) browser:
gcloud auth application-default loginOnce you login to Google Cloud, your credentials will be stored in a JSON file, under the following default locations:
- Mac/Linux:
$HOME/.config/gcloud/application_default_credentials.json - Windows:
%APPDATA%gcloudapplication_default_credentials.json
Alternatively, if you are using a Service Account, you can generate the JSON token by visiting the Service Account service on GCP. Note however that service account keys create a security risk and are not recommended. Some more powerful and perhaps more secure approaches include impersonation and Workload Identity Pool.
Setting GOOGLE_APPLICATION_CREDENTIALS env variable
In order to provide the location of the credentials JSON file, you need to make use of the GOOGLE_APPLICATION_CREDENTIALS environment variable.
Therefore, when working with Python, you can programmatically set the environment variable using the code snippet below:
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] ='$HOME/.config/gcloud/application_default_credentials.json'Alternatively, you can also create an instance of google.oath2.service_account.Credentials and then pass it to the Google client before start interacting with it.
The following example, demonstrates how to authenticate the Gmail Client in Python:
from google.oauth2 import service_account
from googleapiclient.discovery import build
credentials = service_account.Credentials.from_service_account_file(
'$HOME/.config/gcloud/application_default_credentials.json'
)
service = build('gmail', 'v1', credentials=credentials)Note that the above code snippets assume that your JSON credentials file is stored under the default directory when creating them with gcloud. Make sure to point to the right directory if this is different from the default one.
Final Thoughts
In conclusion, this tutorial covered how to properly set Application Default Credentials (ADC) for Google Cloud and Python in order to authenticate the application and grant all the required permissions for programmatic interactions with Google Cloud Platform services.
The ADC is a strategy used on Google Cloud to infer credentials based on the application environment, allowing for code to run in different environments without requiring changes to the authentication process.
In this tutorial we also covered how to create the required JSON credentials file, either by using user credentials or a Service Account, and how to set the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the location of the file.
Subscribe to Data Pipeline, a newsletter dedicated to Data Engineering
Related articles you may also like
