The U.S. National Oceanic and Atmospheric Administration (NOAA) stores extensive weather and climate data. Learn how to write a Python program to call an API to retrieve the data and analyze it with Tableau Public or another visualization tool.
Weather and Climate data are valuable resources, not only to forecast the weather but for dozens of purposes across industries, governments, and in personal life. Opportunities to apply data analytics and data science techniques to these data to inform and support decision-making are almost endless.
NOAA’s National Centers for Environmental Information (NCEI) is the world’s most extensive weather and climate data collection. Its databases contain over 37 petabytes of atmospheric, coastal, oceanic, and geophysical data. This article describes the data, how to download its data files, and instructions to acquire it with a Python program that calls its Data Service API. It also demonstrates a Tableau Public dashboard to visualize a sample set of wind data retrieved with the API.
Weather and Climate Data Uses
Weather and climate data serve a multitude of needs in society. Here are some examples:
- Agriculture – Farmers can use weather data to determine when to plant and harvest crops, apply fertilizer and herbicides, and so on.
- Allergies – Weather plays a significant role in allergy seasons, especially for those related to pollen and other plant materials.
- Building and road construction – Weather may impact outdoor construction work.
- Civil Engineering – Engineers who design structures such as bridges, roads, and tunnels, must be aware of weather effects on them and their materials over short and long periods.
- Climatology – Climate scientists use weather and climate data to understand past, present, and potential future climates.
- Clothing and fashion – The types of clothing we wear vary with the season and weather conditions. Some people choose their style and colors by season.
- Emergency management – Emergency managers use weather and climate data to determine the potential risks of catastrophic events, such as wildfires, tornadoes, hurricanes, and floods.
- Forensic meteorology – Forensic meteorologists analyze weather data to piece together past weather events.
- Forestry and Land Management – Foresters use weather and climate data to manage forests, understand the effects of droughts, mitigate the risks of fires, and more. The U.S. Forest Service, for example, operates an extensive network of automated weather stations.
- Migration of People – Weather and climate are factors that people use to help them determine where to live.
- Outdoor recreation – Outdoor activities such as hiking, sailing, skiing, camping, fishing, and hunting vary by season and weather. People evaluate historical climate and weather data and current conditions and forecasts to choose good times for their outdoor activities.
- Product sales and marketing – Retailers, for example, use weather data to determine when to stock certain types of goods, such as bathing suits or snow skis, in their stores.
- Shipping – Shipping may be affected by the weather. For example, the shipment of grain and other bulk goods in barges on inland waterways that may freeze in the winter is affected by the weather.
- Tourism and travel – People rely on weather and climate data to plan short and long trips to their chosen destinations.
- Wave activity and hydrodynamic impact – Data from weather instruments mounted on buoys can help project the effects of waves and weather on coastlines and beaches.
- Weather forecasting – Meteorologists use climate and weather data to help them predict future weather.
Data Overview
NCEI is the leading authority for environmental data in the U.S. It manages extensive archives of atmospheric, coastal, geophysical, and oceanic research data.
NCEI manages a variety of data products, many of which are available to the public. Too numerous to mention, and beyond the scope of this article, here are some examples of its data product categories:
- Air temperature and atmospheric properties
- Arctic and sea ice
- Ecosystems and natural resources
- Geomagnetism
- Natural hazards, disasters, and severe weather
- Ocean climate
- Precipitation
- Space weather
- U.S. and regional climate
Download Weather and Climate Data
In addition to application programming interfaces (APIs), NCEI provides tools to find, request, and download weather and climate data.
Find a Station and Order and Download its Data
Weather observing stations located in the United States and worldwide capture much of the weather data managed by NCEI. Navigate to the Data Tools: Find a Station page to search for stations by city, county, state, ZIP code, or country.
In the example shown below, I use the _Find a Statio_n tool to search a location of San Jose, California, the Daily Summaries category, a date range 2000–01–01 through 2020–12–31, and the Wind data category. Note the observation station symbols in Fremont and San Jose, California, on the map.
When I click on the station symbol in San Jose, a STATION DETAILS window pops up, as shown below. Note the component of the ID value to the right of "GHCND:." This value, "USW00023293," is the observing station’s unique identifier at the San Jose International Airport. This ID will be used later in a call to the NCEI Data Service API to retrieve weather data captured by this station programmatically.
To start the process to retrieve this data, click on the [ADD TO CART] button. Then, click on the [Cart (Free Data)…] button in the screen’s upper-right corner.
On the Cart: Daily Summaries page, select the "Custom GHCN-Daily CSV" format. This selection will result in the download of wind data in a comma-separated value (CSV) file. The file format will be similar to the CSV file that a Python program will save after calling the Data Service API.
Review the search values, including the "Date Range." When ready, click the [CONTINUE] button.
On the Custom Options: Daily Summaries page, make the selections shown in the screenshot below. Note that a Units value of "Standard" will return wind speed values in miles per hour (MPH). Selecting "Metric" would return wind speed values in meters per second.
The "Average wind speed (AWND)" data type will also be used later in the article to demonstrate visualizing the data in Tableau Public. The "Fastest 2-minute wind speed (WSF2)" and "Fastest 5-minute wind speed (WSF5)" data types have been selected only for demonstration.
The "AWND," "WSF2," and "WSF5" data type values will be used later in a call to the Data Service API.
When presented with the Review Order page, review your selections and enter your email address twice. When ready, click on the [SUBMIT ORDER] button. Within seconds or minutes, NCEI will send you an initial email to confirm the order submission and a second email to confirm its completion. The completion email contains a link to download a data file that meets your selection criteria.
Open the second "Order Complete…" email. Click on the [Download] link to download the data file.
![Partial Climate Data Online request completion email. Click on the [Download] button to download the data file. Image captured by the author.](https://towardsdatascience.com/wp-content/uploads/2021/04/04DLpTIIBubH4YzQI.png)
Open the downloaded CSV file in a text editor. It should look similar to the file shown below. Note that the column names (called "data types" by NCEI) in the first row correspond with comma-separated data values in subsequent rows.
Other Data Search Toolsets
Searching for observation stations and selecting and downloading their data is one of many ways to retrieve data from NCEI datasets. Use the links on the Climate Data Online page to explore other methods.
NOAA also offers the OneStop data search platform to find and view a variety of data interactively. Some data are available for download.
Calling the Access Data Service API with Python
NCEI provides four RESTful APIs to acquire weather and climate data. This article describes how to use the Data Service API to obtain daily summary data for several weather stations.
The Data Service API retrieves data based on parameters sent to a web server with a predefined URL. See the brief Data Service API documentation for an overview of how it works.
URL and Parameters
To use the Data Service API, append parameters to this predefined URL:
https://www.ncei.noaa.gov/access/services/data/v1
The sample code described in the next section will query the daily-summaries dataset to retrieve the daily average wind (‘AWND’) speed value for several U.S. locations for each day between January 1, 2000, and December 31, 2020. The API returns AWND and other wind values in miles per hour (MPH). To retrieve the desired data, append these parameters to the URL shown above:
- dataset – The ‘daily-summaries’ dataset provides summarized daily weather data, including wind data.
- dataTypes – We are interested in daily average wind speed (‘AWND’) in this example. But for possible future use, the parameter value is set to obtain the fastest 2-minute windspeed (‘WSF2’) and the fastest 5-second windspeed (‘WSF5’). To get these three types, set the dataTypes parameter value to ‘AWND,WSF2,WSF5.’
- stations – In this example, we are interested in the daily average wind speed for these stations: Chicago O’Hare International Airport, Rochester International Airport (Minnesota), and San Jose International Airport. Provide these stations’ identifiers to the API in this comma-delimited list: ‘USW00094846,USW00014925,USW00023293.’ Search for station IDs in the Data Tools: Find a Station page as described in a previous section of this article.
- startDate – Records from January 1, 2000, through December 31, 2020, will be retrieved. So, in this example, a value of ‘2000–01–01’ will be set.
- endDate – Since we’re interested in data through 2020, set this value to ‘2020–12–31’.
- boundingBox – Set the bounding box values to select data from a geographic location contained within the coordinates. It includes four comma-separated numbers. North and South range from -90 to 90, and East and West range from -180 to 180. For this example, set boundingBox to ’90,-180,-90,180′.
- units – Values are "metric" or "standard." For the average wind speed data type, the "standard" value will return the wind speed in miles per hour.
Here is the full URL that resulted from appending the parameters to the base URL:
Python Code
Class c_ncei_data_service_api
The file c_ncei_data_service_api.py defines the c_ncei_data_service_api class. The following sections describe its functions.
init() constructor function
The init() constructor function is called when a program creates an instance of the c_ncei_data_service_api class. It takes these input parameters:
- dataset – The name of the NCEI dataset that contains the desired data.
- data_types – The fields to return from the dataset. It consists of a comma-delimited list of field names.
- stations – These are unique identifiers of observation stations of interest. The parameter consists of a comma-delimited list of station IDs.
- start_date_time – The first date of the data to retrieve.
- end_date_time – The last date of the data to retrieve.
- bounding_box – See the description for boundingBox in the previous section.
The function performs these steps:
- Set variable self._base_api_url to the value of the base URL of the NCEI Service Data API.
-
Call self.call_api() with the input parameters to retrieve the requested data.
Note: By default, a call to the URL will return data in CSV format. See documentation for the parameter to set to retrieve the data in JSON and other types.
call_api() function
The call_api() function retrieves data from the specified NCEI dataset. It performs these steps:
- Append parameters to the base URL to create the full_url variable. Note that it sets the API’s ‘units’ parameter in addition to the function’s input parameters.
- Submit the URL with the requests.get() function. The resulting dataset, if any, will be returned into the ‘response’ variable.
-
Return the ‘response.text’ variable to the constructor.
get_data() function
The get_data() function returns the dataset data to its caller.
write_data_file() function
The write_data_file() function writes the data returned from the URL call and stored in the self._dataset variable to a text file. It performs these steps:
- Open the file specified in the parameter in write mode.
- Write the value in the self._dataset variable to the file.
-
Close the file.
Controller Module
The controller module imports the c_ncei_data_service_api module and performs these steps:
- Create an instance of the c_ncei_data_service_api class with the parameters described in the init() constructor description. This step will retrieve data from the NCEI dataset.
- Print the retrieved data to the display.
-
Write the retrieved data to the specified file.
Possible Class and Program Improvements
I developed the c_ncei_data_service_api class and controller module for demonstration and learning purposes. Here are some ideas to add features and make the modules more robust and production-ready:
- Add error-handling to gracefully recover from errors and retry parts of the code if needed.
- Write information about the program’s operations, such as the date and time when it starts and finishes, error information, input parameters, and the number of records retrieved, to a log file.
- Modify the call_api() function to handle more input parameters, such as the desired output data format (CSV, JSON, PDF, and others).
Running the Program
Run the controller module to create an instance of the c_ncei_data_service_api, retreive the desired data from the NCEI Data Service API, display the data, and write it to a file. The output file should appear similar to the sample in the output _weatherdata.csv file shown below. The first row contains column headers, while subsequent rows contain data.
Other NCEI APIs
While this article explores the use of the Data Service API, NCEI has provided these APIs, too:
- Search Service API – Discover datasets and data based on a set of parameters.
- Access Support Service API – Discover metadata and attributes about datasets based on a set of parameters.
- Access Order Service API – Retrieve information about previous orders based on a set of parameters.
Learn more about NCEI datasets on the Access page. It provides links to data discovery tools, data access information, and developer tools, including the APIs, map service directories, climate data web services, and the official NOAA GitHub repository.
Analyzing Data in Tableau Public
Chicago, Illinois, is known as "The windy city." I live in southeastern Minnesota, which is a windy place, too. To visually compare the Chicago areas’ daily average wind speeds with those of Rochester, the largest city in southeastern Minnesota, I loaded the data retrieved with the Data Service API in Tableau Public. There, I created a _Daily Average Wind Speed_ dashboard. My son lives near San Jose, California, so wind speeds at San Jose International Airport are shown, too.
The results surprised me. Wind speeds at Rochester International Airport (Minnesota) are consistently higher than at Chicago O’Hare International Airport. Both have winds much greater than San Jose International Airport.
From 2000 through 2020, the locations have these daily average wind speeds in miles per hour:
- Rochester International Airport (Minnesota) – 11.31 MPH
- Chicago O’Hare International Airport – 9.52 MPH
- San Jose International Airport – 6.44 MPH
The dashboard also shows daily average wind speed by year, month, month by year, and season. Meteorological seasons in the northern hemisphere are defined as follows.
- Spring – March, April, and May.
- Summer – June, July, and August.
- Fall – September, October, and November.
- Winter – December, January, and February.
According to the data, spring and winter are the windiest seasons in Rochester and Chicago, and summer is the calmest. San Jose has relatively light winds year-round.
The dashboard graphs prepared for this article represent a fraction of the possibilities for data visualizations in tools like Tableau Public, Tableau, and Microsoft Power BI.
Summary
Weather and climate data are valuable assets that data analysts and data scientists can leverage for individuals and organizations to apprise them and support decision-making on various fronts. With over 37 petabytes of data, the NCEI service of U.S. NOAA manages tremendous assets. It also provides a variety of user-friendly search and download tools, along with a set of APIs, to make the data available to anyone.
Feel free to use and modify the c_ncei_data_service_api Python class and the controller module to query various weather datasets to meet your needs. Accessing the data through APIs and programs provides a way to integrate weather and climate data into a data analytics or Data Science solution, a website, or a software application.
Randy Runtsch is a writer, data engineer, data analyst, programmer, photographer, and cyclist. He and his wife live in southeastern Minnesota in the U.S.A.
Randy writes extensively about leveraging data from public data sets. See the recent article, Use Public Datasets Cataloged on Data.gov to Power Data Science Projects, to learn about the collection of 280,000+ government data sets cataloged on data.gov.
