The carbon footprint of GPT-4

With recent news alerting us that global average temperatures keep rising [1] it’s important to remind ourselves that most human activities have a carbon footprint that contributes towards global warming and other climate change. This is also true for digital technology in general and AI in particular. This article serves as a reminder of this, as it estimates the carbon emissions of training OpenAI’s large language model GPT-4.
To make such estimates, we need to know:
- How much electricity was used to train GPT-4
- The carbon intensity of the electricity, i.e. the carbon footprint of generating 1 KWh of electricity
Let’s dive right in.
Join Medium with my referral link – Kasper Groes Albin Ludvigsen
The electricity consumption of GPT-4
Let’s first estimate GPT-4’s energy consumption. According to unverified information leaks, GPT-4 was trained on about 25,000 Nvidia A100 GPUs for 90–100 days [2].
Let’s assume the GPUs were installed in Nvidia HGX servers which can host 8 GPUs each, meaning 25,000 / 8 = 3,125 servers were needed.
One way to estimate the electricity consumption from this information, is to consider the thermal design power (TDP) of an Nvidia HGX server. TDP, denoted in watts, expresses the power consumption of a piece of hardware under maximum theoretical load [11], ie the actual power consumption may differ.
Unfortunately, Nvidia does not disclose this information, so let’s instead use the TDP of the similar Nvidia DGX server, which is 6.5 kW [3]. So, if an Nvidia DGX server runs at full power for 1 hour, it will have consumed 6.5 KWh according to the TDP.
Recall that it’s estimated that it took 90–100 days to train GPT-4. That’s 90 or 100 * 24 = 2,160 to 2,600 hours per server. If we assume the servers ran at full power constantly, we can multiply the number of hours by 6.5 kW, and we get that during training, each server may have consumed 14,040 to 16,900 KWh of electricity.
Let’s multiply that by the 3,125 servers needed to host 25,000 GPU’S: 3,125 * 14,040 to 16,900 KWh = 43,875,000 to 52,812,500 KWh.
When calculating the energy consumption of computer hardware, it is customary to multiply the energy consumption of the hardware by the so-called power usage effectiveness (PUE) of the data center in which the hardware runs (see e.g. [4]). PUE is a ratio that describes how efficiently a computer data center uses energy. Let’s assume GPT-4 was trained in a Microsoft Azure data center because of OpenAI‘s partnership with Microsoft. Microsoft Azure data centers have an average PUE of 1.18 [7], but note that this can vary between data centers.
So, let’s multiply the 43,875,000 to 52,812,500 KWh hardware electricity consumption by 1.18. That gives us 51,772,500 to 62,318,750 KWh. I.e. to train GPT-4 may have used between 51,772,500 and 62,318,750 KWh of electricity.
This concludes our estimate of the energy consumption from training GPT-4. Now let’s estimate the carbon footprint of training GPT-4.
#
Above, we estimated GPT-4’s training electricity consumption to be between 51,772,500 and 62,318,750 KWh.
To convert that to its carbon footprint, we need to multiply by the carbon intensity of the electricity used to power the compute.
I want to stress that we don’t know how the power was generated, so we don’t know its carbon intensity, but we can make some assumptions.
Let’s assume GPT-4 was trained in an Azure data center because of OpenAI’s partnership with Microsoft. According to researchers, the lowest carbon intensity of an Azure data center in the US is of the one in California called West US [5]. This has a carbon footprint of 240.6 gCO2e/KWh meaning that generating 1 KWh of electricity in this region emits 240.6 grams CO2e on average.
Thus, we can estimate the carbon footprint of training GPT-4 to be between 12,456 and 14,994 metric tons CO2e if the model was trained on "normal" grid electricity in California.
If OpenAI trained the GPT-4 in a data center in the Canada East region – which boasts a carbon intensity of just 20 gCO2e/KWh, the lowest of all Azure regions – the carbon footprint would be 1,035 to 1,246 metric tons CO2e. The difference is shown is Figure 1 below.

Discussion
This article estimates that training GPT-4 may have emitted upwards of 15 metric tons CO2e. That’s the same as the annual emissions of 938 Americans [8]. Or 0.0000375 % of global emissions assuming global annual emissions of 40 billion tons [9]. That may not be a lot, but it pales in comparison to the hardware manufacturing emissions and the carbon footprint from serving a model like that to a large user base. This is something I’ve written about in my articles Environmental impact of ubiquitous generative AI and ChatGPT’s electricity consumption.
Above, I estimated GPT-4’s electricity consumption to be between 51,772,500 and 62,318,750 KWh. For comparison, it’s estimated that training GPT-3 consumed 1,287,000 KWh [6] as seen in Figure 2. So, if the estimates made here are in the right ballpark, the electricity consumption from training GPT-4 may be around 40–48 times higher than the electricity needed to train GPT-3 although GPT-4’s total parameter count is said to be roughly 10 times that of GPT-3. Obviously, the electricity required to train a model depends on many other factors than the model’s parameter count, but this serves as a reminder that we can’t necessarily say anything about the energy consumption of training some model based on knowledge of the energy consumption of training similar models.

The results shown above showed that carbon emissions could differ by a factor 13 between Azure cloud regions. This clearly shows the huge environmental benefit of training your models in regions with greener energy – something I’ve also touched upon here:
How to estimate and reduce the carbon footprint of machine learning models
Furthermore, please note that the results obtained here are based on unverified information about the number of GPUs used to train GPT-4. For this reason — and because assumptions e.g. regarding PUE, the hardware utilization rate and carbon intensity are made — the results obtained here should be considered educated guesswork. If the unverified information turns out to be true, I believe the numbers presented here are in the right ballpark, but please challenge my assumptions.
Conclusion
This article estimates that training GPT-4 consumed between 51,772,500 and 62,318,750 KWh of electricity and emitted 12,456 and 14,994 metric tons CO2e if trained in California and 1,035 to 1,246 metric tons CO2e if trained in eastern Canada.
Although these numbers may seem small, they pale in comparison to the environmental impact from other stages of an AI model’s life cycle, e.g. the deployment stage.
The carbon footprint estimates presented here can be used by organizations or individuals interested in calculating the total global carbon footprint of AI.
Another interesting finding is that the estimates clearly show the environmental benefit of taking into consideration the carbon intensity of the electricity of the cloud region in which models are trained. When comparing the electricity consumption of GPT-4 to GPT-3’s electricity, we also see that the difference in their electricity consumption is much larger than their difference in size.
That’s it! I hope you enjoyed the story. Let me know what you think!
Get the benefits of Medium and support my writing by signing up for a Medium membership HERE.
Follow me for more on AI and Sustainability and subscribe to get my stories via email when I publish.
I also sometimes write about time series forecasting.
And feel free to connect on LinkedIn.
References
[1] https://edition.cnn.com/2023/07/05/world/hottest-day-world-climate-el-nino-intl/index.html
[3] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-dgx-a100-datasheet.pdf
[4] https://arxiv.org/abs/2211.02001
[5] https://github.com/mlco2/impact/blob/master/data/impact.csv – MIT license – "Permission is hereby granted, free of charge, to any person" [10]
[6] https://arxiv.org/ftp/arxiv/papers/2204/2204.05149.pdf
[8] https://www.forbes.com/sites/robtoews/2020/06/17/deep-learnings-climate-change-problem/
[9] https://www.iea.org/reports/co2-emissions-in-2022
[10] https://github.com/mlco2/impact/blob/master/LICENSE
[11] https://www.intel.com/content/www/us/en/support/articles/000055611/processors.html