The world’s leading publication for data science, AI, and ML professionals.

ChatGPT’s energy use per query

How much electricity does ChatGPT use to answer one question?

ChatGPT’s energy use per query

Photo by Andrey Metelev on Unsplash
Photo by Andrey Metelev on Unsplash

This article presents a range within which the electricity consumption per query of ChatGPT may fall and compares it to the measured energy consumption of two other large language models (LLMs).

This is an interesting undertaking for two reasons:

First of all, if organizations know how much electricity ChatGPT requires to answer one question, they can approximate the carbon footprint associated with their use of ChatGPT or similar services such as OpenAI’s LLM APIs.

For more than 50,000 European businesses, this may soon become highly relevant, as the coming Corporate Social Responsibility Directive (CSRD) will likely force them to disclose scope 3 emissions in their management reports [1]. I expect usage of services like ChatGPT to fall under scope 3 because cloud compute is considered to be scope 3 [2]. I hope this article can provide inspiration for how to estimate your organization’s scope 3 emissions from ChatGPT and similar services.

Another reason why it’s interesting to look into ChatGPT‘s energy use per query is that it’ll enable individuals to come up with their own estimates of ChatGPT’s total electricity consumption or carbon footprint. As such, I hope this blog post will inspire others to publish similar work.

In the remainder of this article, the terms "query" and "request" will be used interchangeably.

Join Medium with my referral link – Kasper Groes Albin Ludvigsen

Methodology for estimating ChatGPT’s electricity consumption per query

In this section, I’ll present the methodology used to produce estimates of ChatGPT’s electricity consumption per query. The estimates rely on two different methods:

  1. One in which we estimate the total energy consumption of the hardware ChatGPT presumably is running on and divide by the assumed number of daily queries
  2. One in which we use the reported floating point operations (FLOPS) required by GPT-4 to make one forward pass to calculate the energy consumption

Estimating ChatGPT’s electricity consumption per request with method # 1

Below is the formula used to produce estimates of ChatGPT’s electricity consumption via method 1. This formula is a standard way of producing estimates of machine learning model energy consumption (see e.g. [3] [4]):

Energy consumption per query (KWh) = (Amount of hardware * average hardware power draw * TDP * 24 * PUE) / total daily queries

For method 1, I’ll assume the following values:

Amount of hardware: 3,617 Nvidia HGX servers containing a total of 28,936 Nvidia A100 GPUs as per SemiAnalysis’s estimates of how much compute it took to serve ChatGPT’s users when the service was based on the GPT-3.5 Llm [5].

Average hardware power draw: 50 % to 75 %.

TDP (Thermal Design Power): TDP is the maximum theoretical power draw of a piece of hardware, but is often used as a proxy for actual maximum hardware power draw. I’ll assume a TDP of 6.5 kW as that’s the TDP of the Nvidia DGX A100 server which is reasonably similar to the Nvidia HGX A100 server.

Power usage effectiveness (PUE): I’ll assume that ChatGPT is deployed to Microsoft data centers. Microsoft’s average PUE is 1.18 [6].

Total daily queries: The estimate of the amount of hardware assumes 13 million daily active users, 15 daily queries each, i.e. 195 million daily queries [5]. It seems like a reasonable estimate and I expect it to increase linearly with the amount of users such that if the number of daily queries doubles, so will the amount of hardware. Under these assumptions, for the purpose of calculating ChatGPT’s energy use per query, it doesn’t matter if ChatGPT currently has fewer or more daily queries, as long as it’s reasonably accurate that 195,000,000 daily queries require 3,617 Nvidia HGX A100 servers.

So the calculation becomes:

Energy consumption per query (KWh) = (3617 * 6.5 * average hardware power draw * 24 * 1.18) / 195,000,000

where average hardware power draw is 0.5 or 0.75.

Environmental Impact of Ubiquitous Generative AI

Estimating ChatGPT’s electricity consumption per request with method 2

Method 2 uses this formula (adapted from Mike Ellis’s approach [7]):

Energy consumption per query (KWh) = (FLOPS per query * joules per FLOP * PUE) / 3600000

I divide by 3,600,000 to convert from joules to KWh.

For method 2, I’ll assume the following values:

FLOPS per query: 560,000,000,000,000 (560 Teraflops) **** [8].

Joules per FLOP: 0.00000000001 [7].

PUE: 1.18 (same as for method 1).

So, the calculation becomes:

Energy consumption per query (KWh) = (560000000000000 * 0.00000000001 * 1.18) / 3600000

Now, let’s apply these methods to produce estimates of ChatGPT’s energy use per query.

The carbon footprint of GPT-4

Estimates of ChatGPT’s electricity consumption per request

In this section, I’ll present the estimates of how much energy ChatGPT needs to answer one request, along with the measured energy consumption per query of two other large language models, BLOOM and GPT-J, as reported elsewhere [9][10].

In Table 1 below, we can see that method 1 and 2 yield similar estimates of ChatGPT’s energy consumption per query. Using method 1, ChatGPT’s estimated energy consumption per query is 0.0017 KWh at the lower range (average power draw is 50 % of TDP) and 0.0026 at the higher range (average power draw is 75 % of TDP). Using method 2, ChatGPT’s estimated energy use per request is 0.0018 KWh.

Table 1: ChatGPT's estimated energy use per query
Table 1: ChatGPT’s estimated energy use per query

Let’s see how the estimates of ChatGPT’s electricty use per query compare to the electricty use per query observed for other LLMs. Table 2 below shows the energy consumption per query for GPT-J and BLOOM. GPT-J has been measured to consume 0.196 KWh per query, while BLOOM has been measured to consume 0.0039 KWh per query. Notice that BLOOM’s electricity consumption per query is not much larger than the estimated electricity consumption per query for ChatGPT, while GPT-J’s energy use is noticeably larger.

Table 2: Comparing the estimated electricity consumption per query for ChatGPT to the measured electricity consumption per query for BLOOM and GPT-J reported in [9][10]
Table 2: Comparing the estimated electricity consumption per query for ChatGPT to the measured electricity consumption per query for BLOOM and GPT-J reported in [9][10]

How to estimate and reduce the carbon footprint of machine learning models

Discussion

Above, we saw that ChatGPT’s electricity consumption per query may be between 0.0017 and 0.0026 KWh depending on the assumptions we use. We can see that the estimated energy use is similar for both method 1 and 2 which to me suggests that the estimates are in the right ballpark. The reason method 2 yields an estimate that is a bit lower than the upper range of method 1’s estimate could be because method 2 only accounts for the energy consumption of the GPUs. However, other hardware (CPU, RAM, networking equipment) also consumes energy.

If we believe the estimates by SemiAnalysis [5] to be in the right ballpark, then – from my perspective – the main source of uncertainty in the ChatGPT electricity consumption estimates are how much electricity each piece of hardware is using. In this article I have assumed that each piece of hardware uses 50–75% of its maximum power draw, which I personally believe to be reasonable, but please challenge this.

Also note that the estimated quantity is the average electricity consumption of a ChatGPT query because we use Microsoft’s average PUE. However, the PUE can vary from data center to data cetner. If you’d like to estimate your electricity consumption from using ChatGPT or similar services, you should use the PUE of the data center you expect handles your requests. The MLCO2 Impact calculator can show you a list of Microsoft Azure regions [11].

This blog post estimates that ChatGPT’s energy use per request is smaller than the measured energy consumption of the LLM called BLOOM. BLOOM is similar in size to GPT-3 – the LLM on which ChatGPT was initially based. It would make sense that ChatGPT is more energy efficient than BLOOM, because the authors of the BLOOM paper did not take any measures to improve the energy efficiency of request processing. In addition, it would be reasonable to expect that OpenAI have made such efforts since it could reduce their costs.

The results also show that GPT-J was measured to consume much more electricity per query than the other models although at 6 billion parameters, GPT-J is significantly smaller than BLOOM (176 billion), GPT-3 (175 billion), and GPT-4 (rumored 1.8 trillion). A likely explanation for this is that in the reported experiment, GPT-J was running on an Nvidia RTX3090 GPU that’s probably less energy efficient than the Nvidia A100 on which ChatGPT is likely running. In addition, no measures were taken to improve the energy efficiency of GPT-J in the experiment.

On AI stack exchange [7], Mike Ellis has calculated ChatGPT’s energy consumption using method 2 and arrived at 0.000083 KWh per query. In his calculations, he uses 30 teraflops, where I in this article use 560 teraflops. Mike Ellis uses 30 teraflops because ChatGPT itself said that it uses 30 teraflops. But as Mike Ellis himself also points out, we should be vary of trusting ChatGPT’s answer because it known to hallucinate and provide false information [12]. Using 560 teraflops as I do here, yields an energy consumption that is closer to the measured energy consumption of BLOOM. Combined with the fact that the 560 teraflops figure stems from SemiAnalysis, I therefore believe 560 teraflops per query to be more realistic.

Finally, to put ChatGPT’s estimated energy consumption per request into perspective: if you turn on a standard 40W light bulb for 1 hour, it will have consumed the same amount of energy as 15 to 24 ChatGPT queries as per my estimates.

Conclusion

This article estimates that ChatGPT may use between 0.0017 and 0.0026 KWh of electricity to answer one query. These numbers can be used by organizations to approximate their carbon footprint from using ChatGPT and similar services. Two different methods were used to obtain these results and both methods yielded estimates that are in the same ballpark. These estimates are lower than the measured energy consumption of the LLMs BLOOM and GPT-J, which have previously been measured to consume 0.0039 and 0.196 KWh per query respectively. According to these estimates of ChatGPT’s energy consumption per query, if you turn on a standard 40W light bulb for 1 hour, it will have consumed the same amount of energy as 15 to 24 ChatGPT queries.


That’s it! I hope you enjoyed the story. Let me know what you think!

Get the benefits of Medium and support my writing by signing up for a Medium membership HERE.

Follow me for more on AI and Sustainability and subscribe to get my stories via email when I publish.

I also sometimes write about time series forecasting.

And feel free to connect on LinkedIn.

References

[1] https://normative.io/insight/csrd-explained/

[2] https://www.bloomberg.com/news/articles/2022-11-17/hidden-emissions-from-cloud-computing-pose-net-zero-threat

[3] https://arxiv.org/pdf/2307.09288.pdf

[4] https://arxiv.org/ftp/arxiv/papers/2204/2204.05149.pdf

[5] https://www.semianalysis.com/p/the-inference-cost-of-search-disruption

[6] https://azure.microsoft.com/en-us/blog/how-microsoft-measures-datacenter-water-and-energy-use-to-improve-azure-cloud-sustainability/

[7] https://ai.stackexchange.com/questions/38970/how-much-energy-consumption-is-involved-in-chat-gpt-responses-being-generated/39418?noredirect=1#comment58882_39418

[8] https://archive.md/2RQ8X

[9] https://arxiv.org/abs/2211.02001

[10] https://borsen.dk/nyheder/baeredygtig/de-har-regnet-paa-chat-gpts-klimaaftryk-nu-raader-de-folk-til-at-taenke-sig-rigtig-godt-om?b_source=topchef-i-sydbank-krigen-i-ukraine-minder-om-finanskrisen&b_medium=row_8&b_campaign=news_2 and BLOOM https://arxiv.org/abs/2211.02001

[11] https://mlco2.github.io/impact/

[12] https://fortune.com/2023/08/01/can-ai-chatgpt-hallucinations-be-fixed-experts-doubt-altman-openai/


Related Articles