Why OpenAI’s API Is More Expensive for Non-English Languages

Beyond words: How byte pair encoding and Unicode encoding factor into pricing disparities

How can it be that the phrase “Hello world” has two tokens in English and 12 tokens in Hindi?

After publishing my recent article on how to estimate the cost for OpenAI’s API, I received an interesting comment that someone had noticed that the OpenAI API is much more expensive in other languages, such as…