The Price of Power, Part 1

Retail Power Prices in Texas: When the ‘Average’ Misrepresents

I read with interest the front-page Wall Street Journal article “Texas Electric Bills Were $28 Billion Higher Under Deregulation” of February 24th of this year. It promised an “analysis” of Texas retail power prices. The headline graph and its title (reproduced here) seemed to provide a definite conclusion: deregulated retail customers are overpaying.

Pricey Power

The roughly 60% of Texans who must choose a retail electricity provider consistently pay more than customers in the state who buy their power from traditional utilities.

Source: Wall Street Journal analysis of U.S. Energy Information Administration data

The article then takes the price differential in this graph (elsewhere defined as a difference of ‘average’ prices for residential customers only), multiplies it by the amount of energy sold via retail providers, and voila: 28 Billion Dollars (which sounds vastly more impressive than the equivalent $26/Customer/Month [Average 5.6MM customers over 192 months]).

Well, as someone who regularly shops power suppliers as a retail consumer in Texas, the numbers seemed…suspiciously high. For example, rummaging around in my records, I found:

August 2018 Texas Electricity Facts Label

And, I have a stack of similar or lower-priced records dating back to 2010. My 2018 contract renewal, and the others, seemed to be considerably below both the ‘Retail Provider’ and ‘Traditional Utility’ numbers provided in the article. It’s not as if I utilize some obscure marketing scheme — each year, when my contract expires, I go to the PUC-sponsored website powertochoose.org, choose “12-month, fixed price, sorted” and select the provider with the lowest price. Not a lot of brainpower involved there (although occasionally I have seen some plans which have different prices based on usage, so that does require some math). In any case, I have a stack of these ‘EFLs’ over the last 10 years with similar results, all lower than both lines this graph. And, I have found the process to be vastly easier than shopping for other household services such as internet service provider, insurance, cell phones, and streaming services.

Questions we will answer, two parts:

1. So where is this data? How can I analyze it myself?
2. Who are the ‘Traditional Utilities’ and ‘Retail Providers’? What can be learned from the price comparison? What are the limitations of such a comparison?
3. What is meant by ‘average’ price? The article clarifies, below the graph: “The Journal calculated separate annual statewide rates for utilities and retailers by adding up all of the revenue each type of provider received and dividing it by the kilowatt-hours of electricity it sold.” So that’s a weighted-average price (total retailer revenues divided by total retailer sales). How well does that weighted measure of central tendency summarize the typical customer experience? Why are my results so different?
4. What has transpired in the (roughly 2/3) of the market for commercial and industrial customers? Why is that not part of this story?
5. How can I place this $28 Billion in context? What are some other insights that can be gained from this data?

This article will focus on questions 1–3, and a subsequent article will focus on 4–5.

Disclosures: Who am I, why am I writing this?

So this is largely a story about data and the problems with using one summary statistic to represent a dynamic and varied market. You don’t need a background in energy to gain something from this analysis; but nonetheless, that is my background. I’ve spent the last decade or so as an independent energy consultant. Previously, I worked in wholesale power trading and risk management. As a consultant, I have done projects for retail power clients[but also banks, refiners, oil traders, etc]. But my point here is: the views here are entirely my own, and this work isn’t associated with any of my clients, past or present. I’m writing this because

1. I think this data tells a more interesting story than has been portrayed,
2. The authors of the article didn’t make the details of their analysis or the [cleaned] dataset public,
3. This is an interesting exercise in extracting, cleaning and analyzing a dataset using python, pandas, and matplotlib.
4. I’m interested in promoting the accurate dissemination and analysis of public data — this is a good example.

Code, Formatting, Conventions

All of the data and code supporting this article can be found on github [1]. If you spot an error, or have an extension, I will gladly take pull requests. All of the code is in python (I used version 3.83, specifically the winpython distribution). The python files are in ordinary *.py format (i.e. they are not jupyter notebooks). I have separated the code into ‘code cells’ consistent with those used by the Spyder IDE. In order to replicate any referenced analysis, simply execute all the code in that block. Finally, I have quite a few tables in this article, which I have embedded as gists on github — click on the embedded object for a full view.

The Data

The data here originate from the Energy Information Agency’s Form EIA-861 data files here, and copied to github for convenience[1]. Those datafiles are in a series of spreadsheets embedded in zipfiles, one file per year. In the zipfiles, there are spreadsheets, with each spreadsheet representing a customer sector, as follows:

The data elements recorded are as follows:

Conversion and Cleaning

The script to extract the EIA data[2] is too long to usefully excerpt, so I’ll simply note the issues with cleaning the data addressed by this script:

1. From 2007 and previous, the EIA reversed the order of Revenue and Sales as noted in the table above.
2. The spreadsheet data started on different lines depending on the year, but always started with Alaska in the case of all sectors except Transportation, which started with Arkansas.
3. Some of the early year spreadsheets had an extraneous 8th (‘Data Check’) column, excluded here.
4. Some of the State values showed as Null, excluded here, likewise all rows where all of Revenue, Sales, and Customers were zero were excluded.

So once the data is pulled from the zipfiles and fixed in accordance with above….

1. I checked that Rev/Sales = AvgPrc, which was true except in some cases where Ownership was 'Other' or 'Behind the Meter' (also Entity was 'adjustment'), in which case AvgPrc was null. Our focus here isn't on adjustment or behind-meters sales, so I left the data as is.
2. Added feature OwneshipType ['Reg' or 'DeReg'] based on whether Ownership is either 'Power Marketer' or 'Retail Provider'...it appears there was a change in terminology over the years.
3. Added features Year and CustClass (i.e. 'commercial', 'industrial', 'residential', 'transportation', 'all'), converted the whole dataset to a 'records' format with Columns 'ValueType' to indicate Revenues, Sales, Customers or AvgPrc and Value to indicate quantity, and then saved the whole thing to parquet for convenient use in other scripts.

So at this point, we have a records table of individual Rev, Sales, Customers or AvgPrc elements 650,900 rows by 8 columns, Year 2004-2019:

What can be learned here?

Well, it’s a comprehensive set of revenue, cost and customer data, so one can certainly do both categorical (across customer classes, ownership types, entities, states) and temporal (across years) comparisons of sales volumes, number of customers, and revenues (with implied unit prices). You can look at market concentration or growth of customer base, for example; there’s a lot of potential directions to take with this data.

What are the limitations?

You most certainly cannot definitively assess the effectiveness of either a regulatory regime or market mechanism from this data alone, in contrast to the quote in the reviewed article “high statewide prices relative to the national average “must be attributed to the deregulated sector of Texas”. Most specifically, there is no data here on:

1. Wholesale prices: You can’t normalize results with prices available via the (deregulated) wholesale bulk power market — that information isn’t here.
2. Distribution or ‘Wires’ charges: retail customers pay a (regulated) fee to access their (unique) distribution system — that value varies widely by customer, but is bundled in the total price results here.
3. Generation and Load Characteristics (Somewhat related to the above) There’s no data here on

• Weather/load characteristics [flatter, easier-to-predict loads should be cheaper to serve, regardless of regulatory choice or market design alternative]
• Local generation asset mix, e.g. Does the local utility own (cheap) hydro generation? Or natural gas generation [a fuel subject to substantial price changes between 2004–2019]? Are the utility’s coal plants scrubbed, or equipped with SCRs,, and if so, when did the installation go into rate base? Are there (non-market-competitive) assets maintained solely for reliability?
• Whether the customer can choose (more expensive)renewable generation in lieu of fossil power, or a plan which favors stable billing over average cost. Those are both choices in Texas, and they both imply a cost vs benefit choice.

So, it’s important to realize the limitations of the data here, and what it can and cannot show. Simple price analysis is a necessary but insufficient requirement to determine the impact of retail electric deregulation. Please bear these caveats in mind as we review this analysis.

So what are some useful preliminary findings of the data?

We’ll cover a few basics: We’ve extracted Texas records from total, that’s 42,616 rows in total. Focusing on the Customer ValueType, we have the following counts across all years[3]:

We’re going to focus on the residential group in this article — we’ll get to the other classes later.

The ‘Retail Provider’ group (‘DeReg’ in our parlance) includes 158 unique names…but a much smaller subset in any one year. The data don’t include ownership, or any unique identifiers, so we can’t track name changes or identify market concentration through ownership. But in any case, we can see the growth in supplier count as observed by the customer. Here is the count over time[3].

For 2019 (most recent data), we can ‘bin’ the 68 retailers by customer size, I’ve chosen the bin size as follows[3]:

Total Customers: 6,451,123

Total Retailers: 68

And we can see a highly asymmetric concentration of customers among a few large suppliers. We’ll return to that theme below.

The ‘Traditional Utilities’ group is of course more stable over time, but they have little in common with each other; they serve customers in the 270K square mile state of Texas, and they are regulated monopoly providers. They span multiple ownership types, all three major North American interconnects, multiple climates, and types/levels of asset ownership. In 2019, there were 139 such entities, in the following categories[3]:

So, on to the examination of prices.

Okay, what’s this ‘average’ price?

So we can copy the article data straight from the article’s html (we actually did that previously — it’s json data embedded in the article’s source code).
We start simple: exploit pandas pivot table functionality to get simple averages and medians for traditional utilities and retail providers, and compare the article data to those calculated values. That result is:

So right away, we can see that:

1. The article values in bold (weighted averages) are significantly higher than either (1) the price of the average provider or (2) the price of the median provider.
2. So it follows that the price distribution by sales must be significantly skewed for retail providers. For the most recent year, the weighted average is 24% above the median supplier’s price. By contrast, for ‘Traditional Utilities’ that difference is minimal.
3. My 2018–2019 fixed price contract detailed above makes a lot more sense now…below the median, but closer to that than to the article’s weighted average.

We definitely should do a visualization here…all 6 rows would be a bit busy, so we’ll drop the ‘simple mean’ values:[4].

So this allows us to see the trends a little better. So we see here that (1) the median retail provider’s price dropped below the median ‘traditional utility’s’ price around 2011 and stayed below. (2) the median retail provider’s price is always below the ‘weighted average’, but that spread has increased for more recent data.

I would assert, given the significant difference in measures of central tendency for this data, basic data transparency dictates that the preceding graph is the one that should have appeared in the article. Of course, such a presentation turns a simple narrative into one which is not so clear — different (and I maintain, more interesting) questions appear.

Confirm weighted average

Since we haven’t yet confirmed that we are reviewing the same data, good analysis practice would be to confirm the article’s weighted average values. Making use of pandas pivot_table and multi-index capabilities, we calculate a weighted average and compare to the article’s values[4].

And we basically tie on weighted average…certainly down to the ¢0.1/kwh level (1 mill).

Why are my results so different from ‘Average’ ?

Well, we’ve definitely got a clue from the work we’ve accomplished thus far…this distribution is highly skewed. Why? Time for a bar chart, let’s look at 2018 results[4].

So, the retail providers with the most customers also charge the highest prices. But there are 66 retail providers represented here, the majority of whom have substantially better pricing than the weighted average. (and many of whom offer substantially better pricing than the median Municipal, Cooperative or Investor-Owned Utility).

By the way, it’s fair to repeat: in an unregulated market, simple price may not be the only issue…a customer might value Green power, for example, and be willing to pay for it. Companies offer ‘stable bill’ plans, for example, where the 12-month total absolute bill is constant — a form of insurance. Finally, consumer credit is bundled — in cases like my EFL above, there’s implicit financing — I lend money to my supplier in off-peak months, my supplier lends money to me in peak months.

So, who are those large suppliers on the right side of the graph [4]? We’ll sort by customer size:

No surprise here…the two largest are the retail arms of the legacy regulated retail providers for the Dallas and Houston regions — and they drive the price distribution skew. (Also note Green Mountain — not necessarily comparable to the others because their product is green energy vs majority slice-of-system sales elsewhere)

So to summarize, we started with an article in the business press, in which the authors used a weighted average price graph of a subsection of the electricity market to spin a large, headline-grabbing dollar cost attributed to ‘deregulation’. Here, I’ve demonstrated how to pull the data, and provided a more detailed analysis. I’m hopeful this analysis demonstrates the inappropriateness of a simplistic weighted-average calculation to summarize a market situation which is largely just an artifact of customer retention by legacy providers. To detail, based on the bar chart[4], we have a large group of customers who remain with legacy incumbents at higher prices (2.7MM, avg ¢12.8/kwh) and a similarly large group of customers who have chosen alternate suppliers at lower prices (3.6MM, avg ¢9.9/kwh). A more realistic view of the data on residential retail power in Texas demonstrates that customers who switch from legacy providers (especially in recent years) gain a clear benefit, both compared to those legacy providers and compared to the ‘traditional utilities’ (with caveats to that comparison as noted above). In summary, those who choose to shop based on price have had the opportunity to do so and save money — the data clearly demonstrate this assertion. The data also demonstrate the costs for those who choose to remain with the legacy providers. There may be important policy lessons here, positive or negative (e.g. customer education, measures limit market dominance of legacy providers, etc). But what’s clear from this analysis is that summarizing this data using only a weighted average misrepresents a market reality which is both diverse in terms of provider choices and dynamic over time.

Finally, the data and code are posted, so you have the opportunity to decide for yourself the appropriateness of this analysis, create an extension or an alternative. I’m hopeful this article and associated github repo provides some useful example code concerning the use of python, pandas and matplotlib in the cleaning, processing, analysis and visualization of this data.

For the next article, we’ll consider the other sectors of the Texas electric retail market, and do some more in-depth analysis of the residential retail market discussed here.

References

1. Main Code Repository: github.com/dkfurrow/eia-retail-analysis
2. Extraction/Cleaning of EIA Data: eia_retail_extract.py
3. Characterization of Regulated/Deregulated Customers: article_scratchpad.py
4. Main analysis, visualizations: eia_retail_analysis1.py