The world’s leading publication for data science, AI, and ML professionals.

Analyzing eBay’s AdWords Spending: Is This Extra Expense Worth It?

A Simple Case Study in RStudio

Analyzing eBay’s AdWords Spending

A Simple R-Based Case Study

Photo by Stephen Phillips - Hostreviews.co.uk on Unsplash
Photo by Stephen Phillips – Hostreviews.co.uk on Unsplash

Just how effective is paid online advertising for a large website like eBay? Does the benefit provided by an additional expenditure truly outweigh the results that would be expected from organic search behavior? Let’s find out through a simple R-based stats study:

Abstract

This analysis evaluates data retrieved by eBay in an experiment designed to assess the profitability of paid advertising through Google’s search platform in May of 2013. During the month, eBay switched off paid search advertising in a random subset of 70 designated market areas (DMAs) in the United States. These market areas are defined as "regions where the population can receive the same or similar television and radio station offerings and may also include other types of media including newspapers and Internet content." The purpose of this analysis is to determine whether the benefits of paid advertising through Google AdWords outweigh the results that eBay would see as a result of organic search patterns.

All analysis was done in R and RStudio


Notes on the Data

If you’d like to view or download the data for your own use, you can do so here.

The columns are as follows:

DMA: The name of the designated market area in question (e.g. New York, Los Angeles, Miami-Ft. Lauderdale, etc.)

rank: The rank of that DMA by population

tv_homes: The number of homes in the specified DMA with a television

adwords_pause: A 0/1 indicator for treatment vs. control group (in this case, a 1 signifies that the DMA was in the treatment group, and a 0 signifies it was in the control)

rev_before: The revenue generated by a specific DMA in the 30 days prior to May 22nd (the day the experiment started)

rev_after: The revenue generated by a specific DMA in the 30 day period beginning May 22nd

rev_ratio: The ratio of revenue after to revenue before in any given DMA

The variable of interest here is _revratio, as absolute differences in revenue across DMAs can differ greatly due to variance in population. The _revratio variable allows us to compare DMAs on common ground, and will therefore be the basis of further investigation.


Approach

To approach this analysis, I began by first establishing a null hypothesis as a foundation for further Hypothesis Testing. In the case of the problem, this is the assertion that both the treatment and control groups would see the same revenue ratio before and after the experiment – in other words, the difference in revenue ratio between paused and unpaused groups after making changes to AdWords spending would be zero.

To further investigate this, I then fit a linear model for revenue ratio versus the binary indicator for experiment group classification like so:

lm_DMA = lm(rev_ratio ~ adwords_pause, data=ebay)
coef(lm_DMA)
Photo from Author using RStudio
Photo from Author using RStudio

The results of this model suggest a negative offset of ~5% when AdWords spending is paused. However, it is possible these results could not accurately generalize due to the chance of random luck. To overcome this, I conducted a randomization test with 10,000 Monte Carlo simulations of the same linear model from above, applied to shuffled data from the experimental set on hand. This allows us to see what kind of results we’d expect to see under the aforementioned null hypothesis:

dma_rand = do(10000)*lm(rev_ratio ~ shuffle(adwords_pause), data=ebay)
ggplot(dma_rand) + 
  geom_histogram(aes(x=adwords_pause))

Executing the above code block yields the following distribution:

Photo from Author using RStudio
Photo from Author using RStudio

From these results, the chance of seeing a difference in revenue ratio of ~5% seems rather unlikely under the null. To bolster this assertion, a p-value can be calculated from running the following:

pval = sum(dma_rand$adwords_pause <= -0.05228145)/10000

This nets a value of a little less than 0.01.

Conclusion

In a rather unsurprising culmination of events, analysis of this experiment seems to suggest that eBay’s extra spending on AdWords was a driver – in at least some capacity – of the increase in revenue ratio seen across DMAs. While there is still a small chance that this observation was a result of luck, a p-value as small as the one observed suggests that a revenue boost of this level is rather unlikely under organic search dynamics alone. While further testing could certainly be done, investing in AdWords spending seems to be like a wise decision for eBay in the near term.


Related Articles