Cross channel effects of search engine advertising on brick & mortar retail sales: Meta analysis of large scale field experiments on Google.com

Abstract

We investigate the cross channel effects of search engine advertising on Google.com on sales in brick and mortar retail stores. Obtaining causal and actionable estimates in this context is challenging: Brick and mortar store sales vary widely on a weekly basis; offline media dominate the marketing budget; search advertising and demand are contemporaneously correlated; and estimates have to be credible to overcome agency issues between the online and offline marketing groups. We report on a meta-analysis of a population of 15 independent field experiments, in which 13 well-known U.S. multi-channel retailers spent over $4 Million in incremental search advertising. In test markets category keywords were maintained in positions 1-3 for 76 product categories with no search advertising on these keywords in the control markets. Outcomes measured include sales in the advertised categories, total store sales and Return on Ad Spending. We estimate the average effect of each outcome for this population of experiments using a Hierarchical Bayesian (HB) model. The estimates from the HB model provide causal evidence that increasing search engine advertising on broad keywords on Google.com had a positive effect on sales in brick and mortar stores for the advertised categories for this population of retailers. There also was a positive effect on total store sales. Hence the increase in sales in the advertised categories was incremental to the retailer net of any sales borrowed from non-advertised categories. The total store sales increase was a meaningful improvement compared to the baseline sales growth rates. The average Return on Ad Spend (ROAS) is positive, but does not breakeven on average although several retailers achieved or exceeded break-even based only on brick and mortar sales. We examine the robustness of our findings to alternative assumptions about the data specific to this set of experiments. Our estimates suggest online and offline are linked markets, that media planners should account for the offline effects in the planning and execution of search advertising campaigns, and that these effects should be adjusted by category and retailer. Extensive replication and a unique research protocol ensure that our results are general and credible.

Appendix

1.1 Degrees of freedom of the t-test, for inferring standard errors

We inferred standard errors from reports of the estimated mean and p-value from a one-sided t-test. The degrees of freedom (df) of the t-test was {number of test stores + number of unique control stores – 2}, which was not available. (Recall that control stores could be matched to more than one test store.) Lacking the actual df, we used for n _{i j} the total number of stores in the chain, which is necessarily larger. The inferred standard error is not sensitive to this choice for df in the range used here, as we now show.

We show this for the combined lift estimate $\overline {L}$. Note that

$$\begin{array}{@{}rcl@{}} & & p\text{-value}=\alpha \\ & \text{if and only if } & \text{Probability}(\overline{L}/\text{SE}>t_{\alpha,df})=\alpha\text{, because the test was one-sided} \\ & \text{if and only if } & \text{SE}=\overline{L}/t_{\alpha,df}, \end{array} $$

(8)

where t _{α, d f} is the 100(1 − α) percentile of the standard t distribution with df degrees of freedom. As df is increased, t _{α, d f} decreases, so SE increases. Thus, by using a df value larger than the correct but unknown df, we infer standard errors that are larger than the true values, which is conservative in the sense of acting as if each experiment was less informative than it actually was.

Further, t _{α, d f} is insensitive to df in the range we used; thus, the inferred standard errors are insensitive. For example, t _{α, d f} is 1.708 for df = 25; 1.671 for df = 60; 1.658 for df = 120; 1.645 for df = ∞ (i.e., the normal distribution). Over this range of df, t _{α, d f} decreases by 3.7%, and the ends of this range differ by far more than our conservative df differs from the actual but unknown df.

1.2 Standard errors for return on ad spending (ROAS)

The data available for this study did not include standard errors for individual experiments’ ROAS estimates, and the corresponding p-values were reported in ranges. To avoid excessive conservatism in inferring standard errors for the ROAS estimates, we computed standard errors for ROAS as follows. The derivation has two steps. The first conditions on each experiment’s average (over test stores) expected sales (i.e., acts as if average expected sales is known), while the second step removes that conditioning (i.e., treats average expected sales as a random variable).

Conditioning on an experiment’s average expected sales $\overline {ES}$, from Section 6.1,

$$\begin{array}{@{}rcl@{}} \overline{L} & = & \frac{1}{n\overline{ES}}\sum\limits_{s}(AS_{s}-ES_{s}) \\ \text{and }ROAS & = & \frac{1}{ADS}\sum\limits_{s}(AS_{s}-ES_{s}). \end{array} $$

(9)

Assume that ${\sum }_{s}(AS_{s}-ES_{s})\sim N(\mu ,\tau ^{2})$; then

$$ \begin{array}{ll}\overline{L} & \sim N\left( \frac{\mu}{n\overline{ES}},\frac{\tau^{2}}{n^{2}\overline{ES}^{2}}\right)\text{ and}\\ ROAS & \sim N\left( \frac{\mu}{ADS},\frac{\tau^{2}}{ADS^{2}}\right). \end{array} $$

(10)

We have $\hat {\sigma }_{L}^{2}$, the estimated variance (square of standard error) for an experiment’s lift estimate; we want an estimate for the variance of ROAS, $\hat {\sigma }_{R}^{2}$. From Eq. 10,

$$ \begin{array}{ll}\hat{\sigma}_{L}^{2} & =\frac{\hat{\tau}^{2}}{n^{2}\overline{ES}^{2}}\\ \text{so that }\hat{\tau}^{2} & =n^{2}\overline{ES}^{2}\hat{\sigma}_{L}^{2},\\ \text{Therefore }\hat{\sigma}_{R}^{2} & =\frac{\hat{\tau}^{2}}{ADS^{2}}\\ & =\frac{n^{2}\overline{ES}^{2}}{ADS^{2}}\hat{\sigma}_{L}^{2}. \end{array} $$

(11)

Conditional on $\overline {ES}$, then, we can estimate the standard error of ROAS. However, $\overline {ES}$ is itself a random variable because test stores were randomly sampled. To remove this conditioning, note that by Eq. 9, conditional on the vector of expected sales for the test stores, ES,

$$ ROAS|\boldsymbol{ES}\sim N\left( \frac{n\overline{ES}}{ADS}\theta,\frac{n^{2}\overline{ES}^{2}}{ADS^{2}}{\sigma_{L}^{2}}\right). $$

(12)

where $\theta =E(\bar {L})$. The variance of ROAS is thus

$$ \begin{array}{ll}\operatorname{Var}(ROAS) & =\operatorname{Var}[\operatorname{E}(ROAS|\boldsymbol{ES})]+\operatorname{E}[\operatorname{Var}(ROAS|\boldsymbol{ES})]\\ & =\operatorname{Var}\left( \frac{n\overline{ES}}{ADS}\theta\right)+\operatorname{E}\left( \frac{n^{2}\overline{ES}^{2}}{ADS^{2}}{\sigma_{L}^{2}}\right)\\ & =\frac{n^{2}\theta^{2}}{ADS^{2}}\operatorname{Var}(\overline{ES})+\frac{n^{2}{\sigma_{L}^{2}}}{ADS^{2}}\operatorname{E}(\overline{ES}^{2})\\ & =\frac{n^{2}\theta^{2}}{ADS^{2}}\operatorname{Var}(\overline{ES})+\frac{n^{2}{\sigma_{L}^{2}}}{ADS^{2}}\{\operatorname{Var}(\overline{ES})+[\operatorname{E}(\overline{ES})]^{2}\}\\ &=\frac{n^{2}{\sigma_{L}^{2}}}{ADS^{2}}[\operatorname{E}(\overline{ES})]^{2}+\frac{n^{2}(\theta^{2}+{\sigma_{L}^{2}})}{ADS^{2}}\operatorname{Var}(\overline{ES}). \end{array} $$

(13)

In the last line above, the first term is the conditional estimate of ${\sigma _{R}^{2}}$ with $\operatorname {E}(\overline {ES})$ substituted for $\overline {ES}$. Thus the second term is the primary consequence of removing the conditioning on $\overline {ES}$.

The data available for this study included, for each experiment, the sample mean and standard deviation of the E S _s , so we can estimate $\operatorname {Var}(\overline {ES})$ if we can make a plausible assumption about the correlation between E S _s and E S _t for test stores s and t. To do this, we assumed the E S _s were exchangeable with correlation r between each pair of test stores. Therefore,

$$ \begin{array}{ll}\operatorname{Var}(\overline{ES}) & =\frac{1}{n^{2}}\operatorname{Cov}\left( {\sum}_{s=1}^{n}ES_{s},{\sum}_{t=1}^{n}ES_{t}\right)\\ & =\frac{1}{n^{2}}\left( n\sigma_{ES}^{2}+n(n-1)r\sigma_{ES}^{2}\right)\\ & =\sigma_{ES}^{2}\left( \frac{1}{n}+\frac{n-1}{n}r\right). \end{array} $$

(14)

We roughly estimated the correlation coefficient r as n _{s t} /10, where n _{s t} is the number of control stores shared by the s ^th and t ^th test stores. Simulations in which control stores were assigned at random to test stores gave the estimates of r in Table 9.

A standard error for each experiment’s ROAS was obtained by substituting Table 9’s estimated r’s and other known quantities (e.g., standard errors for sales lift) into Eq. 13. The unconditional and conditional estimates of ${\sigma _{R}^{2}}$ (with $\operatorname {E}(\overline {ES})$ substituted for $\overline {ES}$) were very similar. Meta-analyses using the two different estimates gave very similar results, so Section 6.2.2 presents the simpler analysis using the conditional estimates of ${\sigma _{R}^{2}}$.