The role of slant and message consistency in political advertising effectiveness: evidence from the 2016 presidential election

Abstract

We explore the relationship between the content of political advertising on television and ad effectiveness. Specifically, we investigate how slant – the extremeness of the message – and consistency with the candidate’s primary campaign messaging in national ad buys relate to two measures of voter behavior: online word-of-mouth (WOM) and voter preference (captured through daily polls) for the candidates. Using data from the 2016 presidential election, we find that ad messages that are more (1) centrist and (2) consistent with a candidate’s primary-election platform associate with increases in online WOM and voter preference for the candidate. We further find that consistency is more important in the early (pre-October) stages of the campaign. Our results suggest that while there may be a benefit to candidates moderating their message after winning the primary election, they need to be careful about shedding their messaging from the primary election during the early stages of the general election. Additionally, our results enrich our understanding of the use of extreme messaging in political advertising, a phenomenon that is on the rise, by showing that it may have a cost of decreased candidate-related WOM and voter preference for the candidate.

Appendices

Appendix 1. Slant variable details

In this appendix, we present details on the calculations of the slant index.

1.1 Political slant

We first pre-process texts using the NLKT module in Python: we make words lower case and remove stop words,¹⁷ punctuations, and numbers. We then tokenize each of the texts and stem the words in the text.

Following Gentzkow and Shapiro (2010) and their notations for consistency, we first derive the mapping between a vector of word counts a congressperson used and the political leanings of their district. For each word p in the 114th Congressional Record, we count the number of times the word p is used by each of the two parties and calculate a chi-square statistic, ${\mathcal{x}}_p^2$.¹⁸ We restrict our focus to words that occur at least 2 times but fewer than 100 times in the candidates’ public speeches and transcribed ad texts.¹⁹ This removes some of the most common and least common words, which are not useful for the analysis. We then select the 1000 words with the highest values of ${\mathcal{x}}_p^2$.

Among the 1000 selected words, we regress congressperson c’s relative frequency of word p, $\tilde{f}_{pc}$, on their ideology, ideology_c, measured by the Republican vote share in the district²⁰ from the 2012 presidential election (collected from https://dailykos.com) and estimate an intercept parameter α_p and slope parameter β_p. A positive (negative) slope estimate suggests that the word p is associated with the Republican (Democratic) party.

We then compute the political slant for each of the ad creatives by applying the same mapping between the relative word frequencies and political slants of those words used in the ad. Specifically, the political slant of ad creative n is computed as $\tilde{y}_{n} = \frac{{\mathop \sum \nolimits_{p = 1}^{p = 1000} b_{p} \left( {\tilde{f}_{pn} - a_{p} } \right)}}{{\mathop \sum \nolimits_{p = 1}^{p = 1000} b_{p}^{2} }}$.²¹

Finally, we re-index the estimated political slant of ad creative n, $\tilde{y}_{n}$ to $\tilde{y}_{nc}$ to denote the candidate c that ad creative n supports. Our slant measure is then calculated as $\hat{y}_{cn} = - \left( {\tilde{y}_{cn} - 0.5} \right)$ if c is Clinton and $\hat{y}_{cn} = \tilde{y}_{cn} - 0.5$ if c is Trump. Thus, a greater (lower) slant measure always corresponds to a more politically extreme (centrist) message for both candidates.

Appendix 2. Doc2Vec validation

In this appendix, we provide additional evidence to confirm the validity of the performance of the doc2vec algorithm by presenting two ad creatives with the other ad creatives that are found to be the most similar, excluding the chosen ad creative in consideration, and the least similar.

Tables 8 and 9

Table 8 Example of an Ad that supports Clinton

Table 9 Example of an ad that supports Trump

Appendix 3. Sensitivity to size of vector space in dov2vec

In this appendix, we demonstrate that our results in the main text are robust to the dimension of the vector space, one of the most important hyper-parameters, by providing results for different vector sizes. In all cases, standard errors are clustered at the candidate level.

Tables 10, 11, 12, and 13

Table 10 Effects of political slant and message consistency on WOM (vector = 150)

Table 11 Effects of political slant and message consistency on WOM (vector = 300)

Table 12 Effect of slant and message consistency on voter preference (vector = 150)

Table 13 Effect of slant and message consistency on voter preference (vector = 300)

1.1 Appendix 4. Sensitivity of the WOM results to the time window used

In this appendix, we show that our results in the main text are robust to different time windows by providing results for two- and three-minute windows.

Tables 14 and 15

Table 14 Effects on WOM (time window = 2 min)

Table 15 Effects on WOM (time window = 3 min)

Appendix 5. Tests on the level of clustering

In this appendix, we conduct the statistical test for the appropriate level of clustering proposed by MacKinnon et al. (2020). MacKinnon et al. (2020) test the null hypothesis of a finer clustering level against the alternative hypothesis of a coarser clustering level. In our context, we can test whether clustering at the ad creative level (null hypothesis) against clustering at the candidate level (alternative hypothesis). The results, shown below, reveal that the no clustering case (the finest case) is rejected for both the ad creative level and the candidate level, but clustering at the ad creative level is rejected against the candidate level. Taken together, these results suggest that we cluster standard errors at the candidate level.

	Estimated Model: Eq. (2)
Test	Statistic	Bootstrapped p value
N vs A	76.63	0.000
N vs C	29.35	0.014
A vs C	19.08	0.031

1. Note: N denotes no clustering; A denotes clustering at the ad creative; C denotes clustering at the candidate level

Appendix 6. Influence of outliers on the WOM analysis

In this appendix, we show that our results remain very similar to removal of outliers. Specifically, we run our WOM analysis after both winsorizing and trimming the post-WOM volume at the 1 and 99% level.

Tables 16 and 17.

Table 16 Effects on WOM (winsorized Post-WOM at the 1 and 99% level)

Table 17 Effects on WOM (trimmed Post-WOM at the 1 and 99% level)

Appendix 7. Comparison of national primetime ads versus local primetime ads

In this appendix, we show that Clinton and Trump’s national primetime ads exhibit very similar airing patterns as their local primetime ads in terms of time, day of the week, and month aired. Note that the summary statistics for the local prime ads are generated from the raw Stradegy data on the candidate’s primetime advertising and have not been cleaned for data errors.

Tables 18, 19, 20.

Table 18 National versus local primetime ad airings by time

Table 19 National versus local primetime ad airings by day of the week

Table 20 National versus local primetime ad airings by month