LSTM Response Models for Direct Marketing Analytics: Replacing Feature Engineering with Deep Learning

Highlights

• Traditional customer response models rely heavily on feature engineering.

• Their performance depends on the analyst's domain knowledge and expertise to craft relevant predictors.

• In contrast, long-short term memory (LSTM) neural networks rely exclusively on raw data.

• Still, we demonstrate LSTM models provide superior performance to traditional models.

• LSTM neural networks are excellent candidates for direct marketing, brand choices, clickstream data, and churn predictions.

Abstract

In predictive modeling, firms often deal with high-dimensional data that span multiple channels, websites, demographics, purchase types, and product categories. Traditional customer response models rely heavily on feature engineering, and their performance depends on the analyst's domain knowledge and expertise to craft relevant predictors. As the complexity of data increases, however, traditional models grow exponentially complicated. In this paper, we demonstrate that long-short term memory (LSTM) neural networks, which rely exclusively on raw data as input, can predict customer behaviors with great accuracy. In our first application, a model outperforms standard benchmarks. In a second, more realistic application, an LSTM model competes against 271 hand-crafted models that use a wide variety of features and modeling approaches. It beats 269 of them, most by a wide margin. LSTM neural networks are excellent candidates for modeling customer behavior using panel data in complex environments (e.g., direct marketing, brand choices, clickstream data, churn prediction).

Introduction

In direct marketing, a firm targets a customer with a marketing solicitation such as a catalog, a direct solicitation, or a coupon, and the customer decides whether or not to respond. Since soliciting a customer unlikely to respond is unprofitable, and not soliciting a potentially profitable customer leaves money on the table, the ability to predict customers' responses has long been a crucial endeavor for both practitioners and academics (e.g., Malthouse, 1999, Roberts and Berger, 1999).

Response models in direct marketing predict customer responses from past customer behavior and marketing activity. These models often summarize past events using features such as recency or frequency¹ (e.g., Blattberg et al., 2008, Malthouse, 1999, Van Diepen et al., 2009), and the process of feature engineering has received significant attention (Kuhn and Johnson, 2019, Zheng and Casari, 2018).

In machine learning, a feature refers to a variable that describes some aspect of individual data objects (Dong & Liu, 2018). Feature engineering has been used broadly to refer to multiple aspects of feature creation, extraction, and transformation. Essentially, it refers to the process of using domain knowledge to create useful features that can be fed as predictors into a model.

However, feature engineering presents its own set of challenges.

First, the same features might identically summarize widely different behavior sequences (Blattberg et al., 2008, Fader et al., 2005). Consider the customer behavior pattern depicted in Fig. 1. All four customers in the figure have the same seniority (date of first purchase), recency (date of last purchase), and frequency (number of purchases). However, each of them has a visibly different transaction pattern. A response model relying exclusively on seniority, recency, and frequency would not be able to distinguish between customers who have similar features but different behavioral sequences.

Second, in a complex environment where there are multiple streams of data, such as in a data-rich environment where the analyst has access to historical marketing activity of various sorts (e.g., multiple types of solicitations sent through various marketing channels) and diverse customer behaviors (e.g., purchase histories across various product categories and sales channels) observed across different contexts (e.g., multiple business units or websites, see Park & Fader, 2004), the vast number and exponential complexity of inter-sequence and inter-temporal interactions (e.g., sequences of marketing actions, such as email–phone–catalog vs. catalog–email–phone) will make the data analyst's job arduous.

Let us reflect for a moment on one of the simplest and most commonly used features in direct marketing: recency, or the time elapsed since the last customer's purchase. How should the analyst hand-craft relevant recency features in an environment spanning multiple product categories? Should she take into account the last absolute recency, regardless of the product category purchased (hence losing richness and granularity, and potentially hurting the model's predictive power)? Should she include in the model as many recency indicators as there are product categories in the data set (hence creating excruciating multicollinearity issues if customers buy from multiple product categories at each purchase occasion)? Should she combine individual and aggregate recency indicators? When crafting relevant recency indicators, should the analyst consider purchases in brick-and-mortar stores and purchases on the firm's website jointly, or should she treat these indicators separately?

When an analyst uses feature engineering to predict behavior, the performance of the model will depend greatly on the analyst's domain knowledge, and in particular, her ability to translate that domain knowledge into relevant features for the model. In complex environments, such as in the presence of multiple channels or multiple product categories, it can be quite challenging indeed for an analyst to capture all useful inter-sequence and inter-temporal interactions.

In this paper, we explore whether Long-Short Term Memory neural networks (LSTM), a special kind of Recurrent Neural Networks (RNN), which rely on raw sequential data and do away with feature engineering, can offer the promise of a solution to this general class of modeling problems in marketing.

In customer response models, the data are often in the form of panel data, where the firm's actions (e.g., solicitations) and customers' behavior (e.g., purchases) are observed repeatedly over time and along multiple dimensions (e.g., multiple channels or product categories).

Surprisingly, while RNN models are common in natural language processing, their applications—let alone marketing panel data—have been scarce, and even close to nonexistent. In their seminal book, Goodfellow, Bengio, and Courville (2016) cite applications of RNN in the domains of machine translation, prediction of text sequences, handwriting recognition, and speech recognition. Pointer (2019, p. 70) mentions in passing that RNNs are particularly suited for “data that has a temporal domain (e.g., text, speech, video, and time-series data),” but dedicate the chapter to text analysis. Saleh (2018) dedicates an entire section to the numerous applications of RNN (pp. 153–157), but exclusively cites natural language processing, speech recognition, machine translation, unidimensional time-series forecasting, and image recognition. However, as we will demonstrate, RNN models in general, and LSTM models in particular, seem particularly suited for panel data analysis.

We organize the paper as follows. In the first section, we introduce the LSTM model as a special class of recurrent neural networks. Given the newness of the method to social scientists in general, and to marketing analysts in particular, we dedicate significant space to explain its inner working. While LSTM models take raw behavioral data as input and therefore do not rely on feature engineering or domain knowledge, our experience taught us that some fine-tuning is required to achieve optimal LSTM performance; in the second section, we pay special attention to the proper calibration of an LSTM model, including parameter and hyperparameter tuning, which can be fully automated and do not require domain knowledge either. In the third section, we demonstrate the superior performance of the LSTM model in a relatively simple, direct marketing setting with only donations (yes/no) and solicitations (yes/no). We show that the LSTM model, relying on raw data, achieves a better average fit and performance than the feature-based, benchmark models. In the fourth section, we benchmark a vanilla LSTM model in a much more complex environment (e.g., multiple channels and donation types) against 271 hand-crafted models developed by about as many human analysts. The LSTM outperforms 269 of them. In the fifth section, we discuss the marketing applications in which we expect LSTM neural networks to prove valuable, and important technical considerations in the fast-moving field of deep learning in the sixth section. We conclude in the seventh section.