Bag of biterms modeling for short texts

Abstract

Analyzing texts from social media encounters many challenges due to their unique characteristics of shortness, massiveness, and dynamic. Short texts do not provide enough context information, causing the failure of the traditional statistical models. Furthermore, many applications often face with massive and dynamic short texts, causing various computational challenges to the current batch learning algorithms. This paper presents a novel framework, namely bag of biterms modeling (BBM), for modeling massive, dynamic, and short text collections. BBM comprises of two main ingredients: (1) the concept of bag of biterms (BoB) for representing documents, and (2) a simple way to help statistical models to include BoB. Our framework can be easily deployed for a large class of probabilistic models, and we demonstrate its usefulness with two well-known models: latent Dirichlet allocation (LDA) and hierarchical Dirichlet process (HDP). By exploiting both terms (words) and biterms (pairs of words), the major advantages of BBM are: (1) it enhances the length of the documents and makes the context more coherent by emphasizing the word connotation and co-occurrence via bag of biterms, and (2) it inherits inference and learning algorithms from the primitive to make it straightforward to design online and streaming algorithms for short texts. Extensive experiments suggest that BBM outperforms several state-of-the-art models. We also point out that the BoB representation performs better than the traditional representations (e.g., bag of words, tf-idf) even for normal texts.

Appendices

Appendix A: Supplementary experimental result

To strengthen the experimental result in Sect. 5, we conduct some experiments with another evaluation metrics besides log predictive probability (LPP), i.e., normalized pointwise mutual information (NPMI) [36]. We evaluate the NMI of the two models, online HDP-B and online LDA-B, compared with their base models using BoW and BoB. The settings and model in use are exactly the same as those in Sect. 5.1.1. We adopt the four datasets: Tweet, NYtimes, Yahoo and TMNTitle.

Fig. 19

The NPMI of online HDP-B and online HDP (with BoW and BoB)

Fig. 20

The NPMI of online LDA-B, Online BTM and Online LDA (with BoB and BoW)

Normalized pointwise mutual information (NPMI) A standard metric to measure the association between a pair of discrete outcomes x and y, defined as:

$$\begin{aligned} \text {NPMI}(w_i) = \sum _{j}^{N-1} \frac{\log \frac{P(w_i, w_j)}{P(w_i) P(w_j)}}{-\log P(w_i, w_j)} \end{aligned}$$

(9)

Figures 19 and 20 show the evaluation of these models by NPMI. Figure 19 confirms that OHDP-B performs better than HDP in almost all of the cases. Figure 19 shows that OLDA-B achieves better result than the other three models, while OBTM ranks behind only OLDA-B in NYtimes, Yahoo. Also, in most of the cases in both the two figures, the results of using BoW show that this representation is not suitable with short text datasets.

Appendix B: Conversion of topic-over-biterms (distribution over biterms) to topic-over-words (distribution over words)

In BoB, after we finish training the model, we obtain topics that are multinomial distributions over biterms. We would like to convert these topic-over-biterms to the topics-over-words (i.e., distribution over words). Assume that $\varvec{\phi }_k$ is a distribution over biterms of topic k. The procedure to perform this conversion is as follows: $p(w_i \mid z=k) = \sum _{j=1}^{V}p(w_i,w_j \mid z=k) = \sum _{j=1}^{V}p(b_{ij} \mid z=k) = \sum _{j=1}^{V}\phi _{kb_{ij}}$, where V is the vocabulary size in BoW and $b_{ij}$ is the biterm created from the pair ($w_i,w_j$).

As discussed in Sect. 3.1.1, in the implementation of BoB, we can merge $b_{ij}$ and $b_{ji}$ into $b_{ij}$ with $i<j$. Because of the identical occurrence in every document, after finishing the training process, the value of $p(b_{ij}\mid z=k)$ will be expectedly the same as $p(b_{ji}\mid z=k)$. Therefore, in grouping these biterms into one, the conversion version of this implementation is: $p(w_i\mid z=k) = \sum _{j=1}^{V}p(b_{ij} \mid z=k) = \phi _{kb_{ii}} + \frac{1}{2}\sum _{\text {b: biterms contain } w_i}\phi _{kb}$

Appendix C: Parameters inference for LDA-B

1.1 C.1 Lower bound function

The log likelihood is bounded by the lower bound induced from the Jensen inequality:

$$\begin{aligned} L&= \log P (\beta , \theta , z, \tilde{z} \mid \lambda , \gamma , \phi ) \ge E_{q} [ \log P(\beta , \theta , z, \tilde{z}, w \mid \eta , \alpha ] \\&\quad - E_{q} [\log q (\beta , \theta , z, \tilde{z} \mid \gamma , \lambda , \phi ) ] \end{aligned}$$

This lower bound can be written as follows:

$$\begin{aligned} L&= \sum _{k=1}^{K} E_{q} \left[ \log P (\beta _k \mid \eta _k) \right] + \sum _{d=1}^{D} E_{q} \left[ \log P (\theta _d \mid \alpha ) \right] + \sum _{d=1}^{D} \sum _{n=1}^{N_d} E_{q} \left[ \log P (z_{dn} \mid \theta _d) \right] \\&\quad + \sum _{d=1}^{D} \sum _{m=1}^{M_d} E_{q} \left[ \log P (\tilde{z}_{dm} \mid \theta _d) \right] + \sum _{d=1}^{D} \sum _{n=1}^{N_d} E_{q} \left[ \log P (w_{dn} \mid z_{dn}) \right] \\&\quad + \sum _{d=1}^{D} \sum _{m=1}^{M_d} E_{q} \left[ \log P \left( \tilde{w}^{(1)}_{dm}, \tilde{w}^{(2)}_{dm} \mid \tilde{z}_{dm}\right) \right] - \sum _{k=1}^{K} E_{q} \left[ \log q (\beta _k \mid \lambda _k) \right] \\&\quad - \sum _{d=1}^{D} E_{q} \left[ \log q (\theta _d \mid \gamma _d) \right] \\&\quad - \sum _{d=1}^{D} \sum _{n=1}^{N_d} E_{q} \left[ \log q (z_{dn} \mid \phi _{dn}) \right] - \sum _{d=1}^{D} \sum _{m=1}^{M_d} E_{q} \left[ \log q \left( \tilde{z}_{dm} \mid \tilde{\phi }_{dm}\right) \Big ) \right] \end{aligned}$$

where $\tilde{w}^{(1)}_{dm}$ and $\tilde{w}^{(2)}_{dm}$ denoted as the first word and second word of biterm $\tilde{w}_{dm}$. $\tilde{z}_{dm}$ is the topic assignment of $\tilde{w}_{dm}$. The expansion for this equation is similar to [2]. The only difference part lies in the biterm $(\tilde{w}^{(1)}_{dm}, \tilde{w}^{(2)}_{dm})$:

$$\begin{aligned} E_{q} \left[ \log P \left( \tilde{w}^{(1)}_{dm}, \tilde{w}^{(2)}_{dm} \mid \tilde{z}_{dm}\right) \right]&= E_{q} \left[ \log P \left( \tilde{w}^{(1)}_{dm} \mid \tilde{z}_{dm}\right) \right] + E_{q} \left[ \log P \left( \tilde{w}^{(2)}_{dm} \mid \tilde{z}_{dm}\right) \right] \\&= \sum _{v=1}^V \sum _{k=1}^K I\{ \tilde{w}^{(1)}_{dm} = v \} \tilde{\phi }_{dmk} \left( \psi (\lambda _{kv}) - \psi \left( \sum _{u=1}^{V} \lambda _{ku} \right) \right) \\&\quad + \sum _{v=1}^V \sum _{k=1}^K I\{ \tilde{w}^{(2)}_{dm} = v \} \tilde{\phi }_{dmk} \left( \psi (\lambda _{kv}) - \psi \left( \sum _{u=1}^{V} \lambda _{ku} \right) \right) . \end{aligned}$$

Then, we will maximize the lower bound function in each dimension of the variational parameters.

1.2 C.2 Variation parameter $\lambda$

Choose a topic index k. Fix $\gamma , \phi$ and each $\lambda _j$ for $j \ne k$. The updated value for parameter $\lambda$ is computed by set the derivative of $L(\lambda _k)$ to zero:

$$\begin{aligned} \lambda _{k, v} \leftarrow \eta _{k, v} + \sum _{d=1}^D \sum _{n=1}^{N_d} I\left\{ \tilde{w}^{(1)}_{dn} = v \right\} \phi _{dnk} + \sum _{d=1}^D \sum _{m=1}^{M_d} \left[ I\left\{ \tilde{w}^{(1)}_{dm} = v \right\} + I\left\{ \tilde{w}^{(2)}_{dm} = v \right\} \right] \tilde{\phi }_{dmk}. \end{aligned}$$

(10)

1.3 C.3 Variation parameter $\gamma$

Choose a document d, fix $\lambda , \phi$ and each $\gamma _c$ for $c \ne d$. Similar to $\lambda$, we obtain:

$$\begin{aligned} \gamma _{d, k} = \alpha _{k} + \sum _{n=1}^{N_d} \phi _{dnk} + \sum _{m=1}^{M_d} \tilde{\phi }_{dmk}. \end{aligned}$$

(11)

1.4 C.4 Variation parameter $\phi$

Fixing $\lambda , \gamma , \tilde{\phi }$ and $\phi _{cu}$ for $(c,u) \ne (d,v)$. Using the Lagrange multipliers method with constraint $\sum _{k=1}^K \phi _{dnk} = 1$, we have:

$$\begin{aligned} \phi _{dnk} \propto exp \left\{ E_{q} [\log \theta _{d, k}] + E_{q}\left[ \log \beta _{k,w_{dn}}\right] \right\} . \end{aligned}$$

(12)

1.5 C.5 Variation parameter $\tilde{\phi }$

Fixing $\lambda , \gamma , \phi$ and $\tilde{\phi }_{cu}$ for $(c,u) \ne (d,b)$. Using the Lagrange multipliers method with constraint $\sum _{k=1}^K \tilde{\phi }_{dmk} = 1$, we obtain:

$$\begin{aligned} \tilde{\phi }_{d, m, k} \propto exp \left\{ E_{q}[\log \theta _{d, k}] + E_{q}\left[ log \beta _{k,w_{dm}^{(1)}}\right] + E_{q}\left[ \log \beta _{k,w_{dm}^{(2)}}\right] \right\} . \end{aligned}$$

(13)

Here, we denote $\tilde{w}_{dm}^{(1)}$ and $\tilde{w}_{dm}^{(2)}$ as the first word and second word of biterm $\tilde{w}_{dm}$, respectively.