A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment

Word sense induction (WSI) seeks to automatically discover the senses of a word in a corpus via unsupervised methods. We propose a sense-topic model for WSI, which treats sense and topic as two separate latent variables to be inferred jointly. Topics are informed by the entire document, while senses are informed by the local context surrounding the ambiguous word. We also discuss unsupervised ways of enriching the original corpus in order to improve model performance, including using neural word embeddings and external corpora to expand the context of each data instance. We demonstrate significant improvements over the previous state-of-the-art, achieving the best results reported to date on the SemEval-2013 WSI task.


Introduction
Word sense induction (WSI) is the task of automatically discovering all senses of an ambiguous word in a corpus. The inputs to WSI are instances of the ambiguous word with its surrounding context. The output is a grouping of these instances into clusters corresponding to the induced senses. WSI is generally conducted as an unsupervised learning task, relying on the assumption that the surrounding context of a word indicates its meaning. Most previous work assumed that each instance is best labeled with a single sense, and therefore, that each instance belongs to exactly one sense cluster. However, recent work Jurgens, 2013) has shown that more than one sense can be used to interpret certain instances, due to context ambiguity and sense relatedness.
To handle these characteristics of WSI (unsupervised, senses represented by token clusters, multiple senses per instance), we consider approaches based on topic models. A topic model is an unsupervised method that discovers the semantic topics underlying a collection of documents. The most popular is latent Dirichlet allocation (LDA; Blei et al., 2003), in which each topic is represented as a multinomial distribution over words, and each document is represented as a multinomial distribution over topics.
One approach would be to run LDA on the instances for an ambiguous word, then simply interpret topics as induced senses (Brody and Lapata, 2009). However, while sense and topic are related, they are distinct linguistic phenomena. Topics are assigned to entire documents and are expressed by all word tokens, while senses relate to a single ambiguous word and are expressed through the local context of that word. One possible approach would be to only keep the local context of each ambiguous word, discarding the global context. However, the topical information contained in the broader context, though it may not determine the sense directly, might still be useful for narrowing down the likely senses of the ambiguous word.
Consider the ambiguous word cold. In the sentence "His reaction to the experiments was cold", the possible senses for cold include cold temperature, a cold sensation, common cold, or a negative emotional reaction. However, if we know that the topic of the document concerns the effects of low temperatures on physical health, then the negative emotional reaction sense should become less likely. Therefore, in this case, knowing the topic helps narrow down the set of plausible senses. At the same time, knowing the sense can also help determine possible topics. Consider a set of texts that all include the word cold. Without further information, the texts might discuss any of a number of possible topics. However, if the sense of cold is that of cold ischemia, then the most probable topics would be those related to organ transplantation.
In this paper, we propose a sense-topic model for WSI, which treats sense and topic as two separate latent variables to be inferred jointly ( §4). When relating the sense and topic variables, a bidirectional edge is drawn between them to represent their cyclic dependence (Heckerman et al., 2001). We perform inference using collapsed Gibbs sampling ( §4.2), then estimate the sense distribution for each instance as the solution to the WSI task. We conduct experiments on the SemEval-2013 Task 13 WSI dataset, showing improvements over several strong baselines and task systems ( §5).
We also present unsupervised ways of enriching our dataset, including using neural word embeddings (Mikolov et al., 2013) and external Web-scale corpora to enrich the context of each data instance or to add more instances ( §6). Each data enrichment method gives further gains, resulting in significant improvements over existing state-of-the-art WSI systems. Overall, we find gains of up to 22% relative improvement in fuzzy B-cubed and 50% relative improvement in fuzzy normalized mutual information (Jurgens and Klapaftis, 2013).

Background and Related Work
We discuss the WSI task, then discuss several areas of research that are related to our approach, including applications of topic modeling to WSI as well as other approaches that use word embeddings and clustering algorithms.
WSD and WSI: WSI is related to but distinct from word sense disambiguation (WSD). WSD seeks to assign a particular sense label to each target word instance, where the sense labels are known and usually drawn from an existing sense inventory like WordNet (Miller et al., 1990). Although extensive research has been devoted to WSD, WSI may be more useful for downstream tasks. WSD relies on sense inventories whose construction is time-intensive, expensive, and subject to poor inter-annotator agreement (Passonneau et al., 2010). Sense inventories also impose a fixed sense granularity for each ambiguous word, which may not match the ideal granularity for the task of interest. Finally, they may lack domain-specific senses and are difficult to adapt to low-resource domains or languages. In contrast, senses induced by WSI are more likely to represent the task and domain of interest. Researchers in machine translation and information retrieval have found that predefined senses are often not well-suited for these tasks (Voorhees, 1993;Carpuat and Wu, 2005), while induced senses can lead to improved performance (Véronis, 2004;Vickrey et al., 2005;Carpuat and Wu, 2007).
Topic Modeling for WSI: Brody and Lapata (2009) proposed a topic model that uses a weighted combination of separate LDA models based on different feature sets (e.g. word tokens, parts of speech, and dependency relations). They only used smaller units of text surrounding the ambiguous word, discarding the global context of each instance. Yao and Van Durme (2011) proposed a model based on a hierarchical Dirichlet process (HDP; Teh et al., 2006), which has the advantage that it can automatically discover the number of senses. Lau et al. (2012) described a model based on an HDP with positional word features; it formed the basis for their submission (unimelb, Lau et al., 2013) to the SemEval-2013 WSI task (Jurgens and Klapaftis, 2013).
Our sense-topic model is distinct from this prior work in that we model sense and topic as two separate latent variables and learn them jointly. We compare to the performance of unimelb in §5.
For word sense disambiguation, there also exist several approaches that use topic models (Cai et al., 2007;Li et al., 2010); space does not permit a full discussion.
Word Representations for WSI: Another approach to solving WSI is to use word representations built by distributional semantic models (DSMs; Sahlgren, 2006) or neural net language models (NNLMs; Bengio et al., 2003;Mnih and Hinton, 2007). Their assumption is that words with similar distributions have similar meanings. Akkaya et al. (2012) use word representations learned from DSMs directly for WSI. Each word is represented by a co-occurrence vector, and the meaning of an ambiguous word in a specific context is computed through element-wise multiplication applied to the vector of the target word and its surrounding words in the context. Then instances are clustered by hierarchical clustering based on their representations.
Word representations trained by NNLMs, often called word embeddings, capture information via training criteria based on predicting nearby words. They have been useful as features in many NLP tasks (Turian et al., 2010;Collobert et al., 2011;Dhillon et al., 2012;Hisamoto et al., 2013;Bansal et al., 2014). The similarity between two words can be computed using cosine similarity of their embedding vectors. Word embeddings are often also used to build representations for larger units of text, such as sentences, through vector operations (e.g., summation) applied to the vector of each token in the sentence. In our work, we use word embeddings to compute word similarities (for better modeling of our data distribution), to represent sentences (to find similar sentences in external corpora for data enrichment), and in a product-of-embeddings baseline. Baskaya et al. (2013) represent the context of each ambiguous word by using the most likely substitutes according to a 4-gram LM. They pair the ambiguous word with likely substitutes, project the pairs onto a sphere (Maron et al., 2010), and obtain final senses via k-means clustering. We compare to their SemEval-2013 system AI-KU ( §5).
Other Approaches to WSI: Other approaches include clustering algorithms to partition instances of an ambiguous word into sense-based clusters (Schütze, 1998;Pantel and Lin, 2002;Purandare and Pedersen, 2004), or graph-based methods to induce senses (Dorow and Widdows, 2003;Véronis, 2004;Agirre and Soroa, 2007). Figure 1: Proposed sense-topic model in plate notation. There are M D instances for the given target word. In an instance, there are N g global context words (w g ) and N local context words (w ), all of which are observed. There is one latent variable ("topic" t g ) for the w g and two latent variables ("topic" t and "sense" s ) for the w . Each instance has topic mixing proportions θ t and sense mixing proportions θ s . For clarity, not all variables are shown. The complete figure with all variables is given in Appendix A. This is a dependency network, not a directed graphical model, as shown by the directed arrows between t and s ; see text for details. dataset released for SemEval-2013 Task 13 (Jurgens and Klapaftis, 2013), collected from the Open American National Corpus (OANC; Ide and Suderman, 2004). 2 It includes 50 target words: 20 verbs, 20 nouns, and 10 adjectives. There are a total of 4,664 instances across all target words. Each instance contains only one sentence, with a minimum length of 22 and a maximum length of 100. The gold standard for the dataset was prepared by multiple annotators, where each annotator labeled instances based on the sense inventories in WordNet 3.1. For each instance, they rated all senses of a target word on a Likert scale from one to five.

A Sense-Topic Model for WSI
We now present our sense-topic model, shown in plate notation in Figure 1. It generates the words in the set of instances for a single target word; we run the model separately for each target word, sharing no parameters across target words. We treat sense and topic as two separate latent variables to be inferred jointly. To differentiate sense and topic, we use a window around the target word in each instance. Word tokens inside the window are local context words (w ), while tokens outside the window are global context words (w g ). The number of words in the window is fixed to 21 in all experiments (10 words before the target word and 10 after).
Generating global context words: As shown in the left part of Figure 1, each global context word w g is generated from a latent topic variable t g for the instance, which follows the same generative story as LDA. The corresponding probability of the ith global context word w where T is the number of topics, P ψt j (w is the multinomial distribution over words for topic j (parameterized by ψ t j ) and P θt (t Generating local context words: A local context word w is generated from a topic variable t and a sense variable s : where S is the number of senses, Pr(w |t = j, s = k) is the probability of generating word w given topic j and sense k, and Pr(t = j, s = k|d) is the joint probability over topics and senses for d. 4 Unlike in Eq.
(1), we do not use multinomial parameterizations for the distributions in Eq. (2). When parameterizing them, we make several departures from purely-generative modeling. All our choices result in distributions over smaller event spaces and/or those that condition on fewer variables. This helps to mitigate data sparsity issues arising from attempting to estimate highdimensional distributions from small datasets. A secondary benefit is that we can avoid biases caused by particular choices of generative directionality in the model. We later include an empirical comparison to justify some of our modeling choices ( §5).
First, when relating the sense and topic variables, we avoid making a single decision about generative dependence. Taking inspiration from dependency networks (Heckerman et al., 2001), we use the following factorization: where Z d is a normalization constant. We factorize further by using redundant probabilistic events, then ignore the normalization constants during learning, a concept commonly called deficiency (Brown et al., 1993). Deficient modeling has been found to be useful for a wide range of NLP tasks (Klein and Manning, 2002;May and Knight, 2007;Toutanova and Johnson, 2007). In particular, we factor the conditional probabilities in Eq. (3) into products of multinomial probabilities: where Z d,t j and Z d,s k are normalization factors and we have introduced new multinomial parameters θ s , θ s|t j , θ st , and θ t|s k . We use the same idea to factor the word generation distribution: where Z t j ,s k is a normalization factor, and we have new multinomial parameters ψ s k for the sense-word distributions. One advantage of this parameterization is that we naturally tie the topic-word distributions across the global and local context words by using the same parameters ψ t j .

Generative Story
We now give the full generative story of our model. We describe it for generating a set of instances of size M D , where all instances contain the same target word. We use symmetric Dirichlet priors for all multinomial distributions mentioned above, using the same fixed hyperparameter value (α) for all. We use ψ to denote parameters of multinomial distributions over words, and θ to denote parameters of multinomial distributions over topics and/or senses. We leave unspecified the distributions over N (number of local words in an instance) and N g (number of global words in an instance), as we only use our model to perform inference given fixed instances, not to generate new instances. The generative story first follows the steps described in Algo. 1 to generate parameters that are shared across all instances; then for each instance d, it follows Algo. 2 to generate global and local words.
Algorithm 1 Generative story for instance set 1: for each topic j ← 1 to T do 2: Choose topic-word params. ψ tj ∼ Dir(α)

Inference
We use collapsed Gibbs sampling (Geman and Geman, 1984) to obtain samples from the posterior distribution over latent variables, with all multinomial parameters analytically integrated out before sampling. Then we estimate the sense distribution θ s for each instance using maximum likelihood estimation on the samples. These sense distributions are the output of our WSI system.
We note that deficient modeling does not ordinarily affect Gibbs sampling when used for computing posteriors over latent variables, as long as parameters (the θ and ψ) are kept fixed. This is the case during the E step of an EM algorithm, which is the usual setting in which deficiency is used. Only the M step is affected; it becomes an approximate M step by assuming the normalization constants equal 1 (Brown et al., 1993).
However, here we use collapsed Gibbs sampling for posterior inference, and the analytic integration is disrupted by the presence of the normalization constants. To bypass this, we employ the standard approximation of deficient models that all normalization constants are 1, permitting us to use standard formulas for analytic integration of multinomial parameters with Dirichlet priors. Empirically, we found this "collapsed deficient Gibbs sampler" to slightly outperform a more principled approach based on EM, presumably due to the ability of collapsing to accelerate mixing.
During the sampling process, each sampler is run on the full set of instances for a target word, iterating through all word tokens in each instance. If the current word token is a global context word, we sample a new topic for it conditioned on all other latent variables across instances. If the current word is a local context word, we sample a new topic/sense pair for it again conditioned on all other latent variable values.
We write the conditional posterior distribution over topics for global context word token i in instance d as Pr(t is the topic assignment of token i, d is the current instance, t −i is the set of topic assignments of all word tokens aside from i for instance d, s is the set of sense assignments for all local word tokens in instance d, and "·" stands for all other observed or known information, including all words, all Dirichlet hyperparameters, and all latent variable assignments in other instances. The conditional posterior can be computed by: where we use the superscript DT as a mnemonic for "instance/topic" when counting topic assignments in an instance and W T for "word/topic" when counting topic assignments for a word. C DT dj contains the number of times topic j is assigned to some word token in instance d, excluding the current word token w g is assigned to topic j, across all instances, excluding the current word token. W t is the number of distinct word types in the full set of instances. We show the corresponding conditional posterior probabilities underneath each term; the count ratios are obtained using standard Dirichlet-multinomial collapsing. The conditional posterior distribution over topic/sense pairs for a local context word token w (i) can be computed by: where C DS dk contains the number of times sense k is assigned to some local word token in instance d, excluding the current word token; C W S ik contains the number of time word w (i) is assigned to sense k, excluding the current time; C ST kj contains the number of times sense k and topic j are assigned to some local word tokens. W s is the number of distinct local context word types across the collection.
Decoding After the sampling process, we obtain a fixed-point estimate of the sense distribution (θ s ) for each instance d using the counts from our samples. Where we use θ k s to denote the probability of sense k for the instance, this amounts to: This distribution is considered the final sense assignment distribution for the target word in instance d for the WSI task; the full distribution is fed to the evaluation metrics defined in the next section.
To inspect what the model learned, we similarly obtain the sense-word distribution (ψ s ) from the counts as follows, where ψ i s k is the probability of word type i given sense k:

Experimental Results
In this section, we evaluate our sense-topic model and compare it to several strong baselines and stateof-the-art systems.
Evaluation Metrics To evaluate WSI systems, Jurgens and Klapaftis (2013) propose two metrics: fuzzy B-cubed and fuzzy normalized mutual information (NMI). They are each computed separately for each target word, then averaged across target words. Fuzzy B-cubed prefers labeling all instances with the same sense, while fuzzy NMI prefers the opposite extreme of labeling all instances with distinct senses. Hence, we report both fuzzy B-cubed (%) and fuzzy NMI (%) in our evaluation. For ease of comparison, we also report the geometric mean of the 2 metrics, which we denote by AVG. 5 SemEval-2013 Task 13 also provided a trial dataset (TRIAL) that consists of eight target ambiguous words, each with 50 instances  Hyperparameter Tuning We use TRIAL to analyze performance of our sense-topic model under different settings for the numbers of senses (S) and topics (T ); see Table 1. We always set T = 2S for simplicity. We find that small S values work best, which is unsurprising considering the relatively small number of instances and small size of each instance. When evaluating on TEST, we use S = 3 (which gives the best AVG results on TRIAL). Later, when we add larger context or more instances (see §6), tuning on TRIAL chooses a larger S value. During inference, the Gibbs sampler was run for 4,000 iterations for each target word, setting the first 500 iterations as the burn-in period. In order to get a representative set of samples, every 13th sample (after burn-in) is saved to prevent correlations among samples. Due to the randomized nature of the inference procedure, all reported results are average scores over 5 runs. The hyperparameters (α) for all Dirichlet priors in our model are set to the (untuned) value of 0.01, following prior work on topic modeling (Griffiths and Steyvers, 2004;Heinrich, 2005).
Baselines We include two naïve baselines corresponding to the two extremes (biases) preferred by fuzzy B-cubed and NMI, respectively: 1 sense (label each instance with the same single sense) and all distinct (label each instance with its own sense).
We also consider two baselines based on LDA. We run LDA for each target word in TEST, using the set of instances as the set of documents. We treat the learned topics as induced senses. When setting the number of topics (senses), we use the gold-standard number of senses for each target word, making this baseline unreasonably strong. We run LDA both with full context (FULL) and local context (LOCAL), using the same window size as above (10 words before and after the target word).
We also present results for the two best systems in the SemEval-2013 task (according to fuzzy Bcubed and fuzzy NMI, respectively): unimelb and AI-KU. As described in Section 2, unimelb uses hierarchical Dirichlet processes (HDPs). It extracts 50,000 extra instances for each target word as training data from the ukWac corpus−a web corpus of approximately 2 billion tokens. 6 Among all systems in the task, it performs best according to fuzzy Bcubed. AI-KU is based on a lexical substitution method; a language model is built to identify lexical substitutes for target words from the dataset and the ukWac corpus. It performed best among all systems according to fuzzy NMI.

Results
In Table 2, we present results for these systems and compare them to our basic (i.e., without any data enrichment) sense-topic model with S = 3 (row 9). According to both fuzzy B-cubed and fuzzy NMI, our model outperforms the other WSI systems (LDA, AI-KU, and unimelb). Hence, we are able to achieve state-of-the-art results on the SemEval-2013 task even when only using the single sentence of context given in each instance (while AI-KU and unimelb use large training sets from ukWac). We found similar performance improvements when only tested on instances labeled with a single sense.
Bidirectionalilty Analysis To measure the impact of the bidirectional dependency between the topic and sense variables in our model, we also evaluate the performance of our sense-topic model when dropping one of the directions. In Table 3, we compare their performance with our full sense-topic model on TEST. Both unidirectional models perform worse than the full model, and dropping t → s hurts more. This result verifies our intuition that topics would help narrow down the set of likely senses, and suggests that bidirectional modeling between topic and sense is desirable for WSI.
In subsequent sections, we investigate several ways of exploiting additional data to build betterperforming sense-topic models.

Unsupervised Data Enrichment
The primary signal used by our model is word cooccurrence information across instances. If we en-   rich the instances, we can have more robust cooccurrence statistics. The SemEval-2013 dataset may be too small to induce meaningful senses, since there are only about 100 instances for each target word, and each instance only contains one sentence. This is why most shared task systems added instances from external corpora. In this section, we consider three unsupervised ways of enriching data and measure their impact on performance. In §6.1 we augment the context of each instance in our original dataset while keeping the number of instances fixed. In §6.2 we collect more instances of each target word from ukWac, similar to the AI-KU and unimelb systems. In §6.3, we change the distribution of words in each instance based on their similarity to the target word.
Throughout, we make use of word embeddings (see §2). We trained 100-dimensional skip-gram vectors (Mikolov et al., 2013) on English Wikipedia (tokenized/lowercased, resulting in 1.8B tokens of text) using window size 10, hierarchical softmax, and no downsampling. 7 7 We used a minimum count cutoff of 20 during training,

Adding Context
The first way we explore of enriching data is to add a broader context for each instance while keeping the number of instances unchanged. This will introduce more word tokens into the set of global context words, while keeping the set of local context words mostly unchanged, as the window size we use is typically smaller than the length of the original instance. With more global context words, the model has more evidence to learn coherent topics, which could also improve the induced senses via the connection between sense and topic.
The ideal way of enriching context for an instance is to add its actual context from the corpus from which it was extracted. To do this for the SemEval-2013 task, we find each instance in the OANC and retrieve three sentences before the instance and three sentences after. While not provided for the SemEval task, it is reasonable to assume this larger context in many real-world applications, such as information retrieval and machine translation of documents.
However, in other settings, the corpus may only have a single sentence containing the target word (e.g., search queries or machine translation of sentences). To address this, we find a semanticallysimilar sentence from the English ukWac corpus and append it to the instance as additional context. For each instance in the original dataset, we extract its then only retained vectors for the most frequent 100,000 word types, averaging the rest to get a vector for unknown words. most similar sentence that contains the same target word and add it to increase its set of global context words. To compute similarity, we first represent instances and ukWac sentences by summing the word embeddings across their word tokens, then compute cosine similarity. The ukWac sentence (s * ) with the highest cosine similarity to each original instance (d) is appended to that instance: Results Since the vocabulary has increased, we expect we may need larger values for S and T . On TRIAL, we find best performance for S = 10, so we run on TEST with this value. Performance is shown in Table 2 (rows 10 and 11). These two methods have higher AVG scores than all others. Both their fuzzy B-cubed and NMI improvements over the baselines and previous WSI systems are statistically significant, as measured by a paired bootstrap test (p < 0.01; Efron and Tibshirani, 1994).
It is unsurprising that we find best performance with actual context. Interestingly, however, we can achieve almost the same gains when automatically finding relevant context from a different corpus. Thus, even in real-world settings where we only have a single sentence of context, we can induce substantially better senses by automatically broadening the global context in an unsupervised manner.
As a comparative experiment, we also evaluate the performance of LDA when adding actual context (Table 2, row 7). Compared with LDA with full context (FULL) in row 6, performance is slightly improved, perhaps due to the fact that longer contexts induce more accurate topics. However, those topics are not necessarily related to senses, which is why LDA with only local context actually performs best among all three LDA models. Thus we see that merely adding context does not necessarily help topic models for WSI. Importantly, since our model includes both sense and topic, we are able to leverage the additional context to learn better topics while also improving the quality of the induced senses, leading to our strongest results.

Examples
We present examples to illustrate our sense-topic model's advantage over LDA and the further improvement when adding actual context. (1) and (2) below, with target word occurrences in bold:

Consider instances
(1) Nigeria then sent troops to challenge the coup, evidently to restore the president and repair Nigeria's corrupt image abroad. (image%1:07:01::/4) 8 (2) When asked about the Bible's literal account of creation, as opposed to the attractive concept of divine creation, every major Republican presidential candidate-even Bauer-has squirmed, ducked, and tried to steer the discussion back to "faith," "morals," and the general idea that humans "were created in the image of God." (image%1:06:00::/2 image%1:09:02::/4) Both instances share the common word stem president. LDA uses this to put these two instances into the same topic (i.e., sense). In our sense-topic model, president is a local context word in instance (1) but a global context word in instance (2). So the effect of sharing words is decreased, and these two instances are assigned to different senses by our model. According to the gold standard, the two instances are annotated with different senses, so our sense-topic model provides the correct prediction. Next, consider instances (3), (4), and (5): (3) I have recently deliberately begun to use variations of "kick ass" and "bites X in the ass" because they are colorful, evocative phrases; because, thanks to South Park, ass references are newly familiar and hilarious and because they don't evoke particularly vivid mental image of asses any longer. (im-age%1:09:00::/4) (4) Also, playing video games that require rapid mental rotation of visual image enhances the spatial test scores of boys and girls alike. (image%1:06:00::/4) (5) Practicing and solidifying modes of representation, Piaget emphasized, make it possible for the child to free thought from the here and now; create larger images of reality that take into account past, present, and future; and transform those image mentally in the service of logical thinking. (im-age%1:09:00::/4) In the gold standard, instances (3) and (4) have different senses while (3) and (5) have the same sense. However, sharing the local context word "mental" triggers both LDA and our sense-topic model to assign them to the same sense label with high probability. When augmenting the instances by their real contexts, we have a better understanding about the topics. Instance (3) is about phrase variations, instance (4) is about enhancing boys' spatial skills, while instance (5) discusses the effect of makebelieve play for children's development.
When LDA is run with the actual context, it leaves (4) and (5) in the same topic (i.e., sense), while assigning (3) into another topic with high probability. This could be because (4) and (5) both relate to child development, and therefore LDA considers them as sharing the same topic. However, topic is not the same as sense, especially when larger contexts are available. Our sense-topic model built on the actual context makes correct predictions, leaving (3) and (5) into the same sense cluster while labeling (4) with a different sense.

Adding Instances
We also consider a way to augment our dataset with additional instances from an external corpus. We have no gold standard senses for these instances, so we will not evaluate our model on them; they are merely used to provide richer co-occurrence statistics about the target word so that we can perform better on the instances on which we evaluate.
If we added randomly-chosen instances (containing the target word), we would be concerned that the learned topics and senses may not reflect the distributions of the original instance set. So we only add instances that are semantically similar to instances in our original set (Moore and Lewis, 2010;Chambers and Jurafsky, 2011). Also, to avoid changing the original sense distribution by adding too many instances, we only add a single instance for each original instance. As in §6.1, for each instance in the original dataset, we find the most similar sentence in ukWac for each instance using word embeddings and add it into the dataset. Therefore, the number of instances is doubled, and we use the enriched dataset for our sense-topic model.

Results
Similarly to §6.1, on TRIAL, we find best performance for S = 10, so we run on TEST with this value. As shown in Table 2 (row 12), this improves fuzzy B-cubed by 5.4%, but fuzzy NMI is lower, making the AVG worse than the original model. A possible reason for this is that the sense distribution in the added instances disturbs that in the original set of instances, even though we picked the most semantically similar ones to add.

Weighting by Word Similarity
Another approach is inspired by the observation that each local context token is treated equally in terms of its contribution to the sense. However, our intuition is that certain tokens are more indicative than others. Consider the target word window. Since glass evokes a particular sense of window, we would like to weight it more highly than, say, day.
To measure word relatedness, we use cosine similarity of word embeddings. We (softly) replicate each local context word according to its exponentiated cosine similarity to the target word. 9 The result is that the local context in each instance has been modified to contain fewer occurrences of unrelated words and more occurrences of related words. If each cosine similarity is 0, we obtain our original sense-topic model. During inference, the posterior sense distribution for instance d is now given by: Pr(s = k|d, ·) = w∈d exp(sim(w, w * ))1 sw=k + α w ∈d exp(sim(w , w * )) + Sα (8) where d is the set of local context tokens in d, sim(w, w * ) is the cosine similarity between w and target word w * , and 1 sw=k is an indicator returning 1 when w is assigned to sense k and 0 otherwise. The posterior distribution of sampling a token of word w i from sense k becomes: where C W S ik counts the number of times w i is assigned to sense k.

Results
We again use TRIAL to tune S (and still use T = 2S). We find best TRIAL performance at S = 3; this is unsurprising since this approach does not change the vocabulary. In Table 2, we present results on TEST with S = 3 (row 13). We also report Sense Top-5 terms per sense Sense-Topic Model 1 include, depict, party, paint, visual 2 zero, manage, company, culture, figure  3 create, clinton, people, american, popular +weight by similarity ( §6.3) 1 depict, create, culture, mental, include 2 picture, visual, pictorial, matrix, movie 3 public, means, view, american, story an additional baseline: "word embedding product" (row 8), where we represent each instance by multiplying (element-wise) the word vectors of all local context words, and then feed the instance vectors into the fuzzy c-means clustering algorithm (Pal and Bezdek, 1995), c = 3. Compared to this baseline, our approach improves 4.36% on average; compared with results for the original sense-topic model (row 9), this approach improves 0.69% on average. In Table 4 we show the top-5 terms for each sense induced for image, both for the original sense-topic model and when additionally weighting by similarity. We find that the original model provides less distinguishable senses, as it is difficult to derive separate senses from these top terms. In contrast, senses learned from the model with weighted similarities are more distinct. Sense 1 relates to mental representation; sense 2 is about visual representation produced on a surface; and sense 3 is about the general impression that something presents to the public.

Conclusions and Future Work
We presented a novel sense-topic model for the problem of word sense induction. We considered sense and topic as distinct latent variables, defining a model that generates global context words using topic variables and local context words using both topic and sense variables. Sense and topic are related using a bidirectional dependency with a robust parameterization based on deficient modeling.
We explored ways of enriching data using word embeddings from neural language models and external corpora. We found enriching context to be most effective, even when the original context of the instance is not available. Evaluating on the SemEval-2013 WSI dataset, we demonstrate that our model yields significant improvements over current stateof-the-art systems, giving 59.1% fuzzy B-cubed and 9.39% fuzzy NMI in our best setting. Moreover, we find that modeling both sense and topic is critical to enable us to effectively exploit broader context, showing that LDA does not improve when each instance is enriched by actual context.
In future work, we plan to further explore the space of sense-topic models, including non-deficient models. One possibility is to use "switching variables" (Paul and Girju, 2009) to choose whether to generate each word from a topic or sense, with a stronger preference to generate from senses closer to the target word. Another possibility is to use locallynormalized log-linear distributions and include features pairing words with particular senses and topics, rather than redundant generative steps.

Appendix A
The plate diagram for the complete sense-topic model is shown in Figure 2. Figure 2: Plate notation for the proposed sense-topic model with all variables (except α, the fixed Dirichlet hyperparameter used as prior for all multinomial distributions). Each instance has topic mixing proportions θ t and sense mixing proportions θ s . The instance set shares sense/topic parameter θ st , topic-sense distribution θ s|t , sense-topic distribution θ t|s , topic-word distribution ψ t , and sense-word distribution ψ s .