Entity Disambiguation with Web Links

Entity disambiguation with Wikipedia relies on structured information from redirect pages, article text, inter-article links, and categories. We explore whether web links can replace a curated encyclopaedia, obtaining entity prior, name, context, and coherence models from a corpus of web pages with links to Wikipedia. Experiments compare web link models to Wikipedia models on well-known conll and tac data sets. Results show that using 34 million web links approaches Wikipedia performance. Combining web link and Wikipedia models produces the best-known disambiguation accuracy of 88.7 on standard newswire test data.


Introduction
Entity linking (EL) resolves mentions in text to their corresponding node in a knowledge base (KB), or NIL if the entity is not in the KB.Wikipedia and related semantic resources -Freebase, DBpedia, Yago2-have emerged as general repositories of notable entities.The availability of Wikipedia, in particular, has driven work on EL, knowledge base population (KBP), and semantic search.This literature demonstrates that the rich structure of Wikipediaredirect pages, article text, inter-article links, categories -delivers disambiguation accuracy above 85% on newswire (He et al., 2013;Alhelbawy and Gaizauskas, 2014).But what disambiguation accuracy can we expect in the absence of Wikipedia's curated structure?
Web links provide much of the same information as Wikipedia inter-article links: anchors are used to derive alternative names and conditional probabilities of entities given names; in-link counts are used to derive a simple entity popularity measure; the text surrounding a link is used to derive textual context models; and overlap of in-link sources is used to derive entity cooccurrence models.On the other hand, web links lack analogues of additional Wikipedia structure commonly used for disambiguation, e.g., categories, encyclopaedic descriptions.Moreover, Wikipedia's editors ensure a clean and correct knowledge source while web links are a potentially noisier annotation source.
We explore linking with web links versus Wikipedia.Contributions include: (1) a new benchmark linker that instantiates entity prior probabilities, entity given name probabilities, entity context models, and efficient entity coherence models from Wikipedia-derived data sets; (2) an alternative linker that derives the same model using only alternative names and web pages that link to Wikipedia; (3) detailed development experiments, including analysis and profiling of Web link data, and a comparison of link and Wikipedia-derived models.
Results suggest that web link accuracy is at least 93% of a Wikipedia linker and that web links are complementary to Wikipedia, with the best scores coming from a combination.We argue that these results motivate open publishing of enterprise authorities and suggest that accumulating incoming links should be prioritised at least as highly as adding richer internal structure to an authority.and beyond Wikipedia.Jin et al. (2014) describe an unsupervised system for linking to a person KB from a social networking site, and Shen et al. (2014) describe a general approach for arbitrary KBs.Nakashole et al. (2013) and Hoffart et al. (2014) add a temporal dimension to NIL detection by focusing on discovering and typing emerging entities.

Tasks and art
Two evaluations in particular have driven comparative work on EL: the TAC KBP shared tasks and the Yago2 annotation of CoNLL 2003 NER data.We describe these tasks and their respective evaluation setup.A brief survey of results outlines the kind of performance we hope to achieve with link data.For task history, we suggest Hachey et al. (2013) and Shen et al. (2014).For an evaluation survey, see Hachey et al. (2014).
Our evaluation setup follows He et al. (2013) for comparability to their state-of-the-art disambiguation results across CoNLL and TAC data.Table 1 summarises the data sets used.Columns correspond to number of documents (|D|), number of entities (|E|), number of mentions (|M|), and number of non-NIL mentions (|M KB |).The non-NIL mention number represents the set used for evaluation in the disambiguation experiments here.The table also includes average and standard deviation of the candidate set cardinality over M KB ( C ) and the percentage of mentions in M KB where the correct resolution is in the candidate set (R C ).The last column (SOA) gives the state-of-the-art score from the literature.Numbers are discussed below.The standard evaluation measure is precision@1 (p@1) -the percentage of linkable mentions for which the system ranks the correct entity first (Hoffart et al., 2011).Linkable is defined as ground truth mentions for which the correct entity is a member of the candidate set.This factors out errors due to mention detection, coreference handling, and candidate generation, isolating the performance of the proposed ranking models.For comparability, we use There are several key benchmark results for the CoNLL data set.Hoffart et al. (2011) define the task settings and report the first results.They employ a global graph-based coherence algorithm, leading to a score of 82.5.He et al. (2013) present the most comparable approach.Using deep neural networks, they learn entity representations based on similarity between link contexts and article text in Wikipedia.They report performance of 84.8 without collective inference, and 85.6 when integrating Han et al.'s (2011) coherence algorithm.Finally, Alhelbawy and Gaizauskas (2014) report the current best performance of 87.6 using a collective approach over a document-specific subgraph.

TAC 2010
Since 2009, the Text Analysis Conference (TAC) has hosted an annual EL shared task as part of its Knowledge Base Population track (KBP) (Ji and Grishman, 2011).Through 2013, the task is query-driven.Input includes a document and a name that appears in that document.Systems must output a KB identifier for each query, or NIL.The KB is derived from a subset of 818,741 Wikipedia articles.We use data from the 2010 shared task for several reasons.First, it facilitates comparison to current art.Second, it is a linking-only evaluation as opposed to linking plus NIL clustering.Finally, it includes comparable training and test data rather than relying on data from earlier years for training.
The TAC 2010 source collection includes news from various agencies and web log data.Training data includes a specially prepared set of 1,500 web queries.Test data includes 2,250 queries -1,500 news and 750 web log uniformly distributed across person, organisation, and geo-political entities.Candidate generation here uses the DBpedia lexicalizations data set (Mendes et al., 2012), article titles, and redirect titles.We also add titles and redirects stripped of appositions indicated by a comma (e.g., Montgomery, Alabama) or opening round bracket (e.g., Joe Morris (trumpeter)).Candidate recall is 94.4 and 88.5 on the training and test sets -an upper limit on disambiguation accuracy.
Following He et al., we report KB accuracy (A KB ) -the percentage of correctly linked non-NIL mentions -to isolate disambiguation performance.Before evaluation, we map Wikipedia titles in our output to TAC KB identifiers using the Dalton and Dietz (2013) alignment updated with Wikipedia redirects.To our knowledge, Cucerzan (2011) report the best A KB of 87.3 for an end-to-end TAC entity linking system, while He et al. (2013) report the best A KB of 81.0 for a disambiguation-focused evaluation.There are a number of differences, e.g.: mention detection for coherence, coreference modelling, and substring matching in candidate generation.Analysis shows that these can have a large effect on system performance (Hachey et al., 2013;Piccinno and Ferragina, 2014).We use

Wikipedia benchmark models
A wide range of EL approaches have been proposed that take advantage of the clean, well-edited information in Wikipedia.These include entity prior models derived from popularity metrics; alias models derived from Wikipedia redirects, disambiguation pages and inter-article links; textual context models derived from Wikipedia article text; and entity coherence models derived from the Wikipedia inter-article link graph.We survey these models and describe a new benchmark linker that instantiates them from existing Wikipedia-derived data sets.For a more detailed survey of features in supervised systems, see Meij et al. (2012) and Radford (2014).Table 2 contains an overview of p@1 results for individual components on the CoNLL development data.

Entity prior
The simplest approach to entity disambiguation ranks candidate entities in terms of their popularity.For example, 0.000001% of inter-article links in Wikipedia point to Nikola Tesla, while 0.000008% point to Tesla Motors.An entity prior is used in generative models (Guo et al., 2009;Han and Sun, 2011) and in supervised systems that incorporate diverse features (Radford et al., 2012).We define the entity prior as the probability of a link pointing to entity e: where I * ,e ∈ I * , * is the set of pages that link to entity e.We derive this from DBpedia's Wikipedia Pagelinks data set, which contains the link graph between Wikipedia pages.2Missing values are replaced with a small default log probability of -20, which works better than add-one smoothing in development experiments.On the CoNLL development data, entity prior alone achieves 68.4 p@1.

Name probability
Name probability models the relationship between a name and an entity.For example, 0.04% of links with the anchor text 'Tesla' point to Nikola Tesla, while 0.03% point to Tesla Motors.Name probability was introduced as an initial score in coherence-driven disambiguation (Milne and Witten, 2008), and is used in most state-of-the-art systems (Ferragina and Scaiella, 2010;Hoffart et al., 2011;Cucerzan, 2011;Radford et al., 2012).We define name probability as the conditional probability of a name referring to an entity: where M n,e is the set of mentions with name n that refer to entity e and M n, * is all mentions with name n.We use existing conditional probability estimates from the DBpedia Lexicalizations data set (Mendes et al., 2012). 2 This derives mentions from Wikipedia inter-article links, where names come from anchor text and referent entities from link targets.Estimates for entities that have fewer than five incoming links are discarded.We smooth these estimates using addone smoothing.On the CoNLL development data, name probability alone achieves 69.2 p@1.

Textual context
Textual context goes beyond intrinsic entity and name popularity, providing a means to distinguish between entities based on the words with which they occur.For example, references to Tesla the car manufacturer appear in passages with words like 'company', 'electric', 'vehicle'.References to the inventor appear with words like 'engineer', 'ac', 'electrical'.Textual context was the primary component of the top system in the first TAC evaluation (Varma et al., 2009), and is a key component in recent art (Ratinov et al., 2011;Radford et al., 2012).
BOW context We model textual context as a weighted bag of words (BOW), specifically as a term vector t containing tf idf weights: where t is a term, p is a passage of text, f (t, p) is the term frequency of t in p, |D| is the total number of documents, and {d ∈ D|t ∈ d} is the number of documents containing t (Salton and Buckley, 1988).We derive the term frequency for an entity e from the corresponding article content in the Kopiwiki plain text extraction (Pataki et al., 2012).Terms include three million token 1-3 grams from Mikolov et al. (2013), with the top 40 by document frequency as stop words.Candidate entities are scored using cosine distance between a mention context t m and the entity model t e : On the CoNLL development data, BOW context derived from Wikipedia article text achieves 50.6 p@1.
We also build entity models from their mention contexts, i.e., the combined text surrounding all incoming links.We project mentions into Kopiwiki article text, which yields more contexts than actual Wikipedia links.For an article a, we tag as mentions all aliases of entities linked to from a.We use aliases from Yago2 means relations (see Section 3.1).To ensure high precision, we only use aliases that are unambiguous with respect to the outlink set, have a length of at least two characters, include at least one upper-case character, and are not a member of the NLTK stop list.This is a noisy process, but gives us a pivot to assess whether differences observed later between Wikipedia and Web link models are due the way the context is modelled or the source of the context.The term frequency for an entity e is calculated over the concatenation of all contexts for e. BOW context derived from mentions achieves 55.8 p@1 on the CoNLL development data, five points higher than article text.
DBOW context While BOW context models have been very successful, they require exact matching between terms and a large vocabulary.Distributional approaches model terms or concepts as se-mantic vectors (Pereira et al., 1993).Dimensionality reduction and deep learning improve generalisation and reduce vector size (Baroni et al., 2014).He et al. (He et al., 2013) report excellent performance using entity representations that optimise the similarity between mention contexts and article text in Wikipedia.However, this approach necessitates an expensive training process and significant run-time complexity.We introduce a simple distributed bagof-words (DBOW) model that represents context as the tf idf -weighted average over word vectors V: where T p is the set of terms in passage p, and v t ∈ V is the learnt word vector for term t.We use existing 300-dimensional word embeddings (Mikolov et al., 2013) and score candidates using cosine distance between mention context v m and the entity model v e : On the CoNLL development data, DBOW context models derived from article text and mention context achieve 49.9 and 51.2 respectively.

Web link models
The models above all have direct anologues in web links to Wikipedia articles.However, web links are a comparatively noisy source.For instance, anchors are less likely to be well-formed entity mentions, e.g., in links to Semantic Web we observe 'semantic markup' and 'Semantic Web Activity' as anchors.A lack of curation and quality control also allows for the misdirection of links.For example, we observe links to Apple the fruit where the surrounding context indicates an intention to link Apple Inc instead.
It is an open question whether link-derived models are effective in disambiguation.Below, we describe how models are instantiated using link data.We leverage the Wikilinks corpus of 9 million web pages containing a total of 34 million links to 1.7 million Wikipedia pages (Singh et al., 2012).This includes links to English Wikipedia pages that pass the following tests: (1) the page must not have >70% of sentences in common with a Wikipedia article; (2) the link must not be inside  a table, near an image, or in obvious boilerplate material; (3) at least one token in the anchor text must match a token in the Wikipedia title; and (4) the anchor text must match a known alias from Wikipedia.
The corpus provides the web page URL, the link anchor, and local textual content around each link.
Refer back to Table 2 for p@1 results for individual Web link components on the development data.

Entity prior
To instantiate f prior , we build a page-entity link graph from Wikilinks.Where pages and entities are the same in the Wikipedia graph, here we have an unweighted bipartite graph of links from web pages to Wikipedia articles.On the CoNLL development data, the link-derived entity prior achieves 63.0 p@1.Table 3 characterises the two graphs.Note that the high entity count for Wikipedia here includes red links to articles that do not exist.The actual number of entities used in the Wikipedia model is 4.4 million.Nevertheless, while the two graphs have a similar number of pages that contain links, Wikipedia includes three times as many link pairs to 2.5 times as many entities.Furthermore, entities average 11.5 incoming links in the Wikipedia graph, compared to 3.5 in the Wikilinks graph.Nevertheless, the individual performance of the Web link prior is only 5.4 points shy of the corresponding Wikipedia prior.
Relative frequencies in Wikipedia and Wikilinks are similar, especially for entities that show up in the evaluation data.We observe a moderate correlation between entity priors from Wikipedia and Wikilinks (ρ = 0.51, p < 0.01), and a strong correlation across the subset of entities that occur in the development data (ρ = 0.74, p < 0.01).

Name probability
To instantiate f name , we build a name-entity graph from Wikilinks.The structure is the same as the cor-Wikipedia Web links Names 1.4m 3.1m Entities 1.5m 1.7m responding model from Wikipedia, both are bipartite graphs with cooccurrence frequencies on edges.However, names here are sourced from link anchors in web pages rather than Wikipedia articles.For comparability with the Wikipedia model, we ignore links to entities that occur fewer than five times.We observed no improvement using all links in development experiments.On the CoNLL development data, link-derived name probability achieves 58.4 p@1, more than ten points shy of the Wikipedia-derived name probability.Table 4 helps to explain this difference.Wikilinks has twice as many names linking to the same number of entities, resulting in more ambiguity and sparser models.

Textual context
To instantiate f bow and f dbow , we follow the same methodology used for Wikipedia mention contexts.The term frequency for an entity e is calculated over the concatenation of mention contexts for e. Document frequency is also calculated across aggregated entity contexts.Mention contexts include all text included in the Wikilinks data, a window of 46 tokens on average centred on the link anchor.Section 4.3 showed that Wikipedia mention contexts give better individual performance than Wikipedia article texts.Web link mentions result in even better performance.On the CoNLL development data, BOW context achieves 62.2 p@1, ten points higher than commonly used Wikipedia article model and seven points higher than the analogous Wikipedia mention model.DBOW context achieves 54.0 p@1, 2.8 points higher than the Wikipedia mention model.

Learning to rank
To perform disambiguation, we first extract a set of real-valued features for each candidate entity e given a training set of mentions M .Features values are standardised to have zero mean and unit variance.Parameters of the training distribution are saved for consistent standardisation of test data.
We train a Support Vector Machine (SVM) classifier to perform pairwise ranking (Joachims, 2002).For each mention in the training set, we derive training instances by comparing the feature vector of the gold link ( f g ) with each non-gold candidate ( f c ): We create instances for the top-ten non-gold candidates by sum of absolute feature values: In development experiments, this outperformed random selection and difference in activation.Class assignment is alternated to balance the training set.
To capture non-linear feature relationships we incorporate a degree-2 polynomial kernel via explicit feature mapping (Chang et al., 2010).Regularisation parameters are selected via grid search over the development set.Our final model utilises an L1 loss function, L2 weight penalty and C ≈ 0.03.

Feature selection
Sections 4 and 5 describe a total of ten model components, six from Wikipedia and four from Wikilinks.We select the optimal combination through exhaustive search.Figure 1 includes individual and cumulative results on the CoNLL development data.The article, mention and web link models each attain their best performance with all component features (entity, name, BOW, and DBOW): 84.7, 81.1, and 75.0 respectively.Adding mention context features doesn't improve the more conventional Wikipedia article model.Combining all features gives 87.7, while the optimal configuration achieves 88.1 without Wikipedia mention contexts.In the remaining experiments, optimal refers to Wikipedia article plus web link features and Wikipedia refers to article features alone.

Effect of training data size
Figure 2 compares learning curves for each model on CoNLL development data.The x-axis corresponds to p@1 scores and the y-axis corresponds to the number of (randomly selected) mentions used in training.All models stabilise early, suggesting 6,000 annotated mentions are sufficient for the SVM to learn feature weights.Possibly due to higher quality and consistency of features, the Wikipedia model stabilises earlier, before 1,000 annotated mentions.

Ablation analysis
Figure 3 contains an ablation analysis for Wikipedia and Web link features, as well as the optimal overall combination of both.The most striking effect is due to the popularity components.Removing entity prior features reduces p@1 by 3.2 for Wikipedia and 5.0 for Web link.Removing name probability reduces p@1 by 6.5 for Wikipedia and 1.8 for Web link.In the overall model, the Wikipedia popularity components have a much larger impact (prior: -3.2, name: -4.2) than the Web link popularity components (prior: -0.4,name: -0.8).These results show the impact of noisy web links, which appears to be worse for name probability modelling.For context, removing DBOW features have a larger impact than BOW for both Wikipedia (BOW: -0.2, DBOW: -1.3) and Web link (BOW: -0.9, DBOW: -1.4).All individual context features have a small impact on the overall model despite redundancy.

Adding coherence
The model combinations above provide a strong, scalable baseline based on popularity and entity context.Another approach to context leverages the Wikipedia link graph to explicitly model the coherence among possible resolutions.Here, systems define some measure of entity-entity relatedness and maximise the coherence of entity assignments across the query document as a whole.This can be done using global methods over the entity link graph (Hoffart et al., 2011), but these have high runtime complexity.We employ a simple approach based on conditional probabilities: where I e is the set of documents that link to entity e.The candidate-level feature is the average: where C is the set of context entities for candidate entity e.For Wikipedia and Web link coherence, I e models are derived respectively from the set of other articles that link to e and from the set of web pages that link to e.Given the same initial ranking from the optimal base model, Wikipedia and Web link coherence models alone achieve 84.7 and 76.6.

A two-stage classifier
To incorporate coherence, we use a two-stage classifier.First, we obtain an initial candidate ranking for each mention using the basic model described

Final experiments
We report final experiments on the held-out CoNLL and TAC 2010 test sets.As described in Section 3 above, we report p@1 for CoNLL following Hoffart et al. (2011) and A KB for TAC following He et al. (2013).We use a reference implementation to compute evaluation measures and pairwise significance (Hachey et al., 2014).We bold the superior configuration for each column only if the difference is significant (p < 0.05).

Results
Can link components replace KB components?Can links replace a curated KB? Nevertheless, the large improvements on CoNLL provide good evidence for complementarity and recommend using both feature sets when available.
The state of the art Finally, Table 10

Discussion
We set out to determine whether links from external resources can replace a clean, curated KB.Wikipedia is an incredible resource that has advanced our understanding of and capabilities for identifying and resolving entity mentions.However, it covers only a small fraction of all entities.Applications that require other entities must therefore extend Wikipedia or use alternative KBs.We explore a setting where a custom KB is required, but it is possible to harvest external documents with links into the custom KB.Overall, results are promising for using links in a knowledge-poor setting.The link-derived system performs nearly as well as the rich-KB system on both of our held-out data sets.Web link combinations perform at 97% of Wikipedia combinations on average.However, creating a KB as rich as Wikipedia represents an estimated 100 million hours of human effort (Shirky, 2010).We do not have a comparable estimate for the Web link data.However, it is created as byproduct of publishing activities and the labour pool is external.Considering this and the additional noise in web data, it is remarkable that the Web link models do so well with respect to the Wikipedia models.
We also present detailed experiments comparing popularity, context, and coherence components across settings.Here, results are even more surprising.As expected, Web link popularity and coherence models trail Wikipedia models.However, Web link context models outperform Wikipedia context models by 7 to 10 points.
We add the Web link components into the Wikipedia system to achieve, to our knowledge, the best published result of 88.7 on the CoNLL data set.Fur-thermore, results suggest that coherence modelling does not require complex global graph algorithms.Our simple approach improves performance over the basic model by one to three points.On the other hand, our basic system without coherence modelling approaches state-of-the-art performance on its own.This suggests that additional popularity and context features from web links can replace coherence where efficiency is a concern.
We believe these results have a number of implications for management of entity KBs.First, they motivate concerted efforts to link content to KBs since links lead to substantial accuracy improvements over a conventional model based on rich KB data alone.Second, it informs allocation of editorial resources between interlinking data sets and curating KBs.Since models built from link data alone approach state-of-the-art performance, curating links is a reasonable alternative to curating a KB.This is especially true if link curation is cheaper or if links can be created as a byproduct of other content authorship and management activities.
Finally, where KB data is currently proprietary, results here motivate openly publishing KB entities and encouraging their use as a disambiguation endpoint for public content.In addition to providing pathways to paid content, incoming links provide a simple means to harvest rich metadata from external content and this can be used to build high-quality resolution systems.
A key avenue for future work is to evaluate how well our approach generalises to other web KBs.For instance, incorporating links to sites like Freebase or IMDb which complement or extend Wikipedia's entity coverage.

Conclusion
Despite widespread use in entity linking, Wikipedia is clearly not the only source of entity information available on the web.We demonstrate the potential for web links to both complement and completely replace Wikipedia derived data in entity linking.This suggests that, given sufficient incoming links, any knowledge base may be used for entity linking.We argue that this motivates open publishing of enterprise KBs.Code is available under an MIT license at https://github.com/wikilinks/nel.
(i)  and cumulative (c) results for basic features on the CoNLL development data.Combined includes all features while Optimal includes the best subset.Optimal tracks Combined closely, but is just higher.

Figure 2 :
Figure 2: SVM learning curves for best configurations.

Table 1 :
Data sets for disambiguation tasks addressed here.Statistics are described in Section 3.
Hoffart et al.'s Yago2 means relations for candidate generation.These alternative names are harvested from Wikipedia disambiguation pages, redirects and inter-article links.In the Hoffart et al. setting, candidate recall is 100%.
He et al.'s setup to control for differences and for comparability to He et al.'s results.

Table 3 :
Comparison of page-entity link graphs from Wikipedia and Wikilinks (in millions).These graphs are the basis for entity prior features (Sections 4.1, 5.1).

Table 4 :
Comparison of name-entity link graphs from Wikipedia and Wikilinks (in millions).These graphs are the basis for name probability features (Sections 4.2, 5.2).

Table 5 :
Coverage of textual context models for each source over entities (E) and mentions (M).

Table 6 :
Mean in-vocab tokens per entity ( tE ) and tokens per mention ( tM ) for each textual context model.andfourthcolumnscorrespond to coverage of entities (Cov E ) and mentions (Cov M ) from the CoNLL data set.Mention coverage exceeds entity coverage, highlighting the relationship with prevalence in newswire.The last column contains p@1 for the subset of mentions in CoNLL for which the correct resolution is covered by both articles and web links.This isolates context source, demonstrating that link contexts outperform article text.Table6compares context size in Wikilinks to Wikipedia.Wikilinks BOW models are approximately twice the size of Wikipedia article models and half the size of Wikipedia mention models.This helps to explain why individual mention and link models outperform individual article models.

Table 7 :
Web link components vs. Wikipedia.

Table 8 :
Web link combinations vs. Wikipedia.

Table 9 :
Web links complement Wikipedia.
Table8compares performance of the Wikipedia and Web link systems using the basic feature set alone and with coherence.Wikipedia models generally perform better.However, the Web link configurations perform at 93.1, 95.1, 99.9, and 100% of the Wikipedia linker -97% on average.This suggests that a link data set can replace a curated KB, with only a small impact on accuracy.Results also show that adding coherence improves performance in all cases.

Table 10 :
Comparison to the disambiguation literature.