A Graph-based Lattice Dependency Parser for Joint Morphological Segmentation and Syntactic Analysis

Space-delimited words in Turkish and Hebrew text can be further segmented into meaningful units, but syntactic and semantic context is necessary to predict segmentation. At the same time, predicting correct syntactic structures relies on correct segmentation. We present a graph-based lattice dependency parser that operates on morphological lattices to represent different segmentations and morphological analyses for a given input sentence. The lattice parser predicts a dependency tree over a path in the lattice and thus solves the joint task of segmentation, morphological analysis, and syntactic parsing. We conduct experiments on the Turkish and the Hebrew treebank and show that the joint model outperforms three state-of-the-art pipeline systems on both data sets. Our work corroborates findings from constituency lattice parsing for Hebrew and presents the first results for full lattice parsing on Turkish.


Introduction
Linguistic theory has provided examples from many different languages in which grammatical information is expressed via case marking, morphological agreement, or clitics. In these languages, configurational information is less important than in English since the words are overtly marked for their syntactic relations to each other. Such morphologically rich languages pose many new challenges to today's natural language processing technology, which has often been developed for English.
One of the first challenges is the question on how to represent morphologically rich languages and what are the basic units of analysis (Tsarfaty et al., 2010). The Turkish treebank (Oflazer et al., 2003), for example, represents words as sequences of inflectional groups, semantically coherent groups of morphemes separated by derivational boundaries. The treebank for Modern Hebrew (Sima'an et al., 2001) chooses morphemes as the basic unit of representation. A space-delimited word in the treebank can consist of several morphemes that may belong to independent syntactic contexts.
Both Turkish and Hebrew show high amounts of ambiguity when it comes to the correct segmentation of words into inflectional groups and morphemes, respectively. Within a sentence, however, these ambiguities can often be resolved by the syntactic and semantic context in which these words appear.
A standard (dependency) parsing system decides segmentation, morphological analysis (including POS), and syntax one after the other in a pipeline setup. While pipelines are fast and efficient, they cannot model interaction between these different levels of analysis, however. It has therefore been argued that joint modeling of these three tasks is more suitable to the problem (Tsarfaty, 2006). In previous research, several transition-based parsers have been proposed to model POS/morphological tagging and parsing jointly (Hatori et al., 2011;Bohnet and Nivre, 2012;Bohnet et al., 2013). Such parsing systems have been further extended to also solve the segmentation problem in Chinese (Hatori et al., 2012;Li and Zhou, 2012;Zhang et al., 2014). Transition-based parsers are attractive since they do not rely on global optimization and thus deal well with the increased model complexity that comes with joint modeling. Nonetheless, graphbased models have been proposed as well, e.g. by Li et al. (2011) for joint POS tagging and dependency parsing. Their parsers model the joint problem directly at the cost of increased model complexity.
In this paper, we present a graph-based dependency parser for lattice parsing that handles the increased complexity by applying dual decomposition. The parser operates on morphological lattices and predicts word segmentation, morphological analysis, and dependency syntax jointly. It decomposes the problem into several subproblems and uses dual decomposition to find a common solution (Koo et al., 2010;Martins et al., 2010). The subproblems are defined such that they can be solved efficiently and agreement is found in an iterative fashion. Decomposing the problem thus keeps the complexity of the joint parser on a tractable level.
We test the parser on the Turkish and the Hebrew treebank. The segmentation problem in these languages can be tackled with the same approach even though their underlying linguistic motivation is quite different. In our experiments, the lattice dependency parser outperforms three state-of-the-art pipeline systems. Lattice parsing for Hebrew has been thoroughly investigated in constituency parsing (Cohen and Smith, 2007;Goldberg and Tsarfaty, 2008;Goldberg and Elhadad, 2013), demonstrating the viability of joint modeling. To the best of our knowledge, our work is the first to apply full lattice parsing to the Turkish treebank.
We introduce the segmentation problem in Turkish and Hebrew in Section 2 and present the lattice parser in Section 3. Sections 4 and 5 describe the experiments and their results and we discuss related work in Section 6. We conclude with Section 7.

Word Segmentation in Turkish and Hebrew
A lot of morphosyntactic information is overtly marked on words in morphologically rich languages. It is also common to express syntactic information through derivation or composition. As a consequence these words, orthographically written together, actually have word-internal syntactic struc-tures. Moreover, word-external relations may depend on the word-internal structures, e.g., a word could be grammatically related to only parts of another word instead of the whole. For instance, in the Turkish sentence ekmek aldım, each word has two analyses. ekmek means 'bread' or the nominal 'planting' which is derived from the verb stem ek 'plant' with the nominalization suffix mek. aldım has the meaning 'I bought' which decomposes as al-dı-m 'buy-Past-1sg'. It also means 'I was red', which is derived from the adjective al 'red', inflected for past tense, 1st person singular.
Depending on the selected morphological analysis for each word, syntax and semantics of the sentence change. When the first analysis is selected for both words, the syntactic representation of the sentence is given in Figure 1, which corresponds to the meaning 'I bought bread'. When the nominal 'planting' is selected for the first word, it is a grammatical sentence albeit with an implausable meaning. When the derivational analysis of the second word is selected, regardless of the morphological analysis of the first word, the sentence is ungrammatical due to subject-verb agreement failure. Although all morphological analyses for these two words are correct in isolation, when they occur in the same syntactic context only some combinations are grammatical.
ekmek aldım Noun+Nom Verb+Past+1sg OBJ I bought bread. This small example demonstrates that the syntactic structure depends on the morphological disambiguation of the words. At the same time, it shows that syntax can help pick the right morphological analysis. For a joint system to decide the morphological and syntactic representation together, all possible analyses must be available to the system. The possible morphological analyses of a word can be efficiently represented in a lattice structure. The lattice representation of the sentence in Figure 1 is given in Figure 2, with double circles denoting word bound-360 aries. A sentence lattice is the concatenation of its word lattices. A morphological analysis of a word is a full path from the initial state to the final state of its lattice. Labels on the transitions are the surface form and underlying morphological representation of segments. 1 Lattices also capture well the segmentation of words in Hebrew. Different from Turkish, Hebrew segments can be syntactic units like determiners, prepositions, or relativizers attached to stem segments. In an example given by Goldberg and Tsarfaty (2008), the word hneim 'the pleasant/madepleasant' has three analyses corresponding to the lattice in Figure 3.  Both the Hebrew and the Turkish treebank annotate dependencies between units smaller than words. In the Turkish treebank, a space-delimited word is segmented into one or more segments depending on its morphological representation. The number of segments is determined by the number of derivations. If it was derived n times, it is represented as n+1 segments. The derivational boundaries are part of the morphological representation. In the Turkish dependency parsing literature (Eryigit et al., 2008;Ç etinoglu and Kuhn, 2013) these segments 1 Surface forms on the transitions are given for convenience. In the Turkish treebank, only final segments have surface forms (of full words), the surface forms of non-final segments are represented as underscores. are called inflectional groups (IGs). IGs consist of one or more inflectional morphemes. The head of a non-final IG is the IG to its right with a dependency relation DERIV. The head of a final IG could be any IG of another word.
The Hebrew treebank defines relations between morphemes (Sima'an et al., 2001). Those morphemes correspond to what is usually considered a separate syntactic unit in English. In Hebrew script, word classes like prepositions and conjunctions are always written together with the following word. Contrary to Turkish, syntactic heads of both nonfinal and final segments can be internal or external to the same space-delimited word.
For convenience, we will use token to refer to the smallest unit of processing for the remainder of the paper. It corresponds to IGs in Turkish and morphemes in Hebrew. A transition in a morphological lattice therefore represents one token. We will use word to refer to space-delimited words. 2 In standard parsing, these two terms usually coincide with a token in a sentence being separated from the surrounding ones by space.

Lattice Parsing
One can think of lattice parsing as two tasks that the parser solves simultaneously: the parser needs to find a path through the lattice and it needs to find a parse tree. Importantly, the parser solves this task under the condition that the parse tree and the path agree with each other, i.e. the tokens that the parse tree spans over must form the path through the lattice. Decomposing the problem in this way defines the three components for the parser.
Let x be an input lattice and T = {ROOT, t 1 , t 2 , . . . , t n } be the set of tokens in x. In what follows, we assume two different structures, lattices and dependency trees. Dependency trees are represented as directed acyclic trees with a special root node (ROOT), whereas lattices are directed acyclic graphs with one defined start state and one defined end state (see Figures 2 and 3). For dependency trees, we will use the terms node and arc to refer to the vertices and the edges between the vertices, respectively. Tokens are represented as nodes in the dependency tree. For lattices, we use the terms state and transition to refer to the vertices and their edges in the lattice. Tokens are represented as transitions between states in the lattice.
Find The Path. A token bigram in a lattice x is a pair of two transitions t, t , such that the target state of t in x coincides with the source state of t in x. A chain of overlapping bigrams that starts from the initial state and ends in the final state forms a path through the lattice. We represent the ROOT token as the first transition, i.e. a single transition that leaves the initial state of the lattice.
Given a lattice x, we define the index set of token bigrams in the lattice to be S : to be the set of bigrams that have t at the second position. A consecutive path through the lattice is defined as an indicator vector p := p s s∈S where p s = 1 means that bigram s is part of the path, otherwise p s = 0. We define P as the set of all wellformed paths, i.e. all paths that lead from the initial to the final state.
We use a linear model that factors over bigrams. Given a scoring function f P that assigns scores to paths, the path with the highest score can be found byp where φ SEG is the feature extraction function for token bigrams. The highest-scoring path through the lattice can be found with the Viterbi algorithm. We use this bigram model later also as a standalone disambiguator for morphological lattices to find the highest-scoring path in a lattice.
Find The Tree. We define the index set of arcs in a dependency tree as A : being a set of dependency relations. A dependency tree is defined as an indicator vector y := y a a∈A where y a = 1 means that arc a is in the parse, otherwise y a = 0. We define Y to be the set of all well-formed dependency trees.
We follow Koo et al. (2010) and assume an arcfactored model (McDonald et al., 2005) to find the highest-scoring parse. Given a scoring function f T that assigns scores to parses, the problem of finding the highest scoring parse is defined aŝ where φ ARC is the feature extraction function for single arcs and w is the weight vector. We use the Chu-Liu-Edmonds algorithm (CLE) to find the highestscoring parse (Chu and Liu, 1965;Edmonds, 1967). Note that the algorithm includes all tokens of the lattice into the spanning tree, not just some tokens on some path. Chu-Liu-Edmonds furthermore enforces the tree properties of the output, i.e. acyclicity and exactly one head per token.
Agreement Constraints. To make the path and the parse tree agree with each other, we introduce an additional dependency relation NOREL into L. We define a token that is attached to ROOT with relation NOREL to be not on the path through the lattice. These arcs are not scored by the statistical model, they simply serve as a means for CLE to mark tokens as not being part of the path by attaching them to ROOT with this relation. The parser can predict the NOREL label only on arcs attached to root.
We introduce two agreement constraints to ensure that (i) all tokens not on the path are marked with NOREL and must be attached to ROOT and (ii) tokens cannot be dependents of tokens marked with NOREL.
The first constraint is implemented as an XOR (⊕) factor (Martins et al., 2011b) over token bigrams and arcs. It states that for a token t, either one of its bigrams 3 or its NOREL-arc must be active. There is one such constraint for each token in the lattice.
The second constraint ensures that a token that is part of the path will not be attached to a token that is not. It thus guarantees the coherence of the dependency tree over the path through the lattice. It is implemented as an implication (=⇒) factor (Martins et al., 2015). It states that an active NOREL arc for a token h implies an inactive arc for all arcs having h as head. There is one such constraint for each possible arc in the parse.
Deciding on a path through the lattice partitions the tokens into two groups: the ones on the path and the ones that are not. By means of the NOREL label, the CLE is also able to partition the tokens into two groups: the ROOT-NOREL tokens and the proper dependency tree tokens. The two agreement constraints then make sure that the two partionings agree with each other. The first constraint explicitly links the two partitionings by requiring each token to either belong to the path or to the ROOT-NOREL tokens. The second constraint ensures that the partitioning by the CLE is consistent, i.e. tokens attached to ROOT with NOREL cannot mix with the other tokens in the tree structure. Before the parser outputs the parse the tokens that do not belong to the path/tree are discarded.
The objective function of the lattice parser is subject to the two agreement constraints in Equations (1) and (2). We use Alternating Directions Dual Decomposition or AD 3 (Martins et al., 2011a) 4 to find the optimal solution to this constrained optimization problem. CLE can be implemented such that its worst case complexity is O(T 2 ), while the Viterbi algorithm needed to find the path is of worst case complexity O(QT 2 ), where Q is the number of states in the lattice. Instead of combining these two problems directly, which would multiply their complexity, AD 3 combines them additively, such that the complexity of the parser is O(k(T 2 + QT 2 )) with k being the number of iterations that AD 3 is run.
Second-order Parsing. To facilitate second-order features, we use grandparent-sibling head automata as proposed in Koo et al. (2010), which we extend to include dependency relations. The head automata allow the parser to model consecutive sibling and grandparent relations. The architecture of the parser does not need to be changed at all to include the second-order factors. The head automata are simply another component. They compute solutions over the same set of arc indicator variables as the CLE and AD 3 thus ensures that the output of the two algorithms agrees on the tree structure (Koo et al., 2010). The second-order factors dominate the complexity of the entire parser, since solving the head automata is of complexity O(T 4 L).
Pruning. We use rule-based and heuristics-based pruning to reduce the search space of the parser. Arcs between tokens that lie on competing paths through the lattice are cut away as these tokens can never be in a syntactic relation. For the Turkish treebank, we introduce an additional rule based on the annotation scheme of the treebank. In the treebank, the IGs of a word form a chain with each IG having their head immediately to the right and only the last IG choosing the head freely. For the non-final IGs, we therefore restrict the head choice to all IGs that can immediately follow it in the lattice.
In order to restrict the number of heads, we train a simple pairwise classifier that predicts the 10 best heads for each token. It uses the first-order features of the parser's feature model.
Feature Model. The parser extracts features for bigrams (path), arcs (first-order), consecutive siblings, and grandparent relations (both second order). It uses standard features like word form, lemma, POS, morphological features, head direction, and combinations thereof.
Context features are more difficult in lattice parsing than in standard parsing as the left and right context of a token is not specified before parsing. We first extracted context features from all tokens that can follow or precede a token in the lattice. This led to overfitting effects as the model was learning specific lattice patterns. We therefore use latent left and right context and extract features from only one of the left/right neighbor tokens. The latent context is the left/right context token with the highest score from the path features (raw bigram scores, they are not changed by AD 3 ). The parser extracts context from one token in each direction.
Distance features are also more difficult in lattices since the linear distance between two tokens depends on the actual path chosen by the parser. We define distance simply as the length of the shortest path between two tokens in the lattice, but this distance may not coincide with the actual path.
Context features and distance features show that lattice dependency parsing poses interesting new challenges to feature design. Using latent context features is one way of handling uncertain context, compare also the delayed features in Hatori et al. (2011). A thorough investigation of different options is needed here.
Learning. We train a discriminative linear model using passive-aggressive online learning (Crammer et al., 2003) with cost-augmented inference (Taskar et al., 2005) and parameter averaging (Freund and Schapire, 1999). We use Hamming loss over the arcs of the parse tree excluding NOREL arcs. The model trains one parameter vector that includes features from the tree and from the path.
The maximum number of iterations of AD 3 is set to 1000 during training and testing. The algorithm sometimes outputs fractional solutions. During training, the model is updated with these fractional solutions, weighting the features and the loss accordingly. During testing, fractional solutions are projected to an integer solution by first running the best-path algorithm with the path posteriors output by AD 3 and afterwards running CLE on the selected path weighted by the arc posteriors (Martins et al., 2009). In the experiments, fractional solutions occur in about 9% of the sentences in the Turkish development set during testing.

The Turkish Data
The training set for Turkish is the 5,635 sentences of the METU-Sabancı Turkish Treebank (Oflazer et al., 2003). The 300 sentences of the ITU validation set (Eryigit, 2012) are used for testing. As there is no separate development set, we split the training set into 10 parts and used 2 of them as development data. All models run on this development set are trained on the remaining 8 parts. We also report results from 10-fold crossvalidation on the full training set (10cv).
We use the detached version of the Turkish treebank (Eryigit et al., 2011) where multiword expressions are represented as separate tokens. The training set of this version contains 49 sentences with loops. We manually corrected these sentences and use the corrected version in our experiments. 5 The Turkish raw input is first passed through a morphological analyzer (Oflazer, 1994) in order to create morphological lattices as input to the parser. Gold analyses are added to the training lattices if the morphological analyzer failed to output the correct analyses.
For the pipeline systems, the input lattices are disambiguated by running a morphological disambiguator. We train our own disambiguator using the bigram model from the parser and find the best path through the lattice with the Viterbi algorithm. The disambiguator uses the same bigram features as the lattice parser. The morphological disambiguator is trained on the Turkish treebank as in Ç etinoglu (2014).

The Hebrew Data
The data for Hebrew comes from the SPMRL Shared Task 2014 (Seddah et al., 2014), which is based on the treebank for Modern Hebrew (Sima'an et al., 2001). It provides lattices and predisambiguated input files. The training and development lattices contained a number of circular structures due to self-loops in some states. We automatically removed the transitions causing these cycles.
Input lattices for training were prepared as for Turkish by adding the gold standard paths if necessary. Compared to the Turkish data, the Hebrew lattices are so large that training times for the lattice parser became unacceptable. We therefore used our morphological disambiguator to predict the 10 best paths for each lattice. All transitions in the lattice that were not part of one of these 10 paths were discarded. Note that the number of actual paths in these pruned lattices is much higher than 10, since the paths converge after each word. All experiments with the joint model for Hebrew are conducted on the pruned lattices. As for Turkish we preprocess the input lattices for all baselines with our own morphological disambiguator.

Baselines
We compare the lattice parser (JOINT for Turkish, JOINT10 for Hebrew) to three baselines: MATE, TURBO, and PIPELINE.
The first two baseline systems are off-the-shelf dependency parsers that currently represent the state-of-the-art.
Mate parser 6 (Bohnet, 2009;Bohnet, 2010) is a graph-based dependency parser that uses Carreras' decoder (Carreras, 2007) and approximate search (McDonald and Pereira, 2006) to produce non-projective dependency structures. Tur-boParser 7 ) is a graph-based parser that uses a dual decomposition approach and outputs non-projective structures natively. The third baseline system runs the lattice parser on a predisambiguated lattice, i.e. in a pipeline setup.
All three baselines are pipeline setups and use the same disambiguator to predict a path through the lattice. The bigram features in the disambiguator are the same as in the joint model. There is thus no difference between the lattice parser and the baselines with respect to the features that are available during segmentation. As opposed to lattice parsing, baseline systems are trained on the gold standard segmentation (and thus gold morphological analyses) in the training data, since automatically predicted paths would not guarantee to be compatible with the gold dependency structures.
The purpose of the first two baselines is to compare the joint parser to the current state-of-the-art. However, the feature sets are different between the joint parser and the off-the-shelf baselines. A difference in performance between the joint parser and the first two baseline systems may thus simply be caused by a difference in the feature set. The third baseline eliminates this difference in the feature sets since it is the actual lattice parser that is run on a disambiguated lattice. Because the morphological disambiguator for the PIPELINE baseline is using the same feature set as the lattice parser (the bigram model), the fact that the joint parser is trained and tested on full lattices is the only difference between these two systems. The PIPELINE baseline thus allows us to test directly the effect of joint decoding compared to a pipeline setup.

Evaluation
Standard labeled and unlabeled attachment scores are not applicable when parsing with uncertain segmentation since the number of tokens in the output of the parser may not coincide with the number of tokens in the gold standard. Previous work therefore suggests alternative methods for evaluation, e.g. by means of precision, recall, and f-score over tokens, see e.g. Tsarfaty (2006) or Cohen and Smith (2007).
The uncertainty of segmentation furthermore makes it very hard to evaluate the other levels of analysis independently of the segmentation. In order to decide whether the morphological analysis of a token (or its syntactic attachment) is correct, one always needs to find out first to which token in the gold standard it corresponds. By establishing this correspondence, the segmentation is already being evaluated. Evaluating syntax isolated from the other levels of analysis is therefore not possible in general. Hatori et al. (2012) count a dependency relation correct only when both the head and the dependent have the correct morphological analysis (here POS) and segmentation. Goldberg (2011, page 53) proposes a similar approach, but only requires surface forms to match between gold standard and prediction. These metrics compute precision and recall over tokens. Eryigit et al. (2008) and Eryigit (2012) define an accuracy (IGeval) for Turkish parsing by taking advantage of the annotation scheme in the Turkish treebank: A non-final IG in the Turkish treebank always has its head immediately to the right, always with the same label, which makes it possible to ignore the inner dependency relations, i.e. the segmentation, of a dependent word. The metric therefore only needs to check for each word whether the head of the last IG is attached to the correct IG in another word. The metric includes a back-off strategy in case the head word's segmentation is wrong. A dependency arc is then counted as correct if it attaches to an IG in the correct word and the POS tag of the head IG is the same as in the gold standard.
Parsing Evaluation. We follow Hatori et al. (2012) and use a strict definition of precision and recall (PREC, REC, F1) over tokens to evaluate the full task. We first align the tokens of a word in the parser output with the tokens of the corresponding word in the gold standard using the Needleman-Wunsch algorithm (Needleman and Wunsch, 1970), which we modify so it does not allow for mismatches. A token in the parser output that is not in the gold standard is thus paired with a gap and vice versa. Two tokens must have the same morphological analysis in order to match. 8 A true positive is defined as a pair of matching tokens whose heads are also aligned and match. For labeled scores, the dependency relations must match as well. Precision is defined as the number of true positives over the number of tokens in the prediction, recall is defined as the number of true positives over the number of tokens in the gold standard. F-score is the harmonic mean of precision and recall.
This metric is very strict and requires all levels of analysis to be correct. In order to evaluate the syntax as independently as possible, we furthermore report IGeval for Turkish, with and without the aforementioned backoff strategy (IGeval and IGeval STRICT). For Hebrew, we report on a version of precision and recall as defined above that only requires the surface forms of the tokens to match. 9 This metric is almost the one proposed in Goldberg (2011). All reported evaluation metrics ignore punctuation.
We do not use TedEval as defined in Tsarfaty et al. (2012) even though it has been used previously to evaluate dependency parsing with uncertain segmentation (Seddah et al., 2013;Zhang et al., 2015). The reason is that it is not an inherently dependency-based framework and the conversion from constituency structures to dependency structures interferes with the metric. 10 The metric 8 The method does not create cross, many-to-one, or one-tomany alignments, which can be important because in very rare cases the same token occurs twice in one word. 9 The metric would not work for Turkish, as the surface forms of non-final IGs are all represented as underscores. 10 As an experiment, we took a Turkish treebank tree and created artificial parses by attaching one token to a different head each time. All other tokens remained attached to their correct head, and segmentation is kept gold. This gave us 11 parses that contained exactly one attachment error and one parse identical with the gold standard. Running TedEval on each of the proposed in Goldberg (2011) implements the same ideas without edit distance and is defined directly for dependencies. Segmentation Evaluation. We use the same token-based precision and recall to measure the quality of segmentation and morphological analysis without syntax. For a token to be correct, it has to have the same morphological analysis as the token in the gold standard to which it is aligned. We furthermore report word accuracy (ACC w ), which is the percentage of words that received the correct segmentation.

Results
Segmentation and Morphology. Table 1 shows the quality of segmentation and morphological analysis. The baseline for Turkish is the Turkish morphological disambiguator by Sak et al. (2008), trained on the Turkish treebank. For Hebrew, the baseline is the disambiguated lattices provided by the SPMRL 2014 Shared Task. 11 The bigram model is our own morphological disambiguator. The joint model is the full lattice parser, which has access to syntactic information.
The results show that the bigram model is clearly outperforming the baselines for both languages. The feature model of the bigram model was developed on the Turkish development set, but the model also works well for Hebrew. Comparing the bigram model to the joint model shows that overall, the joint model performs better than the bigram model. However, the joint model mainly scores in recall rather than in precision, the bigram model is even ahead of the joint model in precision for Hebrew. The joint model outperforms the bigram model and the baseline also in word accuracy. The results demonstrate that syntactic information is relevant to resolve ambiguity in segmentation and morphology for Turkish and Hebrew.  Table 2: Parsing results for Turkish. Statistically significant differences between the joint system and the pipeline system are marked with † (p < 0.01) and * (p < 0.05). Significance testing was performed using the Wilcoxon Signed Rank Test (not for F1).
Turkish. Table 2 presents the results of the evaluation of the three baseline systems and the lattice parser on the Turkish data. The PIPELINE and the JOINT system give better results than the other two baselines across the board. This shows that the feature set of the lattice parser is better suited to the Turkish treebank than the feature set of Mate parser and Turbo parser. It is not a surprising result though, since the lattice parser was developed for Turkish whereas the other two parsers were developed for other treebanks.
The JOINT system outperforms the PIPELINE system with respect to the first three metrics. These metrics evaluate syntax, segmentation, and morphological analysis jointly. Higher scores here mean that these aspects in combination have become better. The differences between the PIPELINE and the JOINT model are consistently statistically significant with respect to recall, but only in some cases with re-spect to precision. The syntactic information that is available to the joint model thus seems to improve recall rather than precision.
The last two columns in Table 2 show an evaluation using IGeval. The IGeval metric is designed to evaluate the syntactic quality with less attention to morphological analysis and segmentation. Here, both PIPELINE and JOINT achieve very similar results and none of the differences is statistical significant. These results suggest that a good part of the improvements in the lattice parser occurs in the morphological analysis/segmentation, whereas the quality of syntactic annotation basically stays the same between the pipeline and the joint model.
Hebrew. The experimental results on the Hebrew data are shown in Table 3. The three baselines perform very similarly. All three baseline systems are run on the output of the same disambiguator, which means that the feature models of the parsers seem to be equally well suited to the Hebrew treebank. The feature model of the lattice parser that is used in the PIPELINE baseline was not adapted to Hebrew in any way, but was used as it was developed for the Turkish data.
Compared to the three baselines, the joint model outperforms them for both labeled and unlabeled scores. As the only difference between PIPELINE and JOINT is the fact that the latter performs joint decoding, the results support the findings in constituency parsing by Tsarfaty (2006), Cohen and Smith (2007), and Goldberg and Tsarfaty (2008), namely that joint decoding is a better model for Hebrew parsing. Judging from statistical significance, the JOINT model improves recall rather than precision, a picture that we found for Turkish as well.  Table 3: Statistically significant differences between the joint system and the pipeline system are marked with † (p < 0.01) and * (p < 0.05). Significance testing was performed using the Wilcoxon Signed Rank Test (not for F1).
As described in Section 4.4, we cannot evaluate the syntax entirely independently on Hebrew, but we can eliminate the morphological level. Table 4 shows the results of the evaluation when only syntax and surface forms are matched. The overall picture compared to the evaluation shown in Table 3 does not change, however. Also when disregarding the quality of morphology, the JOINT model outperforms the PIPELINE, notably with respect to recall.

Related Work
Graph-based Parsing. Our basic architecture resembles the joint constituency parsing and POS tagging model by , but our model  Table 4: Parsing results for Hebrew, evaluated without morphology. Statistically significant differences between the joint system and the pipeline system are marked with †. Significance testing was performed using the Wilcoxon Signed Rank Test with p < 0.01 (not for F1).
needs additional constraints to enforce agreement between the two tasks. Martins et al. (2011a) and Martins et al. (2015) show how such first-order logic constraints can be represented as subproblems in dual decomposition. Similar approaches, where such constraints are used to ensure certain properties in the output structures, have been used e.g. in semantic parsing (Das et al., 2012), compressive summarization , and joint quotation attribution and coreference resolution (Almeida et al., 2014). Parsers that use dual decomposition are proposed in Koo et al. (2010) and Martins et al. (2010). From Koo et al. (2010), we adopted the idea of using the Chu-Liu-Edmonds algorithm to ensure tree properties in the output as well as second-order parsing with head automata. Li et al. (2011) extend several higher-order variants of the Eisner decoder (Eisner, 1997) such that POS tags are predicted jointly with syntax. The complexity of their joint models increases by polynomials of the tag set size. Due to the dual decomposition approach, the complexity of our parser stays equal to the complexity of the most complex subproblem, which is the second-order head automata in our case.
Transition-based Parsing. Joint models in transition-based parsing usually introduce a variant of the shift transition that performs the additional task, e.g. it additionally predicts the POS tag and possibly morphological features of a token that is being shifted (Hatori et al., 2011;Bohnet and Nivre, 2012;Bohnet et al., 2013). Optimization over the joint model is achieved by beam search. To also solve the word segmentation task, several models for Chinese were proposed that parse on the level of single characters, forming words from characters with a special append transition (Hatori et al., 2012;Li and Zhou, 2012) or predicting word internal structure along with syntax (Zhang et al., 2014). To use such a transition-based system for the segmentation task in Turkish or Hebrew, the shift transition would have to be changed to do the opposite of the append transition in the Chinese parsers: segment an incoming token into several ones, for example based on the output of a morphological analyzer.
Easy-first Parsing. Ma et al. (2012) introduce a variant of the easy-first parser (Goldberg and Elhadad, 2010a) that uses an additional operation to POS tag input tokens. The operations are ordered such that the parser can only introduce a dependency arc between two tokens that have received a POS tag already. Tratz (2013) presents a similar system for Arabic that defines several more operations to deal with segmentation ambiguity.
Sampling-based Parsing. Zhang et al. (2015) present a joint model that relies on sampling and greedy hill-climbing for decoding, but allows for arbitrarily complex scoring functions thus opening access to global and cross-level features. Such features could be simulated in our model by adding additional factors in the form of soft constraints (constraints with output, see Martins et al. (2015)), but this would introduce a considerable number of additional factors with a notable impact on performance.
Constituency Parsing. Joint models have also been investigated in constituency parsing, notably for Hebrew. Tsarfaty (2006) already discusses full joint models, but the first full parsers were presented in Cohen and Smith (2007), Goldberg and Tsarfaty (2008), and later Goldberg and Elhadad (2013). Green and Manning (2010) present a similar parser for Arabic. Among these, some authors emphasize the importance of including scores from the morphological model into the parsing model, whereas other models do not use them at all. In our parser, the model is trained jointly for both tasks without weighting the two tasks differently.
Parsing Hebrew and Turkish. Joint models for Hebrew parsing were mostly investigated for constituency parsing (see above). There has been some work specifically on Hebrew dependency parsing (Goldberg and Elhadad, 2009;Goldberg and Elhadad, 2010b;Goldberg, 2011), but not in the context of joint models.
Turkish dependency parsing was pioneered in Eryigit and Oflazer (2006) and Eryigit et al. (2008). They compare parsing based on inflectional groups to word-based parsing and conclude that the former is more suitable for Turkish. Ç etinoglu and Kuhn (2013) are first to discuss joint models for Turkish and present experiments for joint POS tagging and parsing, but use a pipeline to decide on segmentation and morphological features. To the best of our knowledge, there currently exists no work on full lattice parsing for Turkish.

Conclusion
Morphologically rich languages pose many challenges to standard dependency parsing systems, one of them being that the number of tokens in the output is not always known beforehand. Solving this problem in a pipeline setup leads to efficient systems but systematically excludes interaction between the lexical, morphological, and syntactic level of analysis.
In this work, we have presented a graph-based lattice dependency parser that operates on morphological lattices and simultaneously predicts a dependency tree and a path through the lattice. We tested the joint model on the Turkish treebank and the treebank of Modern Hebrew and demonstrated that the joint model outperforms three state-of-theart pipeline models. We presented the first results for full lattice parsing on the Turkish treebank. The results on the Hebrew treebank corroborate findings in constituency parsing (Cohen and Smith, 2007;Goldberg and Tsarfaty, 2008).