Easy-First Dependency Parsing with Hierarchical Tree LSTMs

We suggest a compositional vector representation of parse trees that relies on a recursive combination of recurrent-neural network encoders. To demonstrate its effectiveness, we use the representation as the backbone of a greedy, bottom-up dependency parser, achieving very strong accuracies for English and Chinese, without relying on external word embeddings. The parser’s implementation is available for download at the first author’s webpage.


Introduction
Dependency-based syntactic representations of sentences are central to many language processing tasks (Kübler et al., 2008).Dependency parse-trees encode not only the syntactic structure of a sentence but also many aspects of its semantics.
A recent trend in NLP is concerned with encoding sentences as a vectors ("sentence embeddings"), which can then be used for further prediction tasks.Recurrent neural networks (RNNs) (Elman, 1990), and in particular methods based on the LSTM architecture (Hochreiter and Schmidhuber, 1997), work very well for modeling sequences, and constantly obtain state-of-the-art results on both language-modeling and prediction tasks (see, e.g.(Mikolov et al., 2010)).
Several works attempt to extend recurrent neural networks to work on trees (see Section 8 for a brief overview), giving rise to the so-called recursive neural networks (Goller and Küchler, 1996;Socher et al., 2010).However, recursive neural networks do not cope well with trees with arbitrary branching factors -most work require the encoded trees to be binary-branching, or have a fixed maximum arity.Other attempts allow arbitrary branching factors, in the expense of ignoring the order of the modifiers.
In contrast, we propose a tree-encoding that naturally supports trees with arbitrary branching factors, making it particularly appealing for dependency trees.Our tree encoder uses recurrent neural networks as a building block: we model the left and right sequences of modifiers using RNNs, which are composed in a recursive manner to form a tree (Section 3).We use our tree representation for encoding the partially-build parse trees in a greedy, bottom-up dependency parser which is based on the easy-first transition-system of Goldberg and Elhadad (2010).
Using the Hierarchical Tree LSTM representation, and without using any external embeddings, our parser achieves parsing accuracies of 92.6 UAS and 90.2 LAS on the PTB (Stanford dependencies) and 86.1 UAS and 84.4 LAS on the Chinese treebank, while relying on greedy decoding.
To the best of our knowledge, this is the first work to demonstrate competitive parsing accuracies for full-scale parsing while relying solely on recursive, compositional tree representations, and without using a reranking framework.We discuss related work in Section 8.
While the parsing experiments demonstrate the suitability of our representation for capturing the structural elements in the parse tree that are useful for predicting parsing decisions, we are interested in exploring the use of the RNN-based compositional vector representation of parse trees also for seman-tic tasks such as sentiment analysis (Socher et al., 2013b;Tai et al., 2015), sentence similarity judgements (Marelli et al., 2014) and textual entailment (Bowman et al., 2015).
2 Background and Notation

Dependency-based Representation
A dependency-based syntactic representation is centered around syntactic modification relations between head words and modifier words.The result are trees in which each node is a word in the sentence, and every node except for one designated root node has a parent node.A dependency tree over a sentence with n words w 1 , . . ., w n can be represented as a list of n pairs of the form (h, m), where 0 ≤ h ≤ n and 1 ≤ m ≤ n.Each such pair represents an edge in the tree in which h is the index of a head word (including the special ROOT node 0), and m is the index of a modifier word.In order for the dependency trees to be useful for actual downstream language processing tasks, each edge is labeled with a syntactic relation.The tree representation then becomes a list of triplets (h, m, ), where 1 ≤ ≤ L is the index of a dependency relation out of a designated set of L syntactic relations.
Dependency trees tend to be relatively shallow, with some nodes having many children.Looking at trees in the PTB training set we find that 94% of the trees have a height of at most 10, and 49% of the trees a height of at most 6.In terms of width, 93% of the trees have at least one node with an arity of 4 or more, and 56% of the trees have at least one node with an arity of 6 or more.

Recurrent Networks and LSTMs
Recurrent neural networks (RNNs), first proposed by Elman (1990) are statistical learners for modeling sequential data.In this work, we use the RNN abstraction as a building block, and recursively combine several RNNs to obtain our tree representation.We briefly describe the RNN abstraction below.For further detail on RNNs, the reader is referred to sources such as (Goldberg, 2015;Bengio et al., 2015;Cho, 2015).
The RNN abstraction is a function RN N that takes in a sequence of inputs vectors x 1 , . . ., x n (x i ∈ R d in ), and produces a sequence of state vec-tors (also called output vectors) y 1 , . . ., y n (y i ∈ R dout ).Each y i is conditioned on all the inputs x 1 , . . ., x i preceding it.Ignoring the intermediate outputs y 1 , . . ., y n−1 , the RNN can be thought of as encoding the sequence x 1 , . . ., x n into a final state y n .Our notation in this paper follows this view.
The RNN is defined recursively using two functions: Here, a function N takes as input a vector x i and a state vector s i−1 and returns as output a new state s i .One can then extract an output vector y i from s i using the function O (the function O is usually the identity function, or a function that returns a subset of the elements in s i ).
Taking an algorithmic perspective, one can view the RNN as a state object with three operations: s = RNN.initial()returns a new initial state, s.advance(x) takes an input vector and returns a new state, and s.output() returns the output vector for the current state.When clear from the context, we abbreviate and use the state's name (s) instead of s.output() to refer to the output vector at the state.
The functions N and O defining the RNN are parameterized by parameters θ (matrices and vectors), which are trained from data.Specifically, one is usually interested in using some of the outputs y i for making predictions.The RNN is trained such that the encoding y i is good for the prediction task.That is, the RNN learns which aspects of the sequence x 1 , . . ., x i are informative for the prediction.
We use subscripts (i.e.RN N L , RN N R ) to indicate different RNNs, that is, RNNs that have different sets of parameters.
Specific instantiations of N and O yield different recurrent network mechanisms.In this work we use the Long Short Term Memory (LSTM) variant (Hochreiter and Schmidhuber, 1997) which is shown to be a very capable sequence learner.However, our algorithm and encoding method do not rely on any specific property of the LSTM architecture, and the LSTM can be transparently switched for any other RNN variant.

Tree Representation
We now describe our method for representing a tree as a d-dimensional vector.We assume trees in which the children are ordered and there are k l ≥ 0 children before the parent node (left children) and k r ≥ 0 children after it (right children).Such trees correspond well to dependency tree structures.We refer to the parent node as a head, and to its children as modifiers.For a node t, we refer to its left modifiers as t.l 1 , t.l 2 , . . ., t.l k l and its right modifiers as t.r 1 , t.r 2 , . . ., t.r kr The indices of the modifier are always from the parent outward, that is t.l 1 is the left modifier closest to the head t:  The gist of the idea is to treat the modifiers of a node as a sequence, and encode this sequence using an RNN.We separate left-modifiers from rightmodifiers, and use two RNNs: the first RNN encodes the sequence of left-modifiers from the head outwards, and the second RNN the sequence of right-modifiers from the head outwards.The first input to each RNN is the vector representation of the head word, and the last input is the vector representation of the left-most or the right-most modifier.The node's representation is then a concatenation of the RNN encoding of the left-modifiers with the RNN encoding of the right-modifiers.The encoding is recursive: the representation for each of the modifier nodes is computed in a similar fashion.
More formally, consider a node t.Let i(t) be the sentence index of the word corresponding to the head node t, and let v i be a vector corresponding to the ith word in the sentence (this vector captures information such as the word form and its part of speech tag, and will be discussed shortly).The vector encoding of a node enc(t) ∈ R denc is then defined as follows: e l (t) =RN N L (v i(t) , enc(t.l 1 ), . . ., enc(t.lk l )) e r (t) =RN N R (v i(t) , enc(t.r 1 ), . . ., enc(t.rkr )) First, the sequences consisting of the head-vector v i(t) followed by left-modifiers and the head-vector followed by right-modifiers are encoded using two RNNs, RN N L and RN N R , resulting in RNN states e l (t) ∈ R dout and e r (t) ∈ R dout .Then, the RNN states are concatenated, resulting in a 2d outdimensional vector (e l (t) • e r (t)), which is reduced back to d-dimensions using a linear transformation followed by a non-linear activation function g.The recursion stops at leaf nodes, for which: Figure 1 shows the network used for encoding the sentence "the black fox who really likes apples did not jump over a lazy dog yesterday".

Representing words
In the discussion above we assume a vector representation v i associated with the ith sentence word.What does v i look like?A sensible approach would be to take v i to be a function of the word-form and the part-of-speech (POS) tag of the ith word, that is: where w i and p i are the embedded vectors of the word-form and POS-tag of the ith word.
This encodes each word in isolation, disregarding its context.The context of a word can be very informative regarding its meaning.One way of incorporating context is the Bidirectional RNN (Schuster and Paliwal, 1997).Bidirectional RNNs are shown to be an effective representation for sequence tagging (Irsoy and Cardie, 2014)   represent a word in the sentence using a concatenation of the end-states of two RNNs, one running from the beginning of the sentence to the word and the other running from the end to the word.The result is a vector representation for each word which captures not only the word but also its context.We adopt the Bidirectional LSTM scheme to enrich our node vector representation, and for an nwords sentence compute the vector representations v i as follows: We plug this word representation as word vectors, allowing each word vector v i to capture information regarding the word form and POS-tag, as well as the sentential context it appears in.The BI-LSTM encoder is trained jointly with the rest of the network towards the parsing objective, using backpropagation.

Embedding vectors
The word and POS embeddings w i and p i are also trained together with the network.For the word embeddings, we experiment with random initialization, as well as with initialization using pre-trained word embeddings.Our main goal in this work is not to provide top parsing accuracies, but rather to evaluate the ability of the proposed compositional architecture to learn and capture the structural cues that are needed for accurate parsing.Thus, we are most interested in the random initialization setup: what can the network learn from the training corpus alone, without relying on external resources.
However, the ability to perform semi-supervised learning by initializing the word-embeddings with vectors that are pre-trained on large amount of unannotated data is an appealing property of the neuralnetwork approaches, and we evaluate our parser also in this semi-supervised setup.When using pretrained word embeddings, we follow (Dyer et al., 2015) and use embedding vectors which are trained using positional context (Ling et al., 2015), as these were shown to work better than traditional skipgram vectors for syntactic tasks such as part-ofspeech tagging and parsing.

A note on the head-outward generation
Why did we choose to encode the children from the head outward, and not the other way around?The head outward generation order is needed to facilitate incremental tree construction and allowing for efficient parsing, as we show in section 4 below.Beside the efficiency considerations, using the headoutward encoding puts more emphasis on the outermost dependants, which are known to be the most informative for predicting parse structure. 2We rely on the RNN capability of extracting information from arbitrary positions in the sequence to incorporate information about the head word itself, which appears in the beginning of the sequence.This seem to work well, which is expected considering that the average maximal number of siblings in one direction in the PTB is 4.1, and LSTMs were demonstrated to capture much longer-range interactions.Still, when using the tree encoding in a situation where the tree is fully specified in advance, i.e. for sentence classification, sentence similarity or translation tasks, using a head-inward generation order (or even a bidirectional RNN) may prove to work better.We leave this line of inquiry to future work.
The head-outward modifier generation approach has a long history in the parsing literature, and goes back to at least Eisner (1996) and Collins (1997).In contrast to previous work in which each modifier could condition only on a fixed small number of modifiers preceding it, and in which the left-and right-sequences of modifiers were treated as independent from one another for computational efficiency reasons, our approach allows the model to access information from the entirety of both the left and the right sequences jointly.

Parsing Algorithm
We now turn to explain how to parse using the tree encoder defined above.We begin by describing our bottom-up parsing algorithm, and then show how the encoded vector representation can be built and maintained throughout the parsing process.

Bottom-up Parsing
We follow a (projective) bottom-up parsing strategy, similar to the easy-first parsing algorithm of Goldberg and Elhadad (2010).
The main data-structure in the parser is a list of partially-built parse trees we call pending.For a sentence with words w 1 , . . ., w n , the pending list is initialized with n nodes, where pending[i] corresponds to word w i .The algorithm then chooses two neighbouring trees in the pending list pending[i] and pending[i + 1] and either attaches the root of pending[i + 1] as the right-most modifier of the root of pending[i], or attaches the root of pending[i] as the left-most modifier of the root of pending[i + 1].The tree which was treated as modifier is then removed from the pending list, shortening it by one.The process ends after n − 1 steps, at which we remain with a single tree in the pending list, which is taken to be the output parse tree.The parsing process is described in Algorithm 1.

Algorithm 1 Parsing
1: Input: Sentence w = w 1 , . . ., w n 2: for i ∈ 1, . . ., n do pend.remove(i) 11: pend.remove(i + 1) 14: arcs.append(h.id,m.id) 15: return arcs This parsing algorithm is both sound and complete with respect to the class of projective depen-dency trees (Goldberg and Elhadad, 2010).The algorithm depends on non-deterministic choices of an index in the pending list and an attachment direction (line 7).When parsing in practice, the nondeterministic choice will be replaced by using a trained classifier to assign a score to each indexdirection pair, and selecting the highest scoring pair.We discuss the scoring function in Section 4.4, and the training algorithm in Section 5.

Bottom-up Tree-Encoding
We would like the scoring function to condition on the vector encodings of the subtrees it aims to connect.Algorithm 2 shows how to maintain the vector encodings together with the parsing algorithm, so that at every stage in the parsing process each item pending[i] is associated with a vector encoding of the corresponding tree.

Labeled Tree Representation
The tree representation described above does not account for the relation labels the parsing algorithm assigns each edge.In cases the tree is fully specified in advance, the relation of each word to its head can be added to the word representations v i .However, in the context of parsing, the labels become known only when the modifier is attached to its parent.We thus extend the tree representation by concatenating the node vector representation with a vector representation assigned to the label connecting the subtree to its parent.Formally, only the final enc(t) equation changes: where is a learned embedding vector associated with the given label.

Scoring Function
The parsing algorithm relies on a function select(A) for choosing the action to take at each stage.We model this function as: where Score(.) is a learned function whose job is to assign scores to possible actions to reflect their quality.Ideally, it will not only score correct actions above incorrect ones, but also more confident (easier) actions above less confident ones, in order to minimize error propagation in the greedy parsing process.
When scoring a possible attachment between a head h and a modifier m with relation , the scoring function should attempt to reflect the following pieces of information: • Are the head words of h and m compatible under relation l? • Is the modifier m compatible with the already existing modifiers of h?In other words, is m a good subtree to connect as an outer-most modifier in the subtree h? • Is m complete, in the sense that it already acquired all of its own modifiers?to this end, the scoring function looks at a window of k subtrees to each side of the head-modifier pair (pend[i − k], . . ., pend[i + 1 + k]) where the neighbouring subtrees are used for providing hints regarding possible additional modifiers of m and h that are yet to be acquired.We use k = 2 in our experiments, for a total of 6 subtrees in total.This window approach is also used in the Easy-First parser of Goldberg and Elhadad (Goldberg and Elhadad, 2010) and works that extend it (Tratz and Hovy, 2011;Ma et al., 2012;Ma et al., 2013).However, unlike the previous work, which made use of extensive feature engineering and rich feature functions aiming at extracting the many relevant linguistic sub-structures from the 6 subtrees and their interactions, we provide the scoring function solely with the vector-encoding of the 6 subtrees in the window.
Modeling the labeled attachment score is more difficult than modeling the unlabeled score and is prone to more errors.Moreover, picking the label for an attachment will cause less cascading error in contrast to picking the wrong attachment, which will necessarily preclude the parser from reaching the correct tree structure.In order to partially overcome this issue, our scoring function is a sum of two auxiliary scoring function, one scoring unlabeled and the other scoring labeled attachments.The unlabeled attachment score term in the sum functions as a fallback which makes it easier for a parser to predict the attachment direction even when there is no sufficient certainty as to the label.Score(pend, i, d, ) = Score U (pend, i, d) Each of Score U and Score L are modeled as multilayer perceptrons: (d, )] where M LP U and M LP L are standard multilayer perceptron classifiers with one hidden layer (M LP X (x) = W 2 g(W 1 x + b 1 ) + b 2 ) and have output layers with size 2 and 2L respectively, [.] is an indexing operation, and we assume the values of d and (d, ) are mapped to integer values.

Computational Complexity
The Easy-First parsing algorithm works in O(n log n) time (Goldberg and Elhadad, 2010).
The parser in this works differ by three aspects: running a BI-LSTM encoder prior to parsing (O(n)); maintaining the tree representation during parsing (lines 11-22 in Algorithm 2) which take a constant time at each parsing step; and local scoring using an MLP rather than a linear classifier (again, a constant-time operation).Thus, the parser maintains the O(n log n) complexity of the Easy-First parser.

Loss and Parameter Updates
At each step of the parsing process we select the highest scoring action (i, d, ).The goal of training is to set the Score function such that correct actions are scored above incorrect ones.We use a marginbased objective, aiming to maximize the margin between the highest scoring correct action and the set of incorrect actions.Formally, we define a hinge loss for each parsing step as follows: where A is the set of all possible actions and G is the set of correct actions at the current stage.
As the scoring function depends on vectorencodings of the all the trees in the window, and each tree-encoding depends on the network's parameters, each parameter update will invalidate all the vector encodings, requiring a re-computation of the entire network.We thus sum the local losses throughout the parsing process, and update the parameter with respect to the sum of the losses at sentence boundaries.Since we are using hinge loss the gradients will become sparser as the training progresses.Fewer non-zero gradient could translate to unreliable updates.In order to increase gradient stability and training speed, we use a variation of minibatch in which we update the parameters only after 50 errors were made.This assures us a sufficient number of gradients for every update thus minimizing the effect of gradient instability.The gradients of the entire network with respect to the sum of the losses are calculated using the backpropagation algorithm.Initial experiments with an SGD optimizer showed very instable results.We settled instead on using the ADAM optimizer (Kingma and Ba, 2014) which worked well without requiring fiddling with learning rates.

Error-Exploration and Dynamic Oracle
Training At each stage in the training process, the parser assigns scores to all the possible actions (i, d, ) ∈ A. It then selects an action, applies it, and moves to the next step.Which action should be chosen?A sensible option is to define G as the set of actions that can lead to the gold tree, and following the highest scoring actions in this set.However, using training in this manner tends to suffer from error propagation at test time.The parser sees only states that result from following correct actions.The lack of examples containing errors in the training phase makes it hard for the parser to infer the best action given partly erroneous trees.In order to cope with this, we follow the error exploration training strategy, in which we let the parser follow the highest scoring action in A during training even if this action is incorrect, exposing it to states that result from erroneous decisions.This strategy requires defining the set G such that the correct actions to take are well-defined also for states that cannot lead to the gold tree.Such a set G is called a dynamic oracle.Error-exploration and dynamic-oracles were introduced by Goldberg and Nivre (2012).
The Dynamic Oracle A dynamic-oracle for the easy-first parsing system we use is presented in (Goldberg and Nivre, 2013a).Briefly, the dynamicoracle version of G defines the set of gold actions as the set of actions which does not increase the number of erroneous attachments more than the minimum possible (given previous erroneous actions).The number of erroneous attachments is increased in three cases: (1) connecting a modifier to its head prematurely.Once the modifier is attached it is removed from the pending list and therefore can no longer acquire any of its own modifiers; (2) connecting a modifier to an erroneous head, when the correct head is still on the pending list; (3) connecting a modifier to a correct head, but an incorrect label.
Dealing with cases (2) and ( 3) is trivial.To deal with (1), we consider as correct only actions in which the modifier is complete.To efficiently identify complete modifiers we hold a counter for each word which is initialized to the number of modifiers the word has in the gold tree.When applying an attachment the counter of the modifier's gold head word is decreased.When the counter reaches 0, the sub-tree rooted at that word has no pending modifiers, and is considered complete.
Aggressive Exploration We found that even when using error-exploration, after one iteration the model remembers the training set quite well, and does not make enough errors to make error-exploration effective.In order to expose the parser to more errors, we employ a cost augmentation scheme: we sometimes follow incorrect actions also if they score below correct actions.Specifically, when the score of the correct action is greater than that of the wrong action but the difference is smaller than the margin constant, we chose to follow the wrong action with probability p aug (we use p aug = 0.1 in our experiments).Pseudocode for the entire training algorithm is given in the supplementary material.

Out-of-vocabulary items and word-dropout
Due to the sparsity of natural language, we are likely to encounter at test time a substantial number of the words that did not appear in the training data (OOV words).OOV words are likely even when pre-training the word representations on a large unannotated corpora.A common approach is to designate a special "unknown-word" symbol, whose associated vector will be used as the word representation whenever an OOV word is encountered at test time.In order to train the unknown-word vector, a possible approach is to replace all the words appearing in the training corpus less than a certain number of times with the unknown-word symbol.This approach give a good vector representation for unknown words but in the expense of ignoring many of the words from the training corpus.
We instead propose a variant of the word-dropout approach (Iyyer et al., 2015).During training, we replace a word with the unknown-word symbol with probability that is inversely proportional to frequency of the word.Formally, we replace a word w appearing #(w) times in the training corpus with the unknown symbol with a probability: Using this approach we learn a vector representation for unknown words with minimal impact on the training of sparse words.

Implementation Details
Our Python implementation will be made available at the first author's website.We use the PyCNN wrapper of the CNN library 3 for building the computation graph of the network, computing the gradients using automatic differentiation, and performing parameter updates.We noticed the error on the development set does not improve after 20 iterations over the training set, therefore, we ran the training for 20 iterations.The sentences where shuffled between iterations.Non-projective sentences were skipped during training.We use the default parameters initialization, step sizes and regularization values provided by the PyCNN toolkit.The hyperparameters of the final networks used for all the reported experiments are detailed in Table 1.Table 1: Hyper-parameter values used in experiments Weiss et al (2015) stress the importance of careful hyperparameter tuning for achieving top accuracy in neural network based parser.We did not follow this advice and made very few attempts at hyper-parameter tuning, using manual hill climbing until something seemed to work with reasonable accuracy, and then sticking with it for the rest of the experiments.

Experiments and Results
We evaluated our parsing model to English and Chinese data.For comparison purposes we followed the setup of (Dyer et al., 2015).
Data For English, we used the Stanford Dependency (SD) (de Marneffe and Manning, 2008) conversion of the Penn Treebank (Marcus et al., 1993), using the standard train/dev/test splitswith the same predicted POS-tags as used in (Dyer et al., 2015;Chen and Manning, 2014).This dataset contains a few non-projective trees.Punctuation symbols are excluded from the evaluation.
When using external word embeddings, we also use the same data as (Dyer et al., 2015). 4xperimental configurations We evaluated the parser in several configuration.
BOT-TOMUPPARSER is the baseline parser, not using the tree-encoding, and instead representing each item in pending solely by the vector-representation (word and POS) of its head word.
BOTTOMUPPARSER+HTLSTM is using our Hierarchical Tree LSTM representation.BOTTOMUPPARSER+HTLSTM+BI-LSTM is the Hierarchical Tree LSTM where we additionally use a BI-LSTM encoding for the head words.Finally, we added external, pre-trained word embeddings to the BOTTOMUPPARSER+HTLSTM+BI-LSTM setup.We also evaluated the final parsers in a -POS setup, in which we did not feed the parser with any POS-tags.

Results
Results for English and Chinese are presented in Tables 2 and 3 respectively.For comparison, we also show the results of the Stack-LSTM transition-based parser model of Dyer et al (2015), which we consider to be a state-of-the-art greedy model which is also very competitive with searchbased models, with and without pre-trained embeddings, and with and without POS-tags.The trends are consistent across the two languages.The baseline Bottom-Up parser performs very poorly.This is expected, as only the headword of each subtree is used for prediction.When adding the tree-encoding, results jump to near stateof-the-art accuracy, suggesting that the composed vector representation is indeed successful in capturing predictive structural information.Replacing the head-words with their BI-LSTM encodings results in another increase in accuracy for English, outperforming the Dyer et al (S-LSTM no external) models on the test-set.Adding the external pre-trained embeddings further improve the results for both our parser and Dyer et al's model, closing the gap between them.When POS-tags are not provided as input, the numbers for both parsers drop.The drop is small for English and large for Chinese, and our parser seem to suffer a little less than the Dyer et al model.

Importance of the dynamic oracle
We also evaluate the importance of using the dynamic oracle and error-exploration training, and find that they are indeed important for achieving high parsing accura-cies with our model (Table 4).When training without error-exploration (that is, the parser follows only correct actions during training and not using the dynamic aspect of the oracle), accuracies of unseen sentences drop by between 0.4 and 0.8 accuracy points (average 0.58).This is consistent with previous work on training with error-exploration and dynamic oracles (Goldberg and Nivre, 2013b), showing that the technique is not restricted to models trained with sparse linear models.
Comparison to other state-of-the-art parsers Our main point of comparison is the model of Dyer et al, which was chosen because it is (a) a very strong parsing model; and (b) is the closest to ours in the literature: a greedy parsing model making heavy use of LSTMs.To this end, we tried to make the comparison to Dyer et al as controlled as possible, using the same dependency annotation schemes, as well as the same predicted POS-tags and the pre-trained embeddings (when applicable).
It is also informative to position our results with respect to other state-of-the-art parsing results reported in the literature, as we do in Table 5.Here, some of the comparisons are less direct: some of the results use different dependency annotation schemes5 , as well as different predicted POS-tags, and different pre-trained word embeddings.While the numbers are not directly comparable, they do give a good reference as to the expected range of  state-of-the-art parsing results.Our system's English parsing results are in range of state-of-the-art and the Chinese parsing results surpass it.These numbers are achieved while using a greedy, bottom up parsing method without any search, and while relying solely on the compositional tree representations.

Related Work
We survey two lines of related work: methods for encoding trees as vectors, and methods for parsing with vector representations.The popular approach for encoding trees as vectors is using recursive neural networks (Goller and Küchler, 1996;Socher et al., 2010;Tai et al., 2015).Recursive neural networks represent the vector of a parent node in a tree as a function of its children nodes.However, the functions are usually restricted to having a fixed maximum arity (usually two) (Socher et al., 2010;Tai et al., 2015;Socher, 2014).While trees can be binarized to cope with the arity restriction, doing so result in deep trees which in turn lead to the vanishing gradient problem when training.To cope with the vanishing gradients, (Tai et al., 2015) enrich the composition function with a gating mechanism similar to that of the LSTM, resulting in the so-called Tree-LSTM model.Another approach is to allow arbitrary arities but ignoring the sequential nature of the modifiers, e.g. by using a bag-of-modifiers representation or a convo-lutional layer (Tai et al., 2015;Zhu et al., 2015).In contrast, our tree encoding method naturally allows for arbitrary branching trees by relying on the well established LSTM sequence model, and using it as a black box.Very recently, (Zhang et al., 2015) proposed an RNN-based tree encoding which is similar to ours in encoding the sequence of modifiers as an RNN.Unlike our bottom-up encoder, their method works top-down, and is therefor not readily applicable for parsing.On the other hand the top-down approach is well suited for generation.In future work, it could be interesting to combine the bottom-up and top-down approaches in an encoder-decoder framework (Sutskever et al., 2014;Kiros et al., 2015).Work by Dyer et al (2016), that was submitted in parallel to ours, introduces a similar LSTM-based representation of syntactic constituents in the context of phrase-grammar parsing.
In terms of parsing with vector representations, there are four dominant approaches: search based parsers that use local features that are fed to a neural-network classifier (Pei et al., 2015;Durrett and Klein, 2015); greedy transition based parsers that use local features that are fed into a neuralnetwork classifier (Chen and Manning, 2014;Weiss et al., 2015), sometimes coupled with a node composition function (Dyer et al., 2015;Watanabe and Sumita, 2015); bottom up parsers that rely solely on recursively combined vector encodings of sub-trees (Socher et al., 2010;Stenetorp, 2013;Socher et al., 2013a); and parse-reranking approaches that first produced a k-best list of parses using a traditional parsing technique, and then scores the trees based on a recursive vector encoding of each node (Le and Zuidema, 2014;Le and Zuidema, 2015;Zhu et al., 2015).
Our parser is a greedy, bottom up parser that rely on compositional vector encodings of subtrees as its sole set of features.Unlike the re-ranking approaches, we do not rely on an external parser to provide k-best lists.Unlike the bottom-up parser in (Socher et al., 2010) who only parses sentences of up to 15 words and the parser of (Stenetorp, 2013) who achieves very low parsing accuracies, we parse arbitrary sentences with near state-of-the-art accuracy.Unlike the bottom up parser in (Socher et al., 2013a) we do not make use of a grammar.The parser of (Weiss et al., 2015) obtains exceptionally high results using local features and no composition function.The greedy version of their parser uses extensive tuning of hyper-parameters and network depth in order to squeeze every possible bit of accuracy.Adding beam search on top of that further improves results.Due to our much more limited resources, we did not perform a methodological search over hyper-parameters, and explored only a tiny space of the possible hyper-parameters, and our parser does not perform search.Finally, perhaps closest to our approach is the greedy, transition-based parser of (Dyer et al., 2015) that also works in a bottomup fashion, and incorporates an LSTM encoding of the input tokens and hierarchical vector composition into its scoring mechanism.Indeed, that parser obtains similar scores to ours, although we obtain somewhat better results when not using pre-trained embeddings.We differ from the parser of Dyer et al by having a more elaborate vector-composition function, relying solely on the compositional representations, and performing fully bottom-up parsing without being guided by a stack-and-buffer control structure.

Conclusions and Future Work
We suggest a compositional vector representation of parse trees that relies on a recursive combination of recurrent-neural network encoders, and demonstrate its effectiveness by integrating it in a bottom-up easy-first parser.Future extensions in terms of parsing include the addition of beam search, handling of unknown-words using character-embeddings, and adapting the algorithm to constituency trees.We also plan to establish the effectiveness of our Hierarchical Tree-LSTM encoder by applying it to more semantic vector representation tasks, i.e. training tree representation for capturing sentiment (Socher et al., 2013b;Tai et al., 2015), semantic sentence similarity (Marelli et al., 2014) or textual inference (Bowman et al., 2015). t

Figure 1 :
Figure1: Network for encoding the sentence "the black fox who really likes apples did not jump over a lazy dog yesterday".Top: the network structure: boxed nodes represent LSTM cells, where L are cells belonging to the leftmodifiers sequence model RN N L , and R to the right-modifiers sequence model RN N R .Circle nodes represent a concatenation followed by a linear transformation and a non-linearity.Bottom: the dependency parse of the sentence.

Table 5 :
(Zhu et al., 2015)014)of various state-of-the-art parsing systems on the English and Chinese datasets.The systems that use embeddings use different pre-trained embeddings.English results use predicted POS tags (different systems use different taggers), while Chinese results use gold POS tags.PTB-YM: English PTB, Yamada and Matsumoto head rules.PTB-SD: English PTB, Stanford Dependencies (different systems may use different versions of the Stanford converter.CTB: Chinese Treebank.reranking/blend in method column indicates a reranking system where the reranker score is interpolated with the base-parser's score.The reranking systems' runtimes are those of the base parsers they use.O(n)+ indicates a linear-time system with a large multiplicative constant.The different systems and the numbers reported from them are taken from: ZhangNivre11: (Zhang and Nivre, 2011); Martins13:(Martins et al., 2013); Weiss15(Weiss et al., 2015); Pei15:(Pei et al., 2015); LeZuidema14(Le and Zuidema, 2014); Zhu15:(Zhu et al., 2015).