In-Order Transition-based Constituent Parsing

Both bottom-up and top-down strategies have been used for neural transition-based constituent parsing. The parsing strategies differ in terms of the order in which they recognize productions in the derivation tree, where bottom-up strategies and top-down strategies take post-order and pre-order traversal over trees, respectively. Bottom-up parsers benefit from rich features from readily built partial parses, but lack lookahead guidance in the parsing process; top-down parsers benefit from non-local guidance for local decisions, but rely on a strong encoder over the input to predict a constituent hierarchy before its construction. To mitigate both issues, we propose a novel parsing system based on in-order traversal over syntactic trees, designing a set of transition actions to find a compromise between bottom-up constituent information and top-down lookahead information. Based on stack-LSTM, our psycholinguistically motivated constituent parsing system achieves 91.8 F1 on the WSJ benchmark. Furthermore, the system achieves 93.6 F1 with supervised reranking and 94.2 F1 with semi-supervised reranking, which are the best results on the WSJ benchmark.


Introduction
Transition-based constituent parsing employs sequences of local transition actions to construct constituent trees over sentences. There are two popular transition-based constituent parsing systems, namely bottom-up parsing (Sagae and Lavie, 2005;Zhang and Clark, 2009;Zhu et al., 2013;Watanabe and Sumita, 2015) and top-down parsing Kuncoro et al., 2017). The parsing strategies differ in terms of the order in which they recognize productions in the derivation tree.
The process of bottom-up parsing can be regarded as post-order traversal over a constituent tree. For example, given the sentence in Figure  1, a bottom-up shift-reduce parser takes the action sequence in Table 2(a) 1 to build the output, where the word sequence "The little boy" is first read, and then an NP recognized for the word sequence. After the system reads the verb "likes" and its subsequent NP, a VP is recognized. The full order of recognition for the tree nodes is 3 → 4 → 5 → 2 → 7 → 9 → 10 → 8 → 6 → 11 → 1 . When making local decisions, rich information is available from readily built partial trees (Zhu et al., 2013;Watanabe and Sumita, 2015;Cross and Huang, 2016), which contributes to local disambiguation. However, there is lack of top-down guidance from lookahead information, which can be useful (Johnson, 1998;Roark and Johnson, 1999;Charniak, 2000;Liu and Zhang, 2017). In addition, binarization must be applied to trees, as shown in Figure 1(b), to ensure a constant number of actions (Sagae and Lavie, 2005), and to take advantage of lexical head information (Collins, 2003). However, such binarization requires a set of language-specific rules, which hampers adaptation of parsing to other languages.
On the other hand, the process of top-down parsing can be regarded as pre-order traversal over a tree. Given the sentence in Figure 1 Figure 1: Syntactic trees of the sentence "The little boy likes red tomatoes.". (a) syntactic tree; (b) binarized syntactic tree, where r and l mean the head is the right branch and the left branch, respectively, and * means this constituent is not completed.
shift-reduce parser takes the action sequence in Table 2(b) to build the output, where an S is first made and then an NP is generated. After that, the system makes a decision to read the word sequence "The little boy" to complete the NP. The full order of recognition for the tree nodes is 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10 → 11 . The top-down lookahead guidance contributes to non-local disambiguation. However, it is difficult to generate a constituent before its sub constituents have been realized, since no explicit features can be extracted from their subtree structures. Thanks to the use of recurrent neural networks, which make it possible to represent a sentence globally before syntactic tree construction, seminal work of neural top-down parsing directly generates bracketed constituent trees using sequence-to-sequence models (Vinyals et al., 2015).   (c) in-order system Figure 2: Action sequences of three types of transition constituent parsing system. Details of the action system are introduced in Section 2.1, Section 2.2 and Section 3, respectively.
In this paper, we propose a novel transition system for constituent parsing, mitigating issues of both bottom-up and top-down systems by finding a compromise between bottom-up constituent information and top-down lookahead information. The process of the proposed constituent parsing can be regarded as in-order traversal over a tree. Given the sentence in Figure 1, the system takes the action sequence in Table 2(c) to build the output. The system reads the word "The" and then projects an NP, which is based on bottom-up evidence. After this, based on the projected NP, the system reads the word sequence "little boy", with top-down guidance from NP. Similarly, based on the completed constituent "(NP The little boy)", the system projects an S, with the bottom-up evidence. With the S and the word "likes", the system projects 414 an VP, which can serve as top-down guidance. The full order of recognition for the tree nodes is 3 → 2 → 4 → 5 → 1 → 7 → 6 → 9 → 8 → 10 → 11 . Compared to post-order traversal, in-order traversal can potentially resolve non-local ambiguity better by top-down guidance. Compared to pre-order traversal, in-order traversal can potentially resolve local ambiguity better by bottom-up evidence.
For example, when reading a word "likes", a human reader could guess that it could be a start of a constituent VP, instead of waiting to read the object "red tomatoes", which is the procedure of a bottom-up system. We compare our system with the two baseline systems (i.e. a top-down system and a bottomup system) under the same neural transition-based framework of . Our final models outperform both of the bottom-up and top-down transition-based constituent parsing by achieving a 91.8 F 1 in English and a 86.1 F 1 in Chinese for greedy fully-supervised parsing, respectively. Furthermore, our final model obtains a 93.6 F 1 with supervised reranking (Choe and Charniak, 2016) and a 94.2 F 1 with semi-supervised reranking, achieving the state-of-the-art results on constituent parsing on the English benchmark. By converting to Stanford dependencies, our final model achieves the state-ofthe-art results on dependency parsing by obtaining a 96.2% UAS and a 95.2% LAS. To our knowledge, we are the first to systematically compare top-down and bottom-up constituent parsing under the same neural framework. We release our code at https://github.com/LeonCrashCode/ InOrderParser.

Transition-based constituent parsing
Transition-based constituent parsing takes a leftto-right scan of the input sentence, where a stack is used to maintain partially constructed phrasestructures, while the input words are stored in a buffer. Formally, a state is defined as [σ, i, f ], where σ is the stack, i is the front index of the buffer, and f is a boolean value showing that the parsing is finished. At each step, a transition action is applied to consume an input word or construct a new phrasestructure. Different parsing systems employ their own sets of actions.

Bottom-up system
We take the bottom-up system of Sagae and Lavie (2005) as our bottom-up baseline. Given a state, the set of transition actions are: • SHIFT: pop the front word from the buffer, and push it onto the stack.
• REDUCE-L/R-X: pop the top two constituents off the stack, combine them into a new constituent with label X, and push the new constituent onto the stack.
• UNARY-X: pop the top constituent off the stack, raise it to a new constituent with label X, and push the new constituent onto the stack.
• FINISH: pop the root node off the stack and end parsing.
The bottom-up parser can be summarized as the deductive system in Figure 3(a). Given the sentence with the binarized syntactic tree in Figure  1(b), the sequence of actions SHIFT, SHIFT, SHIFT, REDUCE-R-NP, REDUCE-R-NP, SHIFT, SHIFT, SHIFT, REDUCE-R-NP, REDUCE-L-VP, SHIFT, REDUCE-L-S, REDUCE-R-S and FINISH, can be used to construct its constituent tree.

Top-down system
We take the top-down system of  as our top-down baseline. Given a state, the set of transition actions are: • SHIFT: pop the front word from the buffer, and push it onto the stack.
• NT-X: open a nonterminal with label X on top of the stack.
• REDUCE: repeatedly pop completed subtrees or terminal symbols from the stack until an open nonterminal is encountered, and then this open NT is popped and used as the label of a new constituent that has the popped subtrees as (c) in-order system its children. This new completed constituent is pushed onto the stack as a single composite item.
The deduction system for the process is shown in Figure 3(b) 2 . Given the sentence in Figure 1, the sequence of actions NT-S, NT-NP, SHIFT, SHIFT, SHIFT, REDUCE, NT-VP, SHIFT, NT-NP, SHIFT, SHIFT, REDUCE, REDUCE, SHIFT and REDUCE, can be used to construct its constituent tree.

In-order system
We propose a novel in-order system for transitionbased constituent parsing. Similar to the bottom-up and top-down systems, the in-order system maintains a stack and a buffer for representing a state. The set of transition actions are defined as: • SHIFT: pop the front word from the buffer, and push it onto the stack.
• PJ-X: project a nonterminal with label X on top of the stack.
• REDUCE: repeatedly pop completed subtrees or terminal symbols from the stack until a projected nonterminal encountered, and then this projected nonterminal is popped and used as the label of a new constituent. Furthermore, one more item on the top of stack is popped and inserted as the leftmost child of the new constituent. The popped subtrees are inserted as the rest of the children. This new completed constituent is pushed onto the stack as a single composite item.
• FINISH: pop the root node off the stack and end parsing.
The deduction system for the process is shown in Figure 3(c). Given the sentence in Figure 1, the sequence of actions SHIFT, PJ-NP, SHIFT, SHIFT, REDUCE, PJ-S, SHIFT, PJ-VP, SHIFT, PJ-NP, SHIFT, REDUCE, REDUCE, SHIFT, REDUCE, FIN-ISH can be used to construct its constituent tree.

Variants
The in-order system can be generalized into variants by modifying k, the number of leftmost nodes traced before the parent node. For example, given the tree "(S a b c d)", the traversal is "a S b c d" if k = 1 while the traversal is "a b S c d" if k = 2. We name each variant with a certain k value as k-in-order systems. In this paper, we only investigate the in-order system with k = 1, the 1-inorder system. Note that the top-down parser can be regarded as a special case of a generalized version of the in-order parser with k = 0, and the bottom-up parser can be regarded as a special case with k = ∞.

Neural parsing model
We employ the stack-LSTM parsing model of  for the three types of transition-based parsing systems in Section 2.1, 2.2 and 3, respectively, where a stack-LSTM is used to represent the stack, a stack-LSTM is used to represent the buffer, and a vanilla LSTM is used to represent the action history, as shown in Figure 4.

Word representation
We follow Dyer et al. (2015), representing each word using three different types of embeddings, including pretrained word embedding, e w i , which is not fine-tuned during the training of the parser, randomly initialized embeddings e w i , which is finetuned, and the randomly initialized part-of-speech embeddings, which is fine-tuned. The three embeddings are concatenated, and then fed to nonlinear layer to derive the final word embedding: where W input and b input are model parameters, w i and p i denote the form and the POS tag of the ith input word, respectively, and f is an nonlinear function. In this paper, we use ReLu for f .  (b) is for binarized trees, where "NP-r*" means that "little boy" is a non-completed noun phrase with head "boy".

Stack representation
We employ a bidirectional LSTM as the composition function to represent constituents on stack 3 . For top-down parsing and in-order parsing, following , as shown in Figure 5 where e nt is the representation of a non-terminal, s j , j ∈ [0, m] is the jth child node, and m is the number of the child nodes. For bottom-up parsing, we make use of the head information in the composition function by requiring the order that the head node is always before the non-head node in the bidirectional LSTM, as shown in Figure 5(b) 4 . The bi-narized composition is computed as: where s h and s o is the representation of the head and the non-head node, respectively.

Greedy action classification
Given a sentence w 0 , w 1 , ..., w n−1 , where w i is the ith word, and n is the length of the sentence, our parser makes local action classification decisions incrementally. For the kth parsing state like [s j , ..., s 1 , s 0 , i, false], the probability distribution of the current action p is: where W and b are model parameters, the representation of stack information h stk is: the representation of buffer information h buf is: x is the word representation, and the representation of action history h ah is: where e act k−1 is the representation of action in the k-1th parsing state. Training Our models are trained to minimize a cross-entropy loss objective with an l 2 regularization term, defined by where θ is the set of parameters, p a ij is the probability of the jth action in the ith training example given by the model and λ is a regularization hyperparameter (λ = 10 −6 ). We use stochastic gradient descent with a 0.1 initialized learning rate with a 0.05 learning rate decay.

Data
We empirically compare our bottom-up, top-down and in-order parsers. The experiments are carried out on both English and Chinese. For English data, we use the standard benchmark of WSJ sections in PTB (Marcus et al., 1993), where the Sections 2-21 are taken for training data, Section 22 for development data and Section 23 for testing both dependency parsing and constituency parsing. We adopt the pretrained English word embeddings generated on the AFP portion of English Gigaword.
For Chinese data, we use Version 5.1 of the Penn Chinese Treebank (CTB) (Xue et al., 2005). We use articles 001-270 and 440-1151 for training, articles 301-325 for system development, and articles 271-300 for final performance evaluation. We adopt the pretrained Chinese word embeddings generated on the complete Chinese Gigaword corpus.
The POS tags in both the English data and the Chinese data are automatically assigned the same as the work of , using Stanford tagger. We follow the work of Choe and Charniak (2016) and adopt the AFP portion of the English Gigaword as the extra resources for the semi-supervised reranking.

Settings
Hyper-parameters For both English and Chinese experiments, we use the same hyper-parameters as the work of  without further optimization, as shown in Table 1.

Reranking experiments
Following the same reranking setting of  and Choe and Charniak (2016) Table 2 shows the development results of the three parsing systems. The bottom-up system performs slightly better than the top-down system. The inorder system outperforms both the bottom-up and the top-down system. Table 3 shows the parsing results on the English test dataset. We find that the bottom-up parser and the top-down parser have similar results under the greedy setting, and the in-order parser outperforms both of them. Also, with supervised reranking, the in-order parser achieves the best results. English constituent results We compare our models with previous work, as shown in Table 4. With the fully-supervise setting 5 , the inorder parser outperforms the state-of-the-art discrete parser (Shindo et al., 2012;Zhu et al., 2013), the state-of-the-art neural parsers (Cross and Huang,5 Here, we only consider the work of a single model.
With the reranking setting, the in-order parser outperforms the best discrete parser (Huang, 2008) and has the same performance as Kuncoro et al. (2017), which extends the work of  by adding a gated attention mechanism on composition functions. With the semi-supervised setting, the inorder parser outperforms the best semi-supervised parser (Choe and Charniak, 2016) by achieving 94.2 F 1 (the oracle is 97.9 F 1 ).
English dependency results As shown in Table  5, by converting to Stanford Dependencies, without additional training data, our models achieve a similar performance with the state-of-the-art system (Choe and Charniak, 2016); with the same additional training data, our models achieve new stateof-the-art results on dependency parsing by achieving 96.2% UAS and 95.2% LAS on standard benchmark.
order parser achieves the best results under the fullysupervised setting. With the supervised reranking, the in-order parser outperforms the state-of-the-art models by achieving 88.0 F 1 (the oracle is 93.47 F 1 ).
Chinese dependency results As shown in Table  7, by converting the results to dependencies 6 , our final model achieves the best results among transitionbased parsing, and obtains comparable results to the state-of-the-art graph-based models.  Table 7: Dependency accuracy (%) on CTB test set. † means graph-based parsing. "-re" means supervised reranking.

Analysis
We analyze the results of Section 23 in WSJ given by our model (i.e. in-order parser) and two baseline models (i.e. the bottom-up parser and the top-down parser) against the sentence length, the span length and the constituent type, respectively.
6.1 Influence of sentence length Figure 6 shows the F 1 scores of the three parsers on sentences of different lengths. Compared to the topdown parser, the bottom-up parser performs better on the short sentences with the length falling in the range . This is likely because the bottomup parser takes advantages of rich local features from partially-built trees, which are useful for parsing short sentences. However, these local structures are can be insufficient for parsing long sentences due to error propagation. On the other hand, the top-down parser obtains better results on long sentences with the length falling in the range [40][41][42][43][44][45][46][47][48][49][50]. This is because, as the length of sentences increase, lookahead features become rich and they could be correctly represented by the LSTM, which is beneficial for parsing non-local structures. We find that the in-order parser performs the best for both short and long sentences, showing the advantages of integrating bottom-up and top-down information.
6.2 Influence of span length Figure 7 shows the F 1 scores of the three parsers on spans of different lengths. The trend of performances of the two baseline parsers are similar. Compared to the baseline parsers, the in-order parser obtains significant improvement on long spans. Linguistically, it is because the in-order traversal, (over   a tree) allows constituent types of spans to be correctly projected based on the information of the beginning (leftmost nodes) of the spans. Then the projected constituents constrain long span construction, which is different from the top-down parser, generating constituent types of spans without trace of the spans. Table 7 shows the F 1 scores of the three parsers on frequent constituent types. The bottom-up parser performs better than the top-down parser on con-stituent types including NP, S, SBAR, QP. We find that the prediction of these constituent types requires, explicitly, modeling of bottom-up structures.

Influence of constituent type
In other words, bottom-up information is necessary for us to know if the span can be a noun phrase (NP) or sentence (S), for example. On the other hand, the top-down parser has better performance on WHNP, which is likely because a WHNP starts with a certain question word, making the prediction easy without bottom-up information. The in-order parser performs the best on all constituent types, demonstrating that the in-order parser can benefit from both bottom-up and top-down information.

Examples
We give output examples from the test set to qualitatively compare the performances of the three parsers using the fully-supervised model without reranking, as shown in Table 9. For example, given the Sentence #2006, the bottom-up and the in-order parsers give both correct results. However, the top-down parser makes an incorrect decision to generate an S, leading to subsequent incorrect decisions on VP to complete S. Sentence pattern ambiguity allows topdown guidance to over-parsing the sentence by recognizing the word "Plans" as a verb, while more bottom-up information is useful for the local disambiguation. Given the Sentence #308, the bottom-up parser prefers construction of local constituents such as "once producers and customers", ignoring the possible clause SBAR, however, which is captured by the in-order parser. The parser projects a constituent SBAR from the word "stick" and continues to complete the clause, showing that top-down lookahead information is necessary for non-local disambiguation. The in-order parser gives the correct output for the Sentence #2066 and the Sentence #308, showing that it can benefit from bottom-up and top-down information. ... whether the new posted prices will stick once producers and customers start to haggle . Gold ... (VP will (VP stick (SBAR once (S (NP producers and customers ) (VP start (S ...) ) ) ) ) ) ... Top-down ... (VP will (VP stick (SBAR once (S (NP producers and customers ) (VP start (S ...) ) ) ) ) ) ... Bottom-up ... (VP will (VP stick (NP once producers and customers ) ) ) ... (VP start (S ...) ) ... In-order ... (VP will (VP stick (SBAR once (S (NP producers and customers ) (VP start (S ...) ) ) ) ) ) ... Sent #1715 This has both made investors uneasy and the corporations more vulnerable . Gold (S (NP This) (VP has (VP both made (S (S investors uneasy) and (S the corporations ...)))) .) Top-down (S (S (NP This) (VP has (S (NP both) (VP made investors uneasy)))) and (S the corporations ...) .) Bottom-up (S (S (NP This) (VP has (S both (VP made investors uneasy)))) and (S the corporations ...) .) In-order (S (NP This) (VP has both (VP made (S (S investors uneasy) and (S the corporations ...)))) .) Table 9: Output examples of the three parsers on the English test set. Incorrect constituents are marked in red.
In the Sentence #1715, there are coordinated objects such as "investors uneasy" and "the corporations more vulnerable". All of the three parsers can recognize coordination. However, the top-down and the bottom-up parsers incorrectly recognize the "This has both made investors uneasy" as a complete sentence. The top-down parser incorrectly generates S, marked in red, at a early stage, leaving no choice but to follow this incorrect non-terminal. The bottom-up parser without lookahead information makes incorrect local decisions. By contrast, the in-order parser reads the word "and" and projects a non-terminal S for coordination after completing "(S investors uneasy)". On the other hand, the inorder parser is confused by projecting for the word "made" or the word "both" into an VP, which we think could be addressed by using a in-order system variant with k=2 described in Section 3.

Related work
Our work is related to left corner parsing. Rosenkrantz and Lewis (1970) formalize this in automata theory, which have appeared frequently in the compiler literature. Roark and Johnson (1999) apply the strategy into parsing. Typical works investigate the transformation of syntactic trees based on left-corner rules (Roark, 2001;Schuler et al., 2010;Van Schijndel and Schuler, 2013). In contrast, we propose a novel general transition-based in-order constituent parsing system.
Neural networks have achieved the state-of-the-art for parsing under various grammar formalisms, including dependency (Dozat and Manning, 2017), constituent Kuncoro et al., 2017), and CCG parsing (Xu, 2016;Lewis et al., 2016). Seminal work employs transition-based methods (Chen and Manning, 2014). This method has been extended by investigating more complex representations of configurations for constituent parsing (Watanabe and Sumita, 2015;.  employ stack-LSTM onto the top-down system, which is the same as our topdown parser. Watanabe and Sumita (2015) employ tree-LSTM to model the complex representation in the stack in bottom-up system. We are the first to investigate in-order traversal by designing a novel transition-based system under the same neural structure model framework.

Conclusion
We proposed a novel psycho-linguistically motivated constituent parsing system based on the inorder traversal over syntactic trees, aiming to find a compromise between bottom-up constituent information and top-down lookahead information. On the standard WSJ benchmark, our in-order system outperforms bottom-up parsing on a non-local ambiguity and top-down parsing on local decision. The resulting parser achieves the state-of-the-art constituent parsing results by obtaining 94.2 F 1 and dependency parsing results by obtaining 96.2% UAS and 95.2% LAS. 422