Joint Arc-factored Parsing of Syntactic and Semantic Dependencies

In this paper we introduce a joint arc-factored model for syntactic and semantic dependency parsing. The semantic role labeler predicts the full syntactic paths that connect predicates with their arguments. This process is framed as a linear assignment task, which allows to control some well-formedness constraints. For the syntactic part, we define a standard arc-factored dependency model that predicts the full syntactic tree. Finally, we employ dual decomposition techniques to produce consistent syntactic and predicate-argument structures while searching over a large space of syntactic configurations. In experiments on the CoNLL-2009 English benchmark we observe very competitive results.


Introduction
Semantic role labeling (SRL) is the task of identifying the arguments of lexical predicates in a sentence and labeling them with semantic roles (Gildea and Jurafsky, 2002;. SRL is an important shallow semantic task in NLP since predicate-argument relations directly represent semantic properties of the type "who" did "what" to "whom", "how", and "why" for events expressed by predicates (typically verbs and nouns).
Predicate-argument relations are strongly related to the syntactic structure of the sentence: the majority of predicate arguments correspond to some syntactic constituent, and the syntactic structure that connects an argument with the predicate is a strong indicator of its semantic role. Actually, semantic roles represent an abstraction of the syntactic form of a predicative event. While syntactic functions of arguments change with the form of the event (e.g., active vs. passive forms), the semantic roles of arguments remain invariant to their syntactic realization.
Consequently, since the first works, SRL systems have assumed access to the syntactic structure of the sentence (Gildea and Jurafsky, 2002;Carreras and Màrquez, 2005). A simple approach is to obtain the parse trees as a pre-process to the SRL system, which allows the use of unrestricted features of the syntax. However, as in other pipeline approaches in NLP, it has been shown that the errors of the syntactic parser severely degrade the predictions of the SRL model (Gildea and Palmer, 2002). A common approach to alleviate this problem is to work with multiple alternative syntactic trees and let the SRL system optimize over any input tree or part of it (Toutanova et al., 2008;Punyakanok et al., 2008). As a step further, more recent work has proposed parsing models that predict syntactic structure augmented with semantic predicate-argument relations (Surdeanu et al., 2008;Hajič et al., 2009;Johansson, 2009;, which is the focus of this paper. These joint models should favor the syntactic structure that is most consistent with the semantic predicate-argument structures of a sentence. In principle, these models can exploit syntactic and semantic features simultaneously, and could potentially improve the accuracy for both syntactic and semantic relations. One difficulty in the design of joint syntacticsemantic parsing models is that there exist important structural divergences between the two layers.  Das et al. (2012) . . . Riedel and McCallum (2011) . . .

A Syntactic-Semantic Dependency Model
We will describe structures of syntactic and semantic dependencies with vectors of binary variables. We will denote by y h,m,l a syntactic dependency from head token h to dependant token m labeled with syntactic function l. Similarly we will denote by z p,a,r a semantic dependency between predicate token p and argument token a labeled with semantic role r. We will use y and z to denote vectors of binary variables indexed by syntactic and semantic dependencies, respectively. A joint model for syntactic and semantic dependency parsing could be defined as: argmax y,z s syn(x, y) + s srl(x, z, y) .
In the equation, s syn(x, y) gives a score for the syntactic tree y. In the literature, it is standard to use arc-factored models defined as s syn(x, y) = y h,m,l =1 s syn (x, h, m, l) , where we overload s syn to be a function that computes scores for individual labeled syntactic dependencies. In discriminative models one has s syn(x, h, m, l) = w syn · f syn (x, h, m, l), where f syn is a feature vector for the syntactic dependency and w syn is a vector of parameters (McDonald et al., 2005).
The other term, s srl(x, z, y), gives a score for a semantic dependency structure using the syntactic structure y as features. Previous work has empirically proved the importance of exploiting syntactic features in the semantic component (Gildea and Jurafsky, 2002;Xue and Palmer, 2004;Punyakanok et al., 2008). However, without further assumptions, this property makes the optimization problem computationally hard. One simple approximation is to use a pipeline model: first compute the optimal syntactic tree, and then optimize for the best semantic structure given the syntactic tree. In the rest of the paper we describe a method that searches over syntactic and semantic dependency structures jointly.
We first impose the assumption that syntactic features of the semantic component are restricted to the syntactic path between a predicate and an argument, following previous work (Johansson, 2009). Formally, for a predicate p, argument a and role r we will define a vector of dependency indicators π p,a,r similar to the ones above: π p,a,r h,m,l indicates if a dependency h, m, l is part of the syntactic path that links predicate p with token a. Figure 1 gives an example of one such paths. Given full syntactic and semantic structures y and z it is trivial to construct a vector π that is a concatenation of vectors π p,a,r for all p, a, r in z. We can now define a linear semantic model as s srl(x, z, π) = z p,a,r =1 s srl(x, p, a, r, π p,a,r ) , (1) where s srl computes a score for a semantic dependency p, a, r together with its syntactic path π p,a,r . As in the syntactic component, this function is typically defined as a linear function over a set of features of the semantic dependency and its path. With this joint model, the inference problem can be formulated as: argmax y,z,π s syn(x, y) + s srl(x, z, π) (2) subject to cTree : y is a valid dependency tree cRole : ∀p, r : a z p,a,r ≤ 1 cArg : ∀p, a : r z p,a,r ≤ 1 cPath : ∀p, a, r : if z p,a,r = 1 then π p,a,r is a path from p to a otherwise π p,a,r = 0 cSubtree : ∀p, r, a : π p,r,a is a subtree of y Figure 1: A sentence with syntactic dependencies (top) and semantic dependencies for the predicates "loves" and "play" (bottom). The thick arcs illustrate a structural divergence where the argument "Mary" is linked to "play" with a path involving three syntactic dependencies. This is clearly seen in dependency-based representations of syntax and semantic roles (Surdeanu et al., 2008), such as in the example in Figure 1: the construct "loves to" causes the argument "Mary" to be syntactically distant from the predicate "play". Linguistic phenomena such as auxiliary verbs, control and raising, typically result in syntactic structures where semantic arguments are not among the direct dependants of their predicate -e.g., about 25% of arguments are distant in the English development set of the CoNLL-2009 shared task. Besides, standard models for dependency parsing crucially depend on arc factorizations of the dependency structure (Mc-Donald et al., 2005;Nivre and Nilsson, 2005), otherwise their computational properties break. Hence, it is challenging to define efficient methods for syntactic and semantic dependency parsing that can exploit features of both layers simultaneously. In this paper we propose a method for this joint task.
In our method we define predicate-centric semantic models that, rather than predicting just the argument that realizes each semantic role, they predict the full syntactic path that connects the predicate with the argument. We show how efficient predictions with these models can be made using assignment algorithms in bipartite graphs. Simultaneously, we use a standard arc-factored dependency model that predicts the full syntactic tree of the sentence. Finally, we employ dual decomposition techniques (Koo et al., 2010; to find agreement between the full dependency tree and the partial syntactic trees linking each predicate with its arguments. In summary, the main contributions of this paper are: • We frame SRL as a weighted assignment problem in a bipartite graph. Under this framework we can control assignment constraints between roles and arguments. Key to our method, we can efficiently search over a large space of syntactic realizations of semantic arguments. • We solve joint inference of syntactic and semantic dependencies with a dual decomposition method, similar to that of Koo et al. (2010).
Our system produces consistent syntactic and predicate-argument structures while searching over a large space of syntactic configurations.
In the experimental section we compare joint and pipeline models. The final results of our joint syntactic-semantic system are competitive with the state-of-the-art and improve over the best results published by a joint method on the CoNLL-2009 English dataset.

A Syntactic-Semantic Dependency Model
We first describe how we represent structures of syntactic and semantic dependencies like the one in Figure 1. Throughout the paper, we will assume a fixed input sentence x with n tokens where lexical predicates are marked. We will also assume fixed sets of syntactic functions R syn and semantic roles R sem . We will represent depencency structures using vectors of binary variables. A variable y h,m,l will indicate the presence of a syntactic dependency from head token h to dependant token m labeled with syntactic function l. Then, a syntactic tree will be denoted as a vector y of variables indexed by syntactic dependencies. Similarly, a variable z p,a,r will indicate the presence of a semantic dependency between predicate token p and argument token a labeled with semantic role r. We will represent a semantic role structure as a vector z indexed by semantic dependencies. Whenever we enumerate syntactic dependencies h, m, l we will assume that they are in the valid range for x, i.e. 0 ≤ h ≤ n, 1 ≤ m ≤ n, h = m and l ∈ R syn , where h = 0 stands for a special root token. Similarly, for semantic dependencies p, a, r we will assume that p points to a predicate of x, 1 ≤ a ≤ n and r ∈ R sem . 220 A joint model for syntactic and semantic dependency parsing could be defined as: argmax y,z s syn(x, y) + s srl(x, z, y) . (1) In the equation, s syn(x, y) gives a score for the syntactic tree y. In the literature, it is standard to use arc-factored models defined as where we overload s syn to be a function that computes scores for individual syntactic dependencies. In linear discriminative models one has s syn(x, h, m, l) = w syn · f syn (x, h, m, l), where f syn is a feature vector for a syntactic dependency and w syn is a vector of parameters (McDonald et al., 2005). In Section 6 we describe how we trained score functions with discriminative methods. The other term in Eq. 1, s srl(x, z, y), gives a score for a semantic dependency structure z using features of the syntactic structure y. Previous work has empirically proved the importance of exploiting syntactic features in the semantic component (Gildea and Jurafsky, 2002;Xue and Palmer, 2004;Punyakanok et al., 2008). However, without further assumptions, this property makes the optimization problem computationally hard. One simple approximation is to use a pipeline model: first compute the optimal syntactic tree y, and then optimize for the best semantic structure z given y. In the rest of the paper we describe a method that searches over syntactic and semantic dependency structures jointly.
We first note that for a fixed semantic dependency, the semantic component will typically restrict the syntactic features representing the dependency to a specific subtree of y. For example, previous work has restricted such features to the syntactic path that links a predicate with an argument (Moschitti, 2004;Johansson, 2009), and in this paper we employ this restriction. Figure 1 gives an example of a subtree, where we highlight the syntactic path that connects the semantic dependency between "play" and "Mary" with role ARG0.
Formally, for a predicate p, argument a and role r we define a local syntactic subtree π p,a,r represented as a vector: π p,a,r h,m,l indicates if a dependency h, m, l is part of the syntactic path that links predicate p with token a and role r. 1 Given full syntactic and semantic structures y and z it is trivial to construct a vector π that concatenates vectors π p,a,r for all p, a, r in z. The semantic model becomes s srl(x, p, a, r, π p,a,r ) , (3) where s srl computes a score for a semantic dependency p, a, r together with its syntactic path π p,a,r . As in the syntactic component, this function is typically defined as a linear function over a set of features of the semantic dependency and its path.
The inference problem of our joint model is: subject to cTree : y is a valid dependency tree cRole : ∀p, r : ∀p, a, r : if z p,a,r = 1 then π p,a,r is a path from p to a, otherwise π p,a,r = 0 cSubtree : ∀p, a, r : π p,a,r is a subtree of y Constraint cTree dictates that y is a valid dependency tree; see (Martins et al., 2009) for a detailed specification. The next two sets of constraints concern the semantic structure only. cRole imposes that each semantic role is realized at most once. 2 Conversely, cArg dictates that an argument can realize at most one semantic role in a predicate. The final two sets of constraints model the syntactic-semantic interdependencies. cPath imposes that each π p,a,r represents a syntactic path between p and a whenever there exists a semantic relation. Finally, cSubtree imposes that the paths in π are consistent with the full syntactic structure, i.e. they are subtrees.
In Section 3 we define a process that optimizes the semantic structure ignoring constraint cSubtree. Then in Section 4 we describe a dual decomposition method that uses the first process repeatedly to solve the joint problem.

SRL as Assignment
In this section we frame the problem of finding semantic dependencies as a linear assignment task. The problem we optimize is: subject to cRole, cArg, cPath In this case we dropped the full syntactic structure y from the optimization in Eq. 4, as well as the corresponding constraints cTree and cSubtree. As a consequence, we note that the syntactic paths π are not tied to any consistency constraint other than each of the paths being a well-formed sequence of dependencies linking the predicate to the argument. In other words, the optimal solution in this case does not guarantee that the set of paths from a predicate to all of its arguments satisfies tree constraints. We first describe how these paths can be optimized locally. Then we show how to find a solution z satisfying cRole and cArg using an assignment algorithm.

Local Optimization of Syntactic Paths
Let z and π be the optimal values of Eq. 5. For any p, a, r , let π p,a,r = argmax π p,a,r s srl(x, p, a, r, π p,a,r ) . (6) For any p, a, r such that z p,a,r = 1 it has to be that π p,a,r =π p,a,r . If this was not true, replacing π p,a,r withπ p,a,r would improve the objective of Eq. 5 without violating the constraints, thus contradicting the hypothesis about optimality of π. Therefore, for each p, a, r we can optimize its best syntactic path locally as defined in Eq. 6. In this paper, we will assume access to a list of likely syntactic paths for each predicate p and argument candidate a, such that the optimization in Eq. 6 can be solved explicitly by looping over each path in the list. The main advantage of this method is that, since paths are precomputed, our model can make unrestricted use of syntactic path features.
(1) Figure 2: Illustration of the assignment graph for the sentence "Mary plays guitar", where the predicate "plays" can have up to three roles: ARG0 (agent), ARG1 (theme) and ARG2 (benefactor). Nodes labeled NULL represent a null role or token. Highlighted edges are the correct assignment.
It is simple to employ a probabilistic syntactic dependency model to create the list of likely paths for each predicate-argument pair. In the experiments we explore this approach and show that with an average of 44 paths per predicate we can recover 86.2% of the correct paths.
We leave for future work the development of efficient methods to recover the most likely syntactic structure linking an argument with its predicate.

The Assignment Algorithm
Coming back to solving Eq. 5, it is easy to see that an optimal solution satisfying constraints cRole and cArg can be found with a linear assignment algorithm. The process we describe determines the predicate-argument relations separately for each predicate. Assume a bipartite graph of size N with role nodes r 1 . . . r N on one side and argument nodes a 1 . . . a N on the other side. Assume also a matrix of non-negative scores W i,j corresponding to assigning argument a j to role r i . A linear assignment algorithm finds a bijection f : i → j from roles to arguments that maximizes N i=1 W i,f (i) . The Hungarian algorithm finds the exact solution to this problem in O(N 3 ) time (Kuhn, 1955;Burkard et al., 2009).
All that is left is to construct a bipartite graph representing predicate roles and sentence tokens, such that some roles and tokens can be left unassigned, which is a common setting for assignment tasks. Algorithm 1 describes a procedure for constructing a weighted bipartite graph for SRL, and Figure 2 illustrates an example of a bipartite graph. We then 222 Algorithm 1 Construction of an Assignment Graph for Semantic Role Labeling Let p be a predicate with k possible roles. Let n be the number of argument candidates in the sentence. This algorithm creates a bipartite graph with N = n + k vertices on each side.
• for 1 ≤ i ≤ n, r k+i is a special NULL role.
2. Create argument vertices a j for j = 1 . . . N , where • for 1 ≤ j ≤ n, a j is the j-th argument candidate, • for 1 ≤ j ≤ k, a n+j is a special NULL argument.
3. Define a matrix of model scores S ∈ R (k+1)×n : (a) Optimization of syntactic paths: For 1 ≤ i ≤ k, 1 ≤ j ≤ n S i,j = max π p,a j ,r i s srl(x, p, a j , r i , π p,aj ,ri ) (b) Scores of NULL assignments 3 : For 1 ≤ j ≤ n S k+1,j = 0 4. Let S 0 = min i,j S i,j , the minimum of any score in S. Define a matrix of non-negative scores W ∈ R N ×N as follows: run the Hungarian algorithm on the weighted graph and obtain a bijection f : r i → a j , from which it is trivial to recover the optimal solution of Eq. 5. Finally, we note that it is simple to allow for multiple instances of a semantic role by adding more role nodes in step 1; it would be straightforward to add penalties in step 3 for multiple instances of roles.

A Dual Decomposition Algorithm
We now present a dual decomposition method to optimize Eq. 4, that uses the assignment algorithm presented above as a subroutine. Our method is similar to that of Koo et al. (2010), in the sense that our joint optimization can be decomposed into two sub-problems that need to agree on the syntactic dependencies they predict. For a detailed description of dual decomposition methods applied to NLP see . We note that in Eq. 4 the constraint cSubtree ties the syntactic and semantic structures, imposing that any path π p,a,r that links a predicate p with an argument a must be a subtree of the full syntactic structure y. Formally the set of constraints is: y h,m,l ≥ π p,a,r h,m,l ∀ p, a, r, h, m, l . These constraints can be compactly written as where c is a constant equal to the number of distinct semantic dependencies p, a, r . In addition, we can introduce a vector non-negative slack variables ξ with a component for each syntactic dependency ξ h,m,l , turning the constraints into: We can now rewrite Eq. 4 as: argmax y,z,π,ξ≥0 s syn(x, y) + s srl(x, z, π) subject to cTree, cRole, cArg, cPath ∀h, m, l : c · y h,m,l − p,a,r π p,a,r h,m,l − ξ h,m,l = 0 As in Koo et al. (2010), we will relax subtree constraints by introducing a vector of Lagrange multipliers λ indexed by syntactic dependencies, i.e. each coordinate λ h,m,l is a Lagrange multiplier for the constraint associated with h, m, l . The Lagrangian of the problem is: L(y, z, π, ξ, λ)= s syn(x, y) + s srl(x, z, π) + λ · c · y − p,a,r π p,a,r − ξ (8) We can now formulate Eq. 7 as: max y,z,π,ξ≥0 s.t. cTree,cRole,cArg,cPath c·y− p,a,r π p,a,r −ξ=0 L(y, z, π, ξ, λ) 223 This optimization problem has the property that its optimum value is the same as the optimum of Eq. 7 for any value of λ. This is because whenever the constraints are satisfied, the terms in the Lagrangian involving λ are zero. If we remove the subtree constraints from Eq. 9 we obtain the dual objective: y,z,π,ξ≥0 s.t. cTree,cRole,cArg,cPath L(y, z, π, ξ, λ) The dual objective is an upper bound to the optimal value of primal objective of Eq. 7. Thus, we are interested in finding the minimum of the dual in order to tighten the upper-bound. We will solve min λ D(λ) using a subgradient method. Algorithm 2 presents pseudo-code. The algorithm takes advantage of the decomposed form of the dual in Eq. 11, where we have rewritten the Lagrangian such that syntactic and semantic structures appear in separate terms. This will allow to compute subgradients efficiently. In particular, the subgradient of D at a point λ is: where y = argmax y s.t. cTree s syn(x, y) + c · y · λ (14) z, π = argmax z,π s.t. cRole,cArg,cPath s srl(x, z, π) − λ· p,a,r π p,a,r (15) Whenever π is consistent with y the subgradient will be zero and the method will converge. When paths π contain a dependency h, m, l that is inconsistent with y, the associated dual λ h,m,l will increase, hence lowering the score of all paths that use h, m, l at the next iteration; at same time, the total score for that dependency will increase, favoring syntactic dependency structures alternative to y. As Algorithm 2 A dual-decomposition algorithm for syntactic-semantic dependency parsing Input: x, a sentence; T , number of iterations; Output: syntactic and semantic structures y and z Notation: we use cSem= cRole ∧ cArg ∧ cPath 1: λ 1 = 0 # initialize dual variables 2: c =number of distinct h, m, l in x 3: for t = 1 . . . T do 4: y = argmax y s.t. cTree s syn(x, y) + c · λ t · y 5: z, π = argmax z,π s.t. cSem s srl(x, z, π) −λ t · p,a,r π p,a,r 6: λ t+1 = λ t # dual variables for the next iteration 7: Set α t , the step size of the current iteration 8: for each h, m, l do 9: q = p,a,r π p,a,r h,m,l # num. paths using h, m, l 10: if q > 0 and y h,m,l = 0 then 11: λ t+1 h,m,l = λ t+1 h,m,l + α t q 12: break if λ t+1 = λ t # convergence 13: return y, z in previous work, in the algorithm a parameter α t controls the size of subgradient steps at iteration t.
The key point of the method is that solutions to Eq. 14 and 15 can be computed efficiently using separate processes. In particular, Eq. 14 corresponds to a standard dependency parsing problem, where for each dependency h, m, l we have an additional score term c · λ h,m,l -in our experiments we use the projected dependency parsing algorithm by (Eisner, 2000). To calculate Eq. 15 we use the assignment method described in Section 3, where it is straightforward to introduce additional score terms −λ h,m,l to every factor π p,a,r h,m,l . It can be shown that whenever the subgradient method converges, the solutions y and z are the optimal solutions to our original problem in Eq. 4 (see (Koo et al., 2010) for a justification). In practice we run the subgradient method for a maximum number of iterations, and return the solutions of the last iteration if it does not converge.

Related Work
Recently, there have been a number of approaches to joint parsing of syntactic and semantic dependencies, partly because of the availability of treebanks in this format popularized by the CoNLL shared tasks (Surdeanu et al., 2008;Hajič et al., 2009).
Like in our method, Johansson (2009) defined a model that exploits features of a semantic depen-dency together with the syntactic path connecting the predicate and the argument. That method uses an approximate parsing algorithm that employs k-best inference and beam search. Similarly,  defined a joint model that forces the predicate structure to be represented in the syntactic dependency tree, by enriching arcs with semantic information. The semantic component uses features of pre-computed syntactic structures that may diverge from the joint structure. In contrast, our joint parsing method is exact whenever the dual decomposition algorithm converges.  augmented a transition-based dependency parser with operations that produce synchronous derivations of syntactic and semantic structures. Instead of explicitly representing semantic dependencies together with a syntactic path, they induce latent representations of the interactions between syntactic and semantic layers.
In all works mentioned the model has no control of assignment constraints that disallow labeling multiple arguments with the same semantic role. Punyakanok et al. (2008) first introduced a system that explicitly controls these constraints, as well as other constraints that look at pairwise assignments which we can not model. They solve SRL using general-purpose Integer Linear Programming (ILP) methods. In similar spirit, Riedel and McCallum (2011) presented a model for extracting structured events that controls interactions between predicateargument assignments. They take into account pairwise assignments and solve the optimization problem with dual decomposition. More recently, Das et al. (2012) proposed a dual decomposition method that deals with several assignment constraints for predicate-argument relations. Their method is an alternative to general ILP methods. To our knowledge, our work is the first that frames SRL as a linear assignment task, for which simple and exact algorithms exist. We should note that these works model predicate-argument relations with assignment constraints, but none of them predicts the underlying syntactic structure.
Our dual decomposition method follows from that of Koo et al. (2010). In both cases two separate processes predict syntactic dependency structures, and the dual decomposition algorithm seeks agreement at the level of individual dependencies. One dif-ference is that our semantic process predicts partial syntax (restricted to syntactic paths connecting predicates and arguments), while in their case each of the two processes predicts the full set of dependencies.

Experiments
We present experiments using our syntacticsemantic parser on the CoNLL-2009 Shared Task English benchmark (Hajič et al., 2009). It consists of the usual WSJ training/development/test sections mapped to dependency trees, augmented with semantic predicate-argument relations from PropBank (Palmer et al., 2005) and NomBank (Meyers et al., 2004) also represented as dependencies. It also contains a PropBanked portion of the Brown corpus as an out-of-domain test set.
Our goal was to evaluate the contributions of parsing algorithms in the following configurations: Base Pipeline Runs a syntactic parser and then runs an SRL parser constrained to paths of the best syntactic tree. In the SRL it only enforces constraint cArg, by simply classifying the candidate argument in each path into one of the possible semantic roles or as NULL. Pipeline with Assignment Runs the assignment algorithm for SRL, enforcing constraints cRole and cArg, but constrained to paths of the best syntactic tree. Forest Runs the assignment algorithm for SRL on a large set of precomputed syntactic paths, described below. This configuration corresponds to running Dual Decomposition for a single iteration, and is not guaranteed to predict consistent syntactic and semantic structures. Dual Decomposition (DD) Runs dual decomposition using the assignment algorithm on the set of precomputed paths. Syntactic and semantic structures are consistent when it reaches convergence.
All four systems used the same type of discriminative scorers and features. Next we provide details about these systems. Then we present the results.

Implementation
Syntactic model We used two discriminative arcfactored models for labeled dependency parsing: a first-order model, and a second-order model with grandchildren interactions, both reimplementations of the parsers by McDonald et al. (2005) and Carreras (2007) respectively. In both cases we used projective dependency parsing algorithms based on (Eisner, 2000). 4 To learn the models, we used a log-linear loss function following Koo et al. (2007), which trains probabilistic discriminative parsers. At test time, we used the probabilistic parsers to compute marginal probabilities p(h, m, l | x), using inside-outside algorithms for first/second-order models. Hence, for either of the parsing models, we always obtain a table of first-order marginal scores, with one score per labeled dependency. Then we run first-order inference with these marginals to obtain the best tree. We found that the higher-order parser performed equally well on development using this method as using second-order inference to predict trees: since we run the parser multiple times within Dual Decomposition, our strategy results in faster parsing times.
Precomputed Paths Both Forest and Dual Decomposition run assignment on a set of precomputed paths, and here we explain how we build it. We first observed that 98.4% of the correct arguments in development data are either direct descendants of the predicate, direct descendants of an ancestor of the predicate, or an ancestor of the predicate. 5 All methods we test are restricted to this syntactic scope. To generate a list of paths, we did as follows: • Calculate marginals of unlabeled dependencies using the first-order parser: p(h, m | x) = l p(h, m, l | x). Note that for each m, the probabilities p(h, m|x) for all h form a distribution (i.e. they sum to one). Then, for each m, keep the most-likely dependencies that cover at least 90% of the mass, and prune the rest.
• Starting from a predicate p, generate a path by taking any number of dependencies that ascend, and optionally adding one dependency that descends. We constrained paths to be projective, and to have a maximum number of 6 ascendant dependencies. • Label each unlabeled edge h, m in the paths with l = argmax l p(h, m, l | x).
On development data, this procedure generated an average of 43.8 paths per predicate that cover 86.2% of the correct paths. In contrast, enumerating paths of the single-best tree covers 79.4% of correct paths for the first-order parser, and 82.2% for the secondorder parser. 6

SRL model
We used a discriminative model with similar features to those in the system of Johansson (2009). In addition, we included the following: • Unigram/bigram/trigram path features. For all n-grams in the syntactic path, patterns of words and POS tags (e.g., mary+loves+to, mary+VB+to).
• Voice features. The predicate voice together with the word/POS of the argument (e.g., pas-sive+mary).
• Path continuity. Count of non-consecutive tokens in a predicate-argument path.
To train SRL models we used the averaged perceptron (Collins, 2002). For the base pipeline we trained standard SRL classifiers. For the rest of models we used the structured Perceptron running the assignment algorithm as inference routine. In this case, we generated a large set of syntactic paths for training using the procedure described above, and we set the loss function to penalize mistakes in predicting the semantic role of arguments and their syntactic path.
Dual Decomposition We added a parameter β weighting the syntactic and semantic components of the model as follows: (1 − β) s syn(x, y) + β s srl(x, z, π) .
As syntactic scores we used normalized marginal probabilities of dependencies, either from the first or the higher-order parser. The scores of all factors of the SRL model were normalized at every sentence to be between -1 and 1. The rest of details  of the method were implemented following Koo et al. (2010), including the strategy for decreasing the step size α t . We ran the algorithm for up to 500 iterations, with initial step size of 0.001.

Results
To evaluate syntactic dependencies we use unlabeled attachment score (UAS), i.e., the percentage of words with the correct head, and labeled attachment scores (LAS), i.e., the percentage of words with the correct head and syntactic label. Semantic predicate-argument relations are evaluated with precision (sem p ), recall (sem r ) and F 1 measure (sem F 1 ) at the level of labeled semantic dependencies. In addition, we measure the percentage of perfectly predicted predicate structures (sem pp ). 7 Table 1 shows the results on the development set for our three first methods. We can see that the pipeline methods running assignment improve over the baseline pipelines in semantic F 1 by about 2 points, due to the application of the cRole constraint. The Forest method also shows an improvement in recall of semantic roles with respect to the pipeline methods. Presumably, the set of paths available in the Forest model allows to recognize a higher number of arguments at an expense of a lower precision. Regarding the percentage of perfect predicateargument structures there is a remarkable improvement in the systems that apply the full set of con-7 Our evaluation metrics differ slightly from the official metric at CoNLL-2009. That metric considers predicate senses as special semantic dependencies and, thus, it includes them in the calculation of the evaluation metrics. In this paper, we are not addressing predicate sense disambiguation and, consequently, we ignore predicate senses when presenting evaluation results. When we report the performance of CoNLL systems, their scores will be noticeably lower than the scores reported at the shared task. This is because predicate disambiguation is a reasonably simple task with a very high baseline around 90%. straints using the assignment algorithm. We believe that the cRole constraint that ensures no repeated roles for a given predicate is a key factor to predict the full set of arguments of a predicate.
The Forest configuration is the starting point to run the dual decomposition algorithm. We ran experiments for various values of the β parameter. Table 2 shows the results. We see that as we increase β, the SRL component has more relative weight, and the syntactic structure changes. The DD methods are always able to improve over the Forest methods, and find convergence in more than 99.5% of sentences. Compared to the pipeline running assignment, DD improves semantic F 1 for first-order inference, but not for higher-order inference, suggesting that 2 nd order predictions of paths are quite accurate. We also observe slight benefits in syntactic accuracy. Table 3 presents results of our system on the test sets, where we run Pipeline with Assignment and Dual Decomposition with our best configuration (β = 0.8/0.9 for 1 st /2 nd order syntax). For comparison, the table also reports the results of the best CoNLL-2009 joint system, Merlo09 (Gesmundo et al., 2009), which proved to be very competitive ranking third in the closed challenge. We also include Lluís09 , which is another joint syntactic-semantic system from CoNLL-2009. 8 In the WSJ test DD obtains the best syntactic accuracies, while the Pipeline obtains the best se-  mantic F 1 . The bottom part of Table 3 presents results on the out-of-domain Brown test corpus. In this case, DD obtains slightly better results than the rest, both in terms of syntactic accuracy and semantic F 1 . Table 4 shows statistical significance tests for the syntactic LAS and semantic F 1 scores of Table 3. We have applied the sign test (Wackerly et al., 2007) and approximate randomization tests (Yeh, 2000) to all pairs of systems outputs. The differences between systems in the WSJ test can be considered significant in almost all cases with p = 0.05. In the Brown test set, results are more unstable and differences are not significant in general, probably because of the relatively small size of that test.
Regarding running times, our implementation of the baseline pipeline with 2 nd order inference parses the development set (1,334 sentences) in less than 7 minutes. Running assignment in the pipeline increases parsing time by ∼8% due to the overhead from the assignment algorithm. The Forest method, with an average of 61.3 paths per predicate, is ∼13% slower than the pipeline due to the exploration of the space of precomputed paths. Finally, Dual Decomposition with 2 nd order inference converges in 36.6 iterations per sentence on average. The first iteration of DD has to perform roughly the same work as Forest, while subsequent iterations only need to re-parse the sentence with respect to the dual upare slightly different. However, note that Merlo09 is an application of the system by . In that paper authors report results on the CoNLL-2008 datasets, and they are comparable to Johansson's.

WSJ
Brown ME PA1 DD1 PA2 DD2 ME PA1 DD1 PA2 DD2 Table 4: Statistical tests of significance for LAS and sem F1 differences between pairs of systems from Table 3.
dates, which are extremely sparse. Our current implementation did not take advantage of the sparsity of updates, and overall, DD was on average 13 times slower than the pipeline running assignment and 15 times slower than the baseline pipeline.

Conclusion
We have introduced efficient methods to parse syntactic dependency structures augmented with predicate-argument relations, with two key ideas. One is to predict the local syntactic structure that links a predicate with its arguments, and seek agreement with the full syntactic structure using dual decomposition techniques. The second is to control linear assignment constraints in the predicateargument structure. In experiments we observe large improvements resulting from the assignment constraints. As for the dual decomposition technique for joint parsing, it does improve over the pipelines when we use a first order parser. This means that in this configuration the explicit semantic features help to find a solution that is better in both layers. To some extent, this empirically validates the research objective of joint models. However, when we move to second-order parsers the differences with respect to the pipeline are insignificant. It is to be expected that as syntactic parsers improve, the need of joint methods is less critical. It remains an open question to validate if large improvements can be achieved by integrating syntactic-semantic features. To study this question, it is necessary to have efficient parsing algorithms for joint dependency structures. This paper contributes with a method that has optimality guarantees whenever it converges.
Our method can incorporate richer families of features. It is straightforward to incorporate better semantic representations of predicates and arguments than just plain words, e.g. by exploiting WordNet or distributional representations as in (Zapirain et al., 2013). Potentially, this could result in larger improvements in the performance of syntactic and semantic parsing.
It is also necessary to experiment with different languages, where the performance of syntactic parsers is lower than in English, and hence there is potential for improvement. Our treatment of local syntactic structure that links predicates with arguments, based on explicit enumeration of likely paths, was simplistic. Future work should explore methods that model the syntactic structure linking predicates with arguments: whenever this structure can be parsed efficiently, our dual decomposition algorithm can be employed to define an efficient joint system.