CoNLL-2009 Shared Task
Filip Jurcicek
SLTC Newsletter, July 2009
Results from the shared task evaluation of the Conference on Computational Natural Language Learning (CoNLL-2009) were presented in Boulder, CO, USA at June 4-5, 2009. Shared tasks have a long tradition at CoNLL and it already was the 11th task for which the organizers provided common task definition, data, and evaluation. In 2009, the shared task was dedicated to the joint parsing of syntactic and semantic dependencies in multiple languages.
This year's task can be considered as an extension of the 2008 shared task. Similar to the CoNLL-2008 task, the main objective of the shared task was to explore joint parsing techniques under a unified dependency based formalism. As in 2008, the organizers of the shared task hypothesized that parsing using information available from a semantic parser for syntactic parsing and vice versa could improve upon the more common cascade approach. Note that the typical approach to syntactic and semantic parsing is to build a pipeline composed of a syntactic parser followed by semantic analysis using features derived during syntactic parsing. In contrast to the monolingual setting of the CoNLL-2008 task, the organizers extended the task to cover multiple languages. The goal of the organizers was to evaluate how different approaches could cope with different languages such as English, Catalan, Chinese, Czech, German, Japanese and Spanish.
Following last year's task, the organizers designed numerous categories in which they evaluated the submitted results. Among others, the most important categories were "close" and "open" challenges. In the closed challenge, participants were only allowed to use training data provided by the organisers. In contrast, the open challenge allowed participants to incorporate any kind of available knowledge. Moreover, the challenges were divided into two tasks: (1) joint task - both syntactic dependency parsing and semantic role labelling had to performed, (2) semantic role labelling only task - the syntactic dependency parses generated by state-of-the-art parsers for the individual languages were provided and only the semantic parsing had to be performed. According to the organizers, the most interesting evaluation was the joint task-closed challenge because it allowed evaluation of the benefits of joint parsing in a "fair" environment which was not influenced by use of different data. This year, 20 systems participated in the closed challenge; 13 systems in the joint task and seven in the semantic role labelling task only. Moreover, two systems competed in the open challenge (joint task).
As the main topic of the shared task was joint parsing, the organizers were very interested in a comparison between the joint and the cascade approaches. However, only four of the 20 systems used the joint model for syntactic and semantic dependency parsing. To give an example of a joint parsing approach, the best joint parser "Merlo", which was third in the main evaluation, used synchronous syntactic and semantic derivations in an incremental parsing model using the joint probability of the syntactic and semantic dependencies. The system maintained two independent stacks of syntactic and semantic derivations synchronized at each word. The synchronous derivations were modelled with an Incremental Sigmoid Belief Network that had latent variables to represent properties of parsing history relevant to the next step in parsing (Gesmundo et al., 2009). On the other hand, the over-all best system "Che" was based on a cascade of three components; syntactic parsing, predicate classification, and semantic role labelling. First, the syntactic parser implemented the high-order Eisner parsing algorithm using spanning trees (Eisner et al., 2000). Second, a support vector machine model was used to classify the predicates. Finally, maximum entropy model was used for semantic role classification combined with integer linear programming to enforce global constraints on the predicted roles (Che et al., 2009). As the best two systems were not using any joint parsing and the number of joint parsers was low, the organizers note that it is not still clear whether joint parsing offers a significant improvement over other approaches.
Overall the shared task of 2009 was very demanding because the participants had to cope with several languages and large amount of data in short development time (about two months). For example, Wanxiang Che, who submitted the winning system of the joint task, says that his group had to use Amazon EC2 cloud computing service to be able to train and test their models for all seven languages in such short time. Although processing the large amount of data, determining initial settings and the right features of the parser was computationally expensive, Wanxiang Che notes that careful feature engineering worked in their favor. Also Hai Zhao, who submitted the best semantic role labelling-only system, emphasized that the crucial part of the training process was determining the right features. Only by using parallel computation and clever optimization was his team able to select the right features for the parser (Zhao et al., 2009).
References
For more information (including links to data sets and task documentation), see:
- CoNLL Shared Task 2009 website: http://ufal.mff.cuni.cz/conll2009-st/
- CoNLL Shared Task 2009 - Evaluation math and script : http://ufal.mff.cuni.cz/conll2009-st/scorer.html
- CoNLL Shared Task 2009 - Evaluation data and procedure description : http://ufal.mff.cuni.cz/conll2009-st/eval-data.html
- CoNLL Shared Task 2008 website: http://barcelona.research.yahoo.net/conll2008/
- Wanxiang Che, Zhenghua Li, Yongqiang Li, Yuhang Guo, Bing Qin and Ting Liu. Multilingual Dependency-based Syntactic and Semantic Parsing. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL- 2009), June 4-5, Boulder, Colorado, USA. June 4-5.
- Jason Eisner. 2000. Bilexical grammars and their cubictime parsing algorithms. In Advances in Probabilistic and Other Parsing Technologies.
- Andrea Gesmundo, James Henderson, Paola Merlo, and Ivan Titov. A latent variable model of synchronous syntactic-semantic parsing for multiple languages. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL- 2009), June 4-5, Boulder, Colorado, USA. June 4-5.
- Hai Zhao, Wenliang Chen, Chunyu Kit, Guodong Zhou. Multilingual Dependency Learning: A Huge Feature Engineering Method to Semantic Dependency Parsing. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL- 2009), June 4-5, Boulder, Colorado, USA. June 4-5.
Add A Comment