About: ---------- This package includes an implementation of the Neutral Edge Direction (NED) evaluation measure. This is a measure used to evaluate dependency parses against a gold standard. This code can also be found in: http://www.cs.huji.ac.il/~roys02/software/ned.html A complete description of NED appears in the paper: "Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing" by Roy Schwartz, Omri Abend, Roi Reichart and Ari Rappoport. Copying and Modifying --------------------------- This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. A copy of the GNU General Public License can be found in the file 'License.txt' which is part of this distribution. Package Contents: --------------------------- This package includes the following files: 1. README - this file 2. NED.pl - executable file which performs the evaluation. 3. figure2_gold.txt, figure2_test.txt - example files. They include two possible annotations of the sentence "I want to eat", as presented in Figure 2 in the paper. 4. ParseSentence.pm, POS.pm - required (perl) packages. Compilation and Installation: NED.pl is a perl script, which requires perl 5 installed. The following (standard) perl packages are also required: Getopt::Long IO::File List::Util No compilation is required. NED.pl is the executable file. ParseSentence.pm and POS.pm must be downloaded and put in the same directory as NED.pl. Running the script must be done from this directory. Running: ------------ To run, type: perl NED.pl -if -gold -ml -h (display this help message) Input: -------- The input and gold files need to be in the following format: Each token appears in its own line. Blank lines separate between sentences. Any of the following formats is supported: 1. Each line contains only the token's parent (e.g., "2") 2. Each line contains tab separated word, POS tag and parent (e.g., "I PRP 2") 3. Each line contains tab separated CoNLL format (e.g., "1 I _ _ PRP _ 2 _ 2 _") The parent field indicates the token's parent, and can be a number in the range 1- or the root, which can be either -1 or 0. In formats where punctuation appears (i.e., format 2 or 3), either of the following POS tags are considered punctuation and are thus ignored: 1. , . : # ( ) 2. -LRB- -RRB- 3. `` '' 4. LS SYM PUNCT Input file and gold standard file must have exactly the same number of lines, where the blank lines must agree. Other file formats will invoke an error. If both files contain words and POS tags (i.e., format 2 or 3), they must also match for each line or an error will be invoked. Using test and gold files in different formats (i.e., a gold file in format 1 and a test file in format 2) is supported (as long as both are appear in one of the formats above, and follow the rules above). Output: ----------- / () Examples: --------------- figure2_gold.txt and figure2_test.txt include the gold parse and some induced parse of the sentence "I want to eat", as presented in figure 2 in the paper and are included in this distribution. Invoking this command: NED.pl -if figure2_test.txt -gold figure2_gold.txt Yields the following output: 4/4 (1.000) This means that the evaluation of figure2_test.txt against figure2_gold.txt results in 4 correct tokens out of 4 - a NED score of 1. Invoking this command: NED.pl -if WSJ_parsed_file -gold WSJ_23_gold -ml 10 Yields the following output: 2155/2672 (0.807) This means that the evaluation of WSJ_parsed_file (not supplied in this distribution) against WSJ10 (not supplied either) results in 2155 correct tokens out of 2672 and a NED score of 0.807. Errors: ---------- Errors are invoked in case the file format does not match one of the formats above, or if the files are inconsistent (see input section). Bad dependency graphs (e.g., graphs which contain a loop or a punctuation token that has dependents) might also invoke an error.