Instructions for William's Significance Test for Evaluation of Automatic MT Metrics 
-----------------------------------------------------------------------------------

Contact: graham.yvette@gmail.com

-----------------------------------------------------------------------------------

Requirements:

1. "R Statistical Software"
	- To install R on the command line: 
	  > sudo apt-get install r-base

2. R's "psych" package 

	To install R's "psych" package:
	- IF your institution uses a proxy server, you need to tell R about it BEFORE 
	  installing any package, here's what to do:
		A) Open your R command line, by typing "R"
		B) Type the following commmand into R, remembering to provide your 
		   actual credentials and proxy server details:

        > Sys.setenv(http_proxy="http://myusername:mypasssord@myproxyserver.com:8080/")

        IF NOT, continue on below. 

	- Open R command line (by typing "R") and enter the following: 

        > install.packages("psych")

	  You'll be given an option of a CRAN site, when you have one selected, you 
          might need to answer "y" to some questions. When "psych" is finished 
          installing, type the following to quit R:

        > quit("no")

-----------------------------------------------------------------------------------

Usage:

bash williams-sig.sh <language pair> <human scores filename> <metric scores filename> <directory to write results>

Example:
bash williams-sig.sh es-en example-data/mt-systems-scored-by-humans example-data/mt-systems-scored-by-metrics example-data

The above command produces two files, the first containing Pearson correlations of 
each metric with human scores: 
	example-data/pearson-corr.es-en
Another containing the resulting p-value of William's test for the pair of metrics: 
	example-data/williams-results.es-en

To check everything ran correctly, you can compare the above files produced by the 
scripts with the following two files included in the download:
	example-output/orig.pearson-corr.es-en
	example-output/orig.williams-results.es-en

The version of William's test carried out is a one-tailed test. The resulting p-value 
of the test will be printed in the ROW of the metric with HIGHEST absolute Pearson 
correlation with human scores. The opposite cell for that same pair of metrics (in the 
row of the lowest absolute Pearson correlation metric) will contain "-". The test 
carried out is for significance of the DIFFERENCE in correlation between a pair of 
metric's absolute Pearson correlation with human scores. 

The script can be run on data for more than two metrics. Example data are included 
in this directory: "example-data-multi". The same command is used to call the script 
on this example data for multiple metrics:

bash williams-sig.sh es-en example-data-multi/mt-systems-scored-by-humans example-data-multi/mt-systems-scored-by-metrics example-data-multi

The input files can contain multiple language pairs data, as in example-data-multi. 
For example, to run the script for "en-fr" language pair, simply specify that language 
pair:

bash williams-sig.sh en-fr example-data-multi/mt-systems-scored-by-humans example-data-multi/mt-systems-scored-by-metrics example-data-multi

-----------------------------------------------------------------------------------

Example Data:

Example data is the Spanish-English shared translation task MT systems from WMT-12. 
They are scored by humans, in addition to all participating metrics in WMT'12 metrics 
shared task. To run the scripts on new data, simply provide scores in the format in 
the example files "example-data/mt-systems-scored-by-humans" and 
"example-data/mt-systems-scored-by-metrics". The required format is also decribed in 
detail below.

You should provide the script with the locations of two input files containing:
	(1) human scores;
	(2) metric scores. 
Below is a detailed description of their required format.

Human Scores File:
------------------
You need to provide the scripts with a tab-delimited file containing the human 
scores for a number of MT systems. The file must have the header row as in 
"example-data/mt-systems-scored-by-humans" (see below), the column METRIC should 
specify "HUMAN" for all human scores and the contents of the LP column should match 
the language pair abbreviation you use to call the script on the command line 
(eg "es-en" as below):
-------------------------------------------------------
METRIC	LP	TESTSET	SYSTEM	SCORE	          <--HEADER ROW SHOULD LOOK LIKE THIS
HUMAN	es-en	newstest2012	onlineA	61.9384
HUMAN	es-en	newstest2012	onlineB	61.3861
HUMAN	es-en	newstest2012	QCRI--primary	60.373
HUMAN	es-en	newstest2012	uedin-wmt12	58.4963
HUMAN	es-en	newstest2012	UPC	56.5355
HUMAN	es-en	newstest2012	GTH-UPM	52.3248
HUMAN	es-en	newstest2012	onlineD	51.1139
HUMAN	es-en	newstest2012	jhu-hiero	47.8846
HUMAN	es-en	newstest2012	onlineC	46.1035
HUMAN	es-en	newstest2012	onlineF	42.2644
HUMAN	es-en	newstest2012	onlineE	41.9155
HUMAN	es-en	newstest2012	uk-dan-moses	18.7297
-------------------------------------------------------

Metrics Scores File:
--------------------
You need to provide a tab-delimited file containing automatic metric scores for the 
same set of MT systems as the human scores file (also provided by you). Each MT 
system should be scored exactly once by each metric and a minimum of two metrics 
should be tested. It is fine just to include two metrics, for example BLEU (as a 
baseline) and your own metric. If more than two metrics are included in this file, 
the script will carry out significance tests for all pairs of metrics and produce 
a matrix of p-values. The header of the metrics scores file you provide to the 
script needs to read as the first line below (same as human file header) as in 
example-data/mt-systems-scored-by-humans (see below). The column LP should contain 
the same abbreviation you use to call the script and METRIC should specify the name 
of the correpsonding metric. You can include multiple language pairs data in the input 
files.
----------------------------------------------------------------------
METRIC	LP	TESTSET	SYSTEM	SCORE
AMBER	es-en	newstest2012	GTH-UPM	0.2107
AMBER	es-en	newstest2012	jhu-hiero	0.2104
AMBER	es-en	newstest2012	onlineA	0.2178
AMBER	es-en	newstest2012	onlineB	0.2224
AMBER	es-en	newstest2012	onlineC	0.1915
AMBER	es-en	newstest2012	onlineD	0.1922
AMBER	es-en	newstest2012	onlineE	0.1890
AMBER	es-en	newstest2012	onlineF	0.1822
AMBER	es-en	newstest2012	QCRI--primary	0.2186
AMBER	es-en	newstest2012	uedin-wmt12	0.2168
AMBER	es-en	newstest2012	uk-dan-moses	0.1759
AMBER	es-en	newstest2012	UPC	0.2167
BLEU_4_closest_cased	es-en	newstest2012	GTH-UPM	0.2888
BLEU_4_closest_cased	es-en	newstest2012	QCRI--primary	0.3286
BLEU_4_closest_cased	es-en	newstest2012	UPC	0.3202
BLEU_4_closest_cased	es-en	newstest2012	jhu-hiero	0.2922
BLEU_4_closest_cased	es-en	newstest2012	onlineA	0.3138
BLEU_4_closest_cased	es-en	newstest2012	onlineB	0.3753
BLEU_4_closest_cased	es-en	newstest2012	onlineC	0.2294
BLEU_4_closest_cased	es-en	newstest2012	onlineD	0.2299
BLEU_4_closest_cased	es-en	newstest2012	onlineE	0.2200
BLEU_4_closest_cased	es-en	newstest2012	onlineF	0.2171
BLEU_4_closest_cased	es-en	newstest2012	uedin-wmt12	0.3343
BLEU_4_closest_cased	es-en	newstest2012	uk-dan-moses	0.2240
posF	es-en	newstest2012	GTH-UPM	50.8772 
posF	es-en	newstest2012	jhu-hiero	50.7861 
posF	es-en	newstest2012	onlineA	52.4084 
posF	es-en	newstest2012	onlineB	52.5585 
posF	es-en	newstest2012	onlineC	48.6851 
posF	es-en	newstest2012	onlineD	48.9753 
posF	es-en	newstest2012	onlineE	47.2743 
posF	es-en	newstest2012	onlineF	47.5800 
posF	es-en	newstest2012	QCRI--primary	54.0183 
posF	es-en	newstest2012	uedin-wmt12	54.2129 
posF	es-en	newstest2012	uk-dan-moses	43.8312 
posF	es-en	newstest2012	UPC	53.6285 
-----------------------------------------------------------------------------------

Publication:

The signigicance test provided is described in detail in: 
"Testing for Significance of Increased Correlation with Human Judgment", 
Yvette Graham & Timothy Baldwin, EMNLP 2014.

-----------------------------------------------------------------------------------