Raw corpora for the PARSEME 1.2 shared task

This page gives links and instructions for downloading raw corpora available for the Parseme Shared Task 1.2.

There are 14 languages in the shared task. On March 22, data is ready for all languages:

Language	url	README	Size (uncompressed)	Nb of files	Nb of sentences	Nb of tokens
DE (German)	DE	DE	13 Gb	10	10,000,000	184,864,965
EL (Greek)	EL	EL	2Gb	110	1,044,268	25,559,954
EU (Basque)	EU	EU	1.3 Gb	150	1,327,630	21,268,325
FR (French)	FR	FR	48 Gb	34	33,979,081	914,824,075
GA (Irish)	GA	GA	2.1 Gb	138	1,379,824	34,211,598
HE (Hebrew)	HE	HE	941 Mb	2	451,408	12,867,409
HI (Hindi)	HI	HI	6.3 Gb	99	3,597,653	77,964,199
IT (Italian)	IT	IT	15 Gb	246	12,280,673	281,418,631
PL (Polish)	PL	PL	161 Gb	140	159,115,022	1,902,279,431
PT (Portuguese)	PT	PT	19Gb	19	26,042,884	306,603,890
RO (Romanian)	RO	RO	938 Mb	100	477,313	12,680,325
SV (Swedish)	SV	SV	176 Gb	188	163,633,807	2,474,800,065
TR (Turkish)	TR	TR	1.6 Gb	30	1,390,791	19,818,460
ZH (Chinese)	ZH	ZH	3.4 Gb	4	4,107,179	67,235,911

How data was prepared and checked

For each language, we provide a list of xz compressed files named raw-001.conllu.xz to raw-xxx.conllu.xz where xxx is the total number of files for the language (see table above).

Each conllu file should be a valid CoNLL-U file. A subset of the first 1000 sentences of most files has been validated with the UD validation script with the command:

validate.py --lang=ud --level=2 file

Level 2 validation ensures that for each sentence,

there is a '# text' line, a '# sent_id' line
the POS and DEPREL are compliant with UD 2
the syntactic annotation form a tree

NB: We prepared the data as carefully as possible given time constraints. If you identify problems in the data, please mail us as soon as possible.

Script for one language download

Replace XX by the two letter code of the language in the script below to download and decompress data for the given language.

mkdir XX
cd XX
wget http://parseme.lchn.fr/XX/files.txt
wget --base http://parseme.lchn.fr/XX/ -i files.txt
xz -d *.xz
cd ..

Script for all languages download

for LANG in DE EL EU FR GA HE HI IT PL PT RO SV TR ZH
do
  mkdir ${LANG}
  cd ${LANG}
  wget http://parseme.lchn.fr/${LANG}/files.txt
  wget --base http://parseme.lchn.fr/${LANG}/ -i files.txt
  xz -d *.xz
  cd ..
done