Constructing two vietnamese corpora and building a lexical database

Abstract

Corpus-based research has formed the backbone of linguistic research in recent decades. Large text corpora are used for solving various kinds of linguistic problems, including those of quantitative linguistics, cognitive linguistics, and psycholinguistics. This paper reports the creation of two corpora of contemporary Vietnamese. It also describes the construction of these two equally sized Vietnamese corpora (a corpus from Vietnamese film subtitles, subtlex-viet, and a general corpus of varieties of online newspapers and stories, genlex-viet). We document the general steps of the construction and extraction of linguistic information from the language corpora and provide a road map for others who would like to create similar corpora. The resultant corpora are available in three versions: plain text, tokenized, and POS tagged. In the second half of the paper, the construction of a lexical database derived from the corpora is described. The database includes measures such as frequency of occurrence, dispersion, Mutual Information, Inverse Document Frequency, as well as vector space measures based on Latent Semantic Analysis and Hyperspace Analogue to Language. We conclude by reporting a comparison of the lexical predictors and a validation using psycholinguistic data from visual lexical decision experiments.

Appendix

POS tags used in the corpora

ID	POS-tags	POS in English	POS in Vietnamese
1	Np	Proper noun	danh từ riêng
2	Nc	Classifier noun	danh từ chỉ loại
3	Nu	Unit noun	danh từ đơn vị
4	N	Common noun	danh từ chung
5	V	Verb	động từ
6	A	Adjective	tính từ
7	P	Pronoun	đại từ
8	R	Adverb	phó từ
9	L	Determiner	định từ
10	M	Numeral	số từ
11	E	Preposition	giới từ
12	C	Subordinating conjunction	liên từ phụ
13	CC	Coordinating conjunction	liên từ kết hợp
14	I	Interjection	từ cảm thán
15	T	Auxiliary word, modal words	trợ từ
16	Y	Abbreviation	từ viết tắt
17	Z	Bound morphemes	yếu tố cấu tạo từ (bất, vô. . .)
18	X	Undetermined	không (hoặc chưa) xác định

POS-tagged XML sample

					<s>
<doc>						<w	pos=“A”>lớn</w>
…						<w	pos=“R”>lên</w>
<s>						<w	pos=“CC”>và</w>
		<w	pos=“P”>Đây</w>			<w	pos=“V”>tin</w>
		<w	pos=“V”>là</w>			<w	pos=“C”>rằng</w>
		<w	pos=“N”>câu_chuyện</w>			<w	pos=“M”>…</w>
		<w	pos=“E”>về</w>		</s>
		<w	pos=“M”>một</w>		<s>
		<w	pos=“N”>chàng_trai</w>			<w	pos=“P”>mình</w>
		<w	pos=“V”>gặp</w>			<w	pos=“R”>sẽ</w>
		<w	pos=“M”>một</w>			<w	pos=“V”>không_bao_giờ</w>
		<w	pos=“N”>cô_gái</w>			<w	pos=“V”>có</w>
		<w	pos=“.”>.</w>			<w	pos=“R”>được</w>
	</s>					<w	pos=“N”>hạnh_phúc</w>
	<s>					<w	pos=“A”>thực_sự</w>
		<w	pos=“N”>Ngày</w>			<w	pos=“…”>…</w>
		<w	pos=“N”>thứ</w>		</s>
		<w	pos=“M”>nhất</w>		<s>
	</s>					<w	pos=“E”>cho_đến</w>
	<s>					<w	pos=“N”>ngày</w>
		<w	pos=“N”>Chàng_trai</w>			<w	pos=“V”>gặp</w>
		<w	pos=“,”>,</w>			<w	pos=“R”>được</w>
		<w	pos=“Np”>Tom_Hansen</w>			<w	pos=“““>“</w>
		<w	pos=“,”>,</w>			<w	pos=“N”>người</w>
		<w	pos=“V”>sinh_ra</w>			<w	pos=“P”>ấy</w>
		<w	pos=“E”>ở</w>			<w	pos=“““>“</w>
		<w	pos=“Np”>Margate</w>			<w	pos=“.”>.</w>
		<w	pos=“,”>,</w>		</s>
		<w	pos=“Np”>New_Jersey</w>	…
		<w	pos=“,”>,</w>	</doc>
	</s>

Dispersion measures

Word	FREQ	RANGE	MAXMIN	SD	VARCOEFF	CHISQUARE	D_EQ	D_UNEQ
cảnh gần	8.00	8.00	1.00	0.01	147.34	220,704.86	0.65	0.53
cánh phấn	2.00	2.00	1.00	0.00	294.69	15,572.46	0.29	0.23
cảnh sát	25,695.00	12,434.00	46.00	0.76	5.16	869,451.64	0.99	0.99
cao lương	100.00	49.00	23.00	0.08	135.55	575,687.79	0.67	0.75
cặp lồng	30.00	23.00	3.00	0.02	94.21	191,176.18	0.77	0.64
cạp nia	9.00	7.00	3.00	0.01	179.34	137,331.11	0.57	0.51

D2	S_EQ	S_UNEQ	D3	DC	IDF	ENGVALL	U_EQ	U_UNEQ	UM_CARR
0.17	0.00	0.00	− 5426.44	0.00	14.41	0.00	5.17	4.28	1.38
0.06	0.00	0.00	− 21,709.50	0.00	16.41	0.00	0.59	0.46	0.11
0.76	0.06	0.05	− 5.66	0.06	3.80	1839.48	25,376.75	25,394.28	19,446.87
0.26	0.00	0.00	− 4592.74	0.00	11.79	0.03	67.47	74.61	25.61
0.25	0.00	0.00	− 2218.07	0.00	12.88	0.00	23.22	19.08	7.61
0.15	0.00	0.00	− 8039.77	0.00	14.60	0.00	5.13	4.63	1.37

AF_EQ	AF_UNEQ	Ur_KROM	F_ARF	AWT	F_AWT	ALD	F_ALD	DP	DPnorm
0.00	0.00	8.00	5.63	7,427,879.64	5.72	7.13	6.27	1.00	1.00
0.00	0.00	2.00	1.20	3,498,9851.57	1.21	7.79	1.38	1.00	1.00
1606.13	1393.76	17,028.34	9384.85	20,117.13	2111.81	4.15	5980.62	0.93	0.93
0.02	0.08	58.00	34.54	2,075,863.59	20.46	6.49	27.38	1.00	1.00
0.00	0.02	26.33	15.24	3,737,474.52	11.37	6.77	14.35	1.00	1.00
0.00	0.00	7.83	4.32	11,980,483.74	3.55	7.30	4.29	1.00	1.00

Abbreviation	Measure
FREQ	Observed frequency of word w
RANGE	Number of parts with word w
MAXMIN	Max. freq. of w/part—min. freq. of w/part
SD	Standard deviation of frequencies
VARCOEFF	Variation coefficient of frequencies
CHISQUARE	Chi square value of the frequency distribution
D_EQ	Juilland et al.’s D (assuming equal parts)
D_UNEQ	Juilland et al.’s D (not assuming equal parts)
D2	Carroll’s D2
S_EQ	Rosengren’s S (assuming equal parts)
S_UNEQ	Rosengren’s S (not assuming equal parts)
D3	Lyne’s D3
DC	Distributional Consistency
IDF	Inverse Document Frequency
ENGVALL	Engvall’s measure
U_EQ	Juilland et al.’s usage coefficient U (assuming equal parts)
U_UNEQ	Juilland et al.’s usage coefficient U (not assuming equal parts)
UM_CARR	Carroll’s Um
AF_EQ	Rosengren’s Adjusted Frequency AF (assuming equal parts)
AF_UNEQ	Rosengren’s Adjusted Frequency AF (not assuming equal parts)
Ur_KROM	Kromer’s UR
F_ARF	Savický and Hlaváčová’s fARF
AWT	Savický and Hlaváˇcová’s AW T
F_AWT	Savický and Hlaváˇcová’s fAW T
ALD	Savický and Hlaváˇcová’s
ALD F_ALD	Savický and Hlaváˇcová’s fALD
SELF_DISP	Washtell’s self-dispersion
DP	Gries’s Deviation of Proportions
DP_norm	Gries’s Deviation of Proportions (normalized)