Please wait a minute...
New Technology of Library and Information Service  2009, Vol. 25 Issue (12): 47-51    DOI: 10.11925/infotech.1003-3513.2009.12.09
article Current Issue | Archive | Adv Search |
Automatic Building of Sentence-Level English-Chinese Parallel Corpus
Wang Dongbo   Su Xinning
(Department of Information Management, Nanjing University, Nanjing 210093, China)
Download:
Export: BibTeX | EndNote (RIS)      
Abstract  

This article gives an account of the steps of how to automatically build a large-scale sentence-level English-Chinese parallel corpus based on websites. Specifically speaking, the following questions are addressed: the criterions which are used to grab websites are set and words library is worked out; the websites are automatically grabbed by making use of the tool ‘Wget’; the English-Chinese parallel sentences extracted from websites are subsequently processed and the Chinese sentences are segmented based on Conditional Random Field. Finally, the building of English-Chinese parallel corpus is completed which includes 1 017 963 English-Chinese parallel sentences stored in database which are automatically extracted from 675 308 websites.
Key words： English-Chinese parallel corpus      Wget      Words library      Conditional random field     
Received: 30 November 2009      Published: 25 December 2009
ZTFLH: 

TP391

 		 
Corresponding Authors: Wang Dongbo     E-mail: jisuanyuyan@163.com
About author:: Wang Dongbo,Su Xinning
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
YU Dong-Bei
SU Xin-Ning

Cite this article:

Wang Dongbo,Su Xinning. Automatic Building of Sentence-Level English-Chinese Parallel Corpus. New Technology of Library and Information Service, 2009, 25(12): 47-51.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.1003-3513.2009.12.09     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2009/V25/I12/47

1］ 王克非.双语对应语料库研制与应用［M］.北京：外语教学与研究出版社,2004：232-233．
［2］ 程岚岚.基于正则表达式的大规模网页术语对抽取研究［J］.情报杂志,2008,27(11)：62-63．
［3］ Zhang  Y， Vines  P.Using the Web for Automated Translation Extraction in Cross-language Information Retrieval［C］. In: Proceedings of SIGIR. Sheffield: University of Sheffield, 2004: l62-167.
［4］ Huang F,Zhang Y,Vogel S. Mining Key Phrase Translations from Web Corpora［C］In:Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing,Vancouver, British Columbia, Canada. Morristown, NJ, USA: Association for Computational Linguistics,  2005:483 - 490.
［5］ 张永臣,孙乐,李飞，等.基于Web数据的特定领域双语词典抽取［J］.中文信息学报,2006,20(2)：16-23．
［6］ 王丽,王同顺.中国英语学习者语用标记语习得研究——一项基于SECCL和BNC的实证研究［J］.现代外语,2008,31(3)：294.
［7］ Wget Manual［EB/OL］.［2009-12-06］.http://www.gnu.org/software/wget/manual/wget.html.
［8］ Ma X, Liberman M. BITS:A Method for Bilingual Text Search over the Web［C］. In: Proceedings of Machine Translation Summit VII. Singapore: National University of Singapore,1999.
［9］ 章成敏,许鑫,章成志.条件随机场标引模型的性能影响因素分析［J］.现代图书情报技术,2008 (6):34-40.
［10］ 李双龙,刘群.基于条件随机场的汉语分词系统［J］.软件天地,2006（10）：178-179.
［11］ The Features of CRF++［EB/OL］.［2009-12-06］. http://crfpp.sourceforge.net/#features.
［12］ Definition of  MySQL［EB/OL］.［2009-12-06］. http://en.wikipedia.org/wiki/MySQL.
[1] Cheng Bin,Shi Shuicai,Du Yuncheng,Xiao Shibin. Keyword Extraction for Journals Based on Part-of-Speech and BiLSTM-CRF Combined Model[J]. 数据分析与知识发现, 2021, 5(3): 101-108.
[2] Zhao Ping,Sun Lianying,Tu Shuai,Bian Jianling,Wan Ying. Identifying Scenic Spot Entities Based on Improved Knowledge Transfer[J]. 数据分析与知识发现, 2020, 4(5): 118-126.
[3] Li Chengliang,Zhao Zhongying,Li Chao,Qi Liang,Wen Yan. Extracting Product Properties with Dependency Relationship Embedding and Conditional Random Field[J]. 数据分析与知识发现, 2020, 4(5): 54-65.
[4] Han Huang,Hongyu Wang,Xiaoguang Wang. Automatic Recognizing Legal Terminologies with Active Learning and Conditional Random Field Model[J]. 数据分析与知识发现, 2019, 3(6): 66-74.
[5] Tang Huihui,Wang Hao,Zhang Zixuan,Wang Xueying. Extracting Names of Historical Events Based on Chinese Character Tags[J]. 数据分析与知识发现, 2018, 2(7): 89-100.
[6] Wang Xiaoyu,Li Bin. Automatically Segmenting Middle Ancient Chinese Words with CRFs[J]. 数据分析与知识发现, 2017, 1(5): 62-70.
[7] Wang Dongbo,Wu Yi,Ye Wenhao,Liu Ruilun. Extracting Events of Food Safety Emergencies with Characteristics Knowledge[J]. 数据分析与知识发现, 2017, 1(3): 54-61.
[8] Zhang Yue,Wang Dongbo,Zhu Danhao. Segmenting Chinese Words from Food Safety Emergencies[J]. 数据分析与知识发现, 2017, 1(2): 64-72.
[9] Zhang Lin,Qin Ce,Ye Wenhao. Automatic Recognition of Legal Language Entities Based on Conditional Random Fields[J]. 数据分析与知识发现, 2017, 1(11): 46-52.
[10] He Huixin,Liu Lijuan. A Scientific Research Object Labeling System Based on Active earning[J]. 现代图书情报技术, 2016, 32(3): 67-73.
[11] Jiang Chuntao. Automatic Annotation of Bibliographical References in Chinese Patent Documents[J]. 现代图书情报技术, 2015, 31(10): 81-87.
[12] He Yu, Lv Xueqiang, Xu Liping. A Chinese Term Extraction System in New Energy Vehicles Domain[J]. 现代图书情报技术, 2015, 31(10): 88-94.
[13] Zeng Zhen, Lv Xueqiang, Li Zhuo. The Automatic Identification of Chinese Names in Query Logs[J]. 现代图书情报技术, 2014, 30(12): 71-77.
[14] Tang Yafen. Research of Automatically Recognizing Name in Pre-Qin Ancient Chinese Classics[J]. 现代图书情报技术, 2013, 29(7/8): 63-68.
[15] Xiong Wenxin. Sentence Alignment and Re-Alignment for Environmental Protection Texts in English-Chinese Parallel Corpus[J]. 现代图书情报技术, 2013, (6): 36-41.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn