[Objective] To implement the textual clustering and classification in distributed environment through open-source tools. [Methods] According to the convergence of words in masses of text, this paper classifies texts based on word-clustering, including text preprocess by open-source tokenizer, cluster analysis by Mahout, classifying the test text by computing the similarity between the text and word-cluster. [Results] The textual clustering based on word-clustering in distributed environment effectively solves the bottleneck of word-clustering of massive texts. The tested result of word-clustering is ideal while the number of text training set exceeds 100 and the iterative convergence threshold is 0.01. [Limitations] The data type is limited in the field of news and the other field-based word-clustering also needs further test, optimization and adjustment. [Conclusions] This study describes the build process and key steps of the textual clustering and classification in distributed environment to help readers with in-depth understood.
赵华茗. 分布式环境下的文本聚类研究与实现[J]. 现代图书情报技术, 2015, 31(1): 82-88.
Zhao Huaming. Research and Implementation of Textual Clustering in Distributed Environment. New Technology of Library and Information Service, 2015, 31(1): 82-88.
[1] 胡建军, 唐常杰, 李川, 等. 基于最近邻优先的高效聚类算法 [J]. 四川大学学报: 工程科学版, 2004, 36(6): 93-99. (Hu jianjun, Tang Changjie, Li Chuan, et al. An Efficient Multi-layer Clustering Algorithm Based on Nearest Neighbors First [J]. Journal of Sichuan University: Engineering Science Edition, 2004, 36(6): 93-99.)
[2] Han J, Kamber M. Data Mining Concepts and Techniques [M]. Beijing: China Machine Press, 2008: 261-284.
[3] Pena J M, Lozano J A, Larranaga P. An Empirical Comparison of Four Initialization Methods for the K-means Algorithm [J]. Pattern Recognition Letters, 1999, 20(10): 1027-1040.
[4] Bradley P S, Fayyad U M. Refining Initial Points for K-means Clustering [C]. In: Proceedings of the 15th International Conference on Machine Learning (ICML'98). San Francisco, USA: Morgan Kaufmann Publishers Inc., 1998: 91-99.
[5] Steinbach M, Karypis G, Kumar V. A Comparison of Document Clustering Techniques [C]. In: Proceedings of KDD 2000 Workshop on Text Mining. 2000: 1-20.
[6] Zhao Y, Karypis G, Fayyad U. Hierarchical Clustering Algorithms for Document Datasets [J]. Data Mining and Knowledge Discovery, 2005, 10(2): 141-168.
[7] Higgs R E, Bemis K G, Watson I A, et al. Experimental Designs for Selecting Molecules from Large Chemical Databases [J]. Journal of Chemical Information and Computer Sciences, 1997, 37(5): 861-870.
[8] Snarey M, Terrett N K, Willet P, et al. Comparison of Algorithms for Dissimilarity-based Compound Selection [J]. Journal of Molecular Graphics ＆ Modelling, 1997, 15(6): 372-385.
[9] Slonim N, Tishby N. Document Clustering Using Word Clusters via the Information Bottleneck Method [C]. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR'00). New York, USA: ACM, 2000: 208-215.
[10] 梁维铿. 基于Hadoop的分布式文本聚类研究[D]. 广州: 华南理工大学, 2011. (Liang Weikeng. Research of Distributed Text Clustering Basic on Hadoop [D]. Guangzhou: South China University of Technology, 2011.)
[11] MapReduce [EB/OL]. [2014-08-06]. http://Hadoop.apache. org/mapreduce/.
[12] Mahout [EB/OL]. [2014-08-06]. http: //mahout.apache.org/.
[13] Hadoop [EB/OL]. [2014-08-06]. http: //hadoop.apache.org/.
[14] Salton G, Wong A, Yang C S. A Vector Space Model for Automatic Indexing [J]. Communications of the ACM, 1975, 18(11): 613-620.
[15] Bezdek J C. Pattern Recognition with Fuzzy Objective Function Algorithms [M]. Springer US, 1981.
[16] Salton G. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer [M]. Boston: Addison-Wesley Longman Publishing Co., Inc., 1989.
[17] 田润涛, 谢培山. 色谱指纹图谱相似度评价方法的规范化研究(一) [J]. 中药新药与临床药理, 2006, 17(1): 40-42. (Tian Runtao, Xie Peishan. Study on the Standardization of Similarity Evaluation Method of Chromatographic Fingerprints (Part I) [J]. Traditional Chinese Drug Research & Clinical Pharmacology, 2006, 17(1): 40-42.)
[18] Pavlo A, Paulson E, Rasin A, et al. A Comparison of Approaches to Large-scale Data Analysis [C]. In: Proceedings of the 35th SIGMOD International Conference on Management of Data (SIGMOD'09). New York, USA: ACM, 2009: 165-178.
[19] Postgresql [EB/OL]. [2014-08-06]. http://www.postgresql.org/.
[20] Eclipse [EB/OL]. [2014-08-06]. http://www.eclipse.org/.
[21] Apache Tomcat [EB/OL]. [2014-08-06]. http://tomcat.apache. org/.
[22] Joachims T. A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization [C]. In: Proceedings of the 14th International Conference on Machine Learning (ICML'97). San Francisco, USA: Morgan Kaufmann Publishers Inc., 1997: 143-151.
