Design and Implementation of Distributed Collaborative Filtering Algorithm on Hadoop
Xiao Qiang1, Zhu Qinghua1, Zheng Hua2, Wu Kewen1
1. School of Information Management, Nanjing University, Nanjing 210093, China; 2. School of Engineering Management, Nanjing University, Nanjing 210093, China
Abstract：Based on Hadoop, this paper demonstrates that traditional collaborative filtering algorithm cannot adjust to cloud computing platform, then improves traditional collaborative filtering algorithm to adapt to the Hadoop platform from similarity and prediction,and also achieves sequential modular MapReduce collaborative filtering computing tasks.
[1] 李树青.个性化信息检索技术综述[J]. 情报理论与实践,2009,32(5):107-113.(Li Shuqin. Review of Personalized Information Retrieval Technology[J].Information Studies:Theory & Application, 2009,32(5):107-113.) [2] Liu Z B,Qu W Y,Li H T,et al.A Hybrid Collaborative Filtering Recommendation Mechanism for P2P Networks[J].Future Generation Computer Systems,2010,26(8):1409-1417. [3] Pan R, Scholz M. Mind the Gaps: Weighting the Unknown in Large-Scale One-Class Collaborative Filtering[C].In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France. New York: ACM, 2009:667-676. [4] Pan R, Zhou Y H, Cao B, et al. One-Class Collaborative Filtering[C].In: Proceedings of the 8th IEEE International Conference on Data Mining, Pisa. Washington, DC, USA: IEEE Computer Society,2008:502-511. [5] Salakhutdinov R, Mnih A. Probabilistic Matrix Factorization[C].In: Proceedings of the 25th International Conference on Machine Learning. New York: ACM,2008:880-887. [6] 侯经川,方静怡.数据引证研究:进展与展望[J]. 中国图书馆学报, 2013,39(1):112-118.(Hou Jingchuan, Fang Jingyi. Review on Data Citation in the Context of Big Data[J].Journal of Library Science in China, 2013,39(1):112-118.) [7] 韩翠峰.大数据带给图书馆的影响与挑战[J]. 图书与情报,2012(5):37-40.(Han Cuifeng. The Impact and Challenges of the Library Based on Big Data[J]. Library & Information, 2012(5):37-40.) [8] Sarwar B, Karypis G, Konstan J,et al.Item-based Collaborative Filtering Recommendation Algorithms[C].In: Proceedings of the 10th International Conference on World Wide Web. New York, NY, USA: ACM,2001:285-295. [9] Resnick P,Iacovou N,Suchak M et al.GroupLens:An Open Architecture for Collaborative Filtering of Netnews[C]. In:Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work. New York, NY, USA:ACM, 1994:175-186. [10] Sarwar B,Karypis G,Konstan J,et al.Item-based Collaborative Filtering Recommendation Algorithms[C].In:Proceedings of the 10th International Conference on World Wide Web. New York, NY, USA:ACM,2001:285-295. [11] White T. Hadoop: The Definitive Guide[M].The 3rd Edition. USA: O'Reilly Media, 2012. [12] Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters[J].Communications of the ACM, 2008,51(1):107-113. [13] Hadoop.HDFS Users Guide[EB/OL].[2012-12-02]. http://hadoop.apache.org/docs/stable/hdfs_user_guide.html. [14] Bahga A, Madisetti V K.Analyzing Massive Machine Maintenance Data in a Computing Cloud[J].IEEE Transactions on Parallel and Distributed Systems,2012,23(10):1831-1843. [15] 冯璐,冷伏海.共词分析方法理论进展[J]. 中国图书馆学报,2006,32(2):88-92.(Feng Lu, Leng Fuhai. Development of Theoretical Studies of Co-word Analysis[J].Journal of Library Science in China,2006,32(2):88-92.) [16] Sarwar B, Karypis G, Konstan J,et al. Analysis of Recommendation Algorithms for E-commerce[C].In: Proceedings of the 2nd ACM Conference on Electronic Commerce. New York: ACM, 2000:158-167.
