摘要
传统的判断2个文档相似性的方法没有考虑到文本背后的语义关联,导致检索系统返回的结果与用户的查询需求之间存在很大的差异。本文提出一种基于LDA主题模型的文本聚类方法,首先介绍LDA主题模型的应用原理,阐述文本挖掘的基本方法,之后构建LDA主题模型,采用Gibbs抽样的方法进行推导,得到特征词的概率分布,最后用优化聚类中心选择的K-means++方法对测试数据集合聚类,并把设计的LDA-Gibbs模型与传统的TF-IDF模型进行聚类评价对比。实验结果表明,该模型能够提高数据的检索效果,具有良好的推广价值。
The traditional method of judging the similarity of two documents does not take into account the semantic relation behind the texts,resulting in a large difference between the results returned by the retrieval system and the user＇s query requirements. This paper presents a text clustering method based on LDA topic model. Firstly,the application principle of LDA topic model is introduced and the basic method of text mining is expounded,and then the LDA topic model is constructed. The Gibbs sampling method is used to derive the probability distribution of the characteristic words. Finally,the sets of test data are clustered with the K-means ＋ ＋ method chosen by the optimization cluster center. And the designed LDA-Gibbs model is compared with the traditional TF-IDF model. Experimental results show that this model can improve the retrieval effect of data and has good promotional value.
作者
李霄野
李春生
李龙
张可佳
LI Xiao-ye;LI Chun-sheng;LI Long;ZHANG Ke-jia(School of Computer and Information Technology,Northeast Petroleum University,Daqing 166618,China)
出处
《计算机与现代化》
2018年第6期7-11,共5页
Computer and Modernization
基金
黑龙江省教育规划重大课题(GJ20170006)
关键词
主题模型
文本聚类
潜在狄利克雷分配模型
聚类评价
信息检索
topic model
text clustering
latent Dirichlet allocation（LDA）
cluster evaluation
information retrieval（IR）
