摘要
通过对分词歧义处理情况的分析,提出一种基于上下文的双向扫描分词算法,对分词词典进行改进,将词组短语的固定搭配引入词典中.讨论了特征项的选择及权重的设定,并引进2χ统计量参与项的权值计算,解决了目前通用TF-IDF加权法的不足,同时提出了项打分分类算法,提高了特征项对于文本分类的有效性.实验结果表明,改进后的权重计算方法性能更优越.
On the basis of the analysis of the process of dealing with the Chinese word segmentation ambiguity, this paper covers bidirectional scan word segmentation algorithm based on the context. In order to improve the word segmentation dictionary, the authors put the fixed phrase into the dictionary and discussed the feature selectionand the weighting schema enactment in detail. In order to solve the problem of general TF-IDF weighting schema at present, we took statistics into consideration, and meanwhile put up the item-scoring method which improves the efficiency of the feature item about text categorization. At last we proved the advantage of the improved weighting schema through test.
出处
《吉林大学学报（理学版）》
CAS
CSCD
北大核心
2009年第4期790-794,共5页
Journal of Jilin University:Science Edition
基金
国家自然科学基金(批准号:60275026)
"十一五"国家科技支撑计划重大项目基金(批准号:2006BAK01A33)
关键词
文本分类
上下文双向扫描
向量空间模型
权重
特征选择
text categorization
context bidirectional scan
vector space model
weighting schema
feature selection
