摘要
相似度计算是相似重复记录清理过程中的一个关键问题,编辑距离算法在其中具有广泛应用。在传统编辑距离算法的基础上,通过分析影响相似度计算结果的序列长度、同义词等因素,得到一种同时引入同义词词库和归一化处理思想的改进的基于语义编辑距离的相似重复记录清理算法,适用于相似记录的识别过程。实验分析表明,改进算法计算结果更符合句子的语义信息,绝大部分结果符合人们的认知经验,从而可以有效地提高相似重复记录识别的准确率和精确度。
Similarity calculation is a key issue in the process of approximately duplicate data cleaning, and edit distance algorithm is widely used in this application. Based on the traditional edit distance algorithm, by analyzing the sequence length, synonyms and other factors which affect the similarity of the results, an improved approximately duplicate data cleaning algorithm based on semantic edit distance is proposed. This algorithm used synonyms thesaurus and normalized distance metric, and it can be applied to similar records identification process. Experimental results show that the calculating results by this improved algorithm become more in line with the sentence semantic information and people＇ s cognitive experience. Thereby, the method effectively improves the accuracy and precision of detect approximately duplicate data
出处
《现代图书情报技术》
CSSCI
北大核心
2011年第7期82-90,共9页
New Technology of Library and Information Service
基金
国家自然科学基金资助项目"持续审计中智能数据处理及其应用框架研究"(项目编号:70972138)
湖北省教育厅人文社会科学基金项目"基于SOA和MAS的金融监管信息系统总体框架研究"(项目编号:2009b080)的研究成果之一
关键词
相似重复记录
编辑距离
语义
同义词词库
Approximately duplicate data Edit distance Semantic Synonyms thesaurus
