摘要
数据仓库中相似重复记录的清洗对于数据质量影响很大,传统的基本邻近排序算法(sorted-neighborhood method,SNM)时间效率和准确率均不高。针对SNM算法的缺陷,提出了一种基于长度过滤和动态容错的SNM改进算法。根据两条记录的长度比例和属性缺失情况,首先排除一部分不可能构成相似重复记录的数据,减少比较次数,提高检测效率;进一步提出了动态容错法,校准字段相似度评判结果,解决了因属性缺失而误判的问题,提高了准确率。针对实际数据集的实验分析表明,在相同的运算环境下,优化算法在准确率和时间效率上有明显优势。
In data warehouse systems, cleaning similar and duplicated records could effectively impact data quality. Traditional SNM（ sorted-neighborhood method） has performance issues with time efficiency and accuracy rate. In order to improve its performance, this paper proposed an enhance SNM algorithm based on length filtering and dynamic fault-tolerance （ LFSNM）. Firstly, it improved the detection efficiency by excluding the records which were impossible to be duplicated according to the length proportion and attribute absence of two records. Then, it calibrated field similarity results using dynamic fault-tolerance method. It ensured accuracy even though some attributes were absent. Experimental results indicate that the LF-SNM performs obviously better than traditional SNM method on actual datasets under the same experimental conditions.
出处
《计算机应用研究》
CSCD
北大核心
2017年第1期147-150,155,共5页
Application Research of Computers
基金
新疆维吾尔自治区青年科技创新人才培养工程基金资助项目(2014721033)
乌鲁木齐高新区发展扶持基金资助项目(2013038)
关键词
数据清洗
相似重复记录
SNM算法
动态容错
字段匹配
data cleaning
similar and duplicated records
SNM algorithm
dynamic fault-tolerance
string match
