摘要
网络论坛中蕴涵着大量具有实用价值和商业价值的信息,是搜索引擎和问答系统信息的重要来源。针对论坛结构复杂、链接种类繁多,以及容易陷入采集陷阱等问题,提出了一种基于结构驱动的采集路径选择方法。首先根据用户标注的少量类型数据,利用DOM树对采样网页基于网页结构进行结构聚类;其次根据各节点的评价进行采集路径选择;最后对翻页链接进行有效的识别和处理。实验表明,该方法采集的覆盖率和有效率明显优于传统算法,并且应用在中国科学院计算所舆情监测平台上取得了良好的效果。
Forums contain much practical and business information,which is the important source of information for search engines and question answering system.Complex structure of the forums,a great variety of links and the issues that being easy to fall into the trap of crawling are all the problems when collect information.This paper proposed a crawling method based on structure-driven path selection to solve these problems.First,used a small number of types of data marked by the users,and used DOM tree to cluster by structure based on Web-based structure.And then,chose the collected route according to the evaluation of each node,at last identified and processed the link to the page effectively.Experiments show that the coverage and efficiency of collection is better than the traditional algorithm.And get good results through the golaxy public opinion monitoring system of ICT.
出处
《计算机应用研究》
CSCD
北大核心
2011年第9期3284-3287,共4页
Application Research of Computers
基金
国家自然科学基金资助项目(60873166)
国家教育部科学技术研究重点资助项目(109028)
北京市教育科学基金资助项目(AHA09110)
关键词
信息检索
论坛采集
结构驱动
聚类
路径选择
information retrieval
forum crawling
structure-driven
clustering
traversal path selection
