摘要
对信息系统运行记录、操作日志、告警信息的采集问题进行了研究,提出了一种面向泛政府行业安全运行管理平台的统一日志采集系统;采用基于消息队列的流式处理架构,实现日志采集、日志处理、日志上报等各个环节的解耦;采用标准化接口和插件技术,实现各种异构日志信息的采集和数据上报;采用消息队列的流量削峰技术,保证日志传输的安全可靠;依据日志流量特征,提出一种支持动态调整消费组的设计模式,达到超过20000条日志/每秒的高性能采集要求;整个系统由日志采集、数据上报、数据管理、系统管理、策略管理、Agent管理、日志源管理模块和日志采集代理(Agent)子系统组成,可满足对各类安全数据的集中分析、安全威胁感知和智能研判。
Aimed at the collection problems of operation record,operation log and alarm information of information systems,and a unified log collection system for the safe operation management platform for the pan-government and industry is proposed.The whole system is composed of log collection,data reporting,data management,system management,policy management,Agent management,log source management module and log collection subsystem.The flow processing architecture based on the message queue is adopted to realize the decoupling of log collection,log processing and log reporting.The standardized interfaces and plug-ins are used to collect the heterogeneous log information and data.The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of the log transmission.According to the characteristics of the log traffic,a supporting dynamic adjustment design modeling of the consumption group is proposed to meet the high collection requirements of over 20000 logs per second.which can meet the centralized analysis,security threat perception and intelligent analysis of various security data.
作者
邵旭东
樊志杰
张敬锋
曹志威
周明富
熊已兴
张林
SHAO Xudong;FAN Zhijie;ZHANG Jingfeng;CAO Zhiwei;ZHOU Mingfu;XIONG Yixing;ZHANG Lin(Department of Information Security Technology,The Third Research Institute of the Ministry of Public Security,Shanghai 200031,China;Science and Technology Information Detachment,Anhui Provincial Public Security Department,Hefei 230061,China;Research and Development Center,Shanghai Chenrui Information Technology Company,Shanghai 200031,China)
出处
《计算机测量与控制》
2023年第4期272-280,共9页
Computer Measurement &Control
基金
上海市人才发展资金资助(2020016)
中国博士后科学基金资助(2020M670998)
上海市自然科学基金资助(21ZR1422000)
公安部科技计划项目资助(2019JZX004)
四川省科技计划项目(重点研发项目)(2021YFS0310)。
关键词
安全运行管理平台
标准化接口
插件技术
消息队列
流式架构
safe operation management platform
standardized interface
plug-in technology
message queue
streaming architecture
