摘要
针对在线系统的动态日志中存在的概念漂移问题,提出一种结合置信度的系统日志在线异常检测模型(COP)。按照时间顺序模拟Hadoop分布式文件系统日志,使用滑动窗口以日志块的方式接收日志并进行预处理;获取一定时间内的日志作为预先知识并计算p值;使用一致性预测得分计算置信度,根据预先知识中获得的显著性水平过滤异常日志;建立当前与之前日志数据之间的联系,动态更新校准集并输出一个可信的检测结果。实验结果表明,COP与用于模拟增量学习的重训练异常检测模型相比,得到的在线检测结果和时间性能更优。
Aimed at the problem of concept drift in the dynamic log of the online system,a system log online anomaly detection model(COP) combined with confidence is proposed.We simulated the Hadoop distributed file system logs in chronological order and used the sliding window to receive the logs in log blocks and to perform preprocessing.We obtained the log within a certain period of time as advance knowledge and calculated the p-value.We used the consistency prediction score to calculate the confidence,and filtered the abnormal log based on the significance level obtained from the prior knowledge.We established the connection between the current and previous log data,dynamically updated the calibration set and output a credible detection result.Experimental results show that,compared with the retrained anomaly detection model used to simulate incremental learning,the online detection results and time performance obtained by COP are better.
作者
吕宗平
梁婷婷
顾兆军
刘春波
王双
王志
LüZongping;Liang Tingting;Gu Zhaojun;Liu Chunbo;Wang Shuang;Wang Zhi(Information Security Evaluation Center,Civil Aviation University of China,Tianjin 300300,China;Institute of Computer Science and Technology,Civil Aviation University of China,Tianjin 300300,China;College of Artificial Intelligence,Nankai University,Tianjin 300071,China)
出处
《计算机应用与软件》
北大核心
2023年第10期314-321,共8页
Computer Applications and Software
基金
国家自然科学基金项目(61601467)
民航安全能力建设基金项目(PESA2020100)
中央高校基本科研业务费项目中国民航大学专项资助(3122018D030)。
关键词
概念漂移
在线学习
一致性预测
异常检测
滑动窗口
置信度
Concept-drift
Online learning
Conformal prediction
Sliding window
Abnormal detection
Confidence level
