期刊文献+

配电网监测数据的分布式Map压缩-查询技术 被引量:4

Distributed Map compression-query technology for distribution network monitoring data
下载PDF
导出
摘要 针对配电自动化中大量电力监测数据的处理问题,提出了回避聚合操作的配电网监测数据分布式Map压缩-查询新方法。通过将监测数据分布式Map压缩存储,利用HQL查询引擎及压缩接口将分布式Map压缩应用到连接查询的混洗阶段中,减小传递到查询聚合端的数据量,提高压缩数据的查询速度,并推导了时效性的相关公式。以北京某动车段10 kV电力远动监控系统的实测数据为例,搭建了四节点测试集群。压缩导入对比测试表明,分布式Map压缩速度快于分布式Reduce压缩,分布式Map的Map_Deflate压缩处理时间比分布式Reduce_Deflate减少了45.3%;压缩-查询测试表明,当数据量为2×10~7记录级时,分布式Map的Map_LZO格式压缩-查询耗时大幅降低,比混洗阶段不压缩-查询时减少了31.6%,验证了分布式Map压缩对加速查询的时效性。 In view of the considerable quantity of monitoring data in distribution automation,an innovative method of distributed Map compression-query for distribution network monitoring data is proposed,in which the aggregation operation can be avoided. The monitoring data is compressed and stored through distributed Map compression. Then,the distributed Map compression is applied to the shuffle stage of join query by HQL query engine and compression interface,which reduces the data quantity transferred to the reducers,and improves the query speed of the compressed data. Besides,the formulas of time effectiveness are deduced.The measured data of a Beijing EMU(Electric Multiple Unit) depot 10 kV power remote monitoring system is taken as an example,where a four-node cluster is constructed to carry out experiments. The compression importing comparison test results demonstrate that the distributed Map compression speed is faster than the distributed Reduce compression,and the Map_Deflate compression processing time decreases by 45.3 %compared to distributed Reduce_Deflate. Meanwhile,the compression-query test results demonstrate that the compression-query time of Map_LZO greatly reduces when the amount of data is 2 × 10~7 records,which decreases by 31.6 % compared to uncompressed-query in shuffle stage. Hence,these results verify the efficiency of distributed Map compression for accelerating query.
出处 《电力自动化设备》 EI CSCD 北大核心 2017年第12期192-199,共8页 Electric Power Automation Equipment
基金 国家自然科学基金资助项目(51567008) 江西省杰出青年人才计划项目(20162BCB23045) 江西省自然科学基金资助项目(20161BAB206156 20171BAB206044)~~
关键词 配电网 大数据 集群平台 分布式压缩 压缩-查询 distribution network massive data cluster platform distributed compression compression-query
  • 相关文献

参考文献15

二级参考文献261

共引文献270

同被引文献53

引证文献4

二级引证文献26

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部