摘要
针对传统MapReduce算法结构在处理大数据时,负载均衡性能不理想的缺点,设计了一种具有负载均衡机制的层次MapReduce模型。该模型利用超立方拓扑结构对MapReduce的映射操作进行改进,通过特定算法将八个结构化的数据中心链接到一个对等的云环境结构中,并使用奇偶直方图组合采样方式的均衡划分方法,实现在用户请求下的节点工作负荷指数均衡。最后,基于Hadoop框架对所提算法进行仿真实验,结果显示所提算法结构相对于原始MapReduce结构,具有更高的并行计算的资源利用率,以及更佳的容错和负载均衡性能,综合性能得到有效提升。
According to the traditional MapReduce algorithm in processing large data structure had no ideal performance for fault tolerant rate and load balancing,this paper designed a hierarchical MapReduce model with load balancing mechanism. The used of hypercube topology mapping operation on the MapReduce improved the performance of the model. and through a specific algorithm,eight structured linked the data center to a peer-to-peer cloud environment structure,then used the equilibrium partitioning method with the parity histogram combined sampling method to realize the node workload balancing under the user＇s request. Finally,it made the simulation on Hadoop framework for the proposed algorithm,which shows that the proposed algorithm is more efficiency in utilization rate of parallel computing resources,and better fault tolerance and load balancing performance than the original MapReduce structure,so the improved method effectively enhances the comprehensive performance of proposed algorithm.
出处
《计算机应用研究》
CSCD
北大核心
2016年第4期1075-1078,共4页
Application Research of Computers
基金
国家自然科学基金资助项目(61462069)
