摘要
通过实验对FIFO、CAP和FAIR三种调度算法的特点与性能进行了对比分析,得出了它们各自优点和存在的问题.针对公平调度算法Fair Scheduler不适用于内存密集型作业调度的缺点,提出一种基于内存平衡的公平调度算法FMScheduler,在整个调度的过程中考虑作业的内存使用和计算节点的内存情况,通过加入内存比较机制、调整作业公平权重计算方法以及引入作业预留机制,对原有Hadoop公平调度算法进行改进与优化.最后,通过仿真实验对FMScheduler进行测试分析,实验结果表明,FMScheduler在高内存作业调度环境下的独立响应时间和作业整体的平均响应时间都比Fair Scheduler有所减少;并且在多用户多作业且包含内存密集型作业的环境中,FMScheduler与Hadoop原有的三种调度算法相比,在处理数据密集型作业和内存密集型作业的混合场景时,能够更合理公平地调度作业.
Abstract： The characteristics and performance of three Hadoop scheduling algorithms, i. e. FIFO, CAP and FAIR, are compared and analyzed through experiment, and both their pros and cons are educed too. Targeting on solving the issue that Fair Scheduler is not suitable for memory-intensive jobs, a novel FMScheduleris proposed on the basis of memory balance. Taking into account the memory usage and the memory of TaskTrackers,FMSchedulerintents to improve and optimize the original Hadoop scheduUng algorithms by mixing memory comparison mechanism, adjusting the calculation method offair weight and introducing job reservation mechanism. Finally,the experimental results demonstrate that compared with existing scheduling algorithms in Hadoop FMScheduler is most suitable and optimal for scheduling in the scene with multi-user, multi-job and containing memory-intensive jobs, because FMScheduler gains the least independent and total average response time of jobs,improves the resource utilization and ensures the memory-intensive jobs more fair execution opportunities.
出处
《小型微型计算机系统》
CSCD
北大核心
2014年第12期2708-2712,共5页
Journal of Chinese Computer Systems
基金
国家自然科学基金项目(71161007)资助
海南省国际科技合作专项项目(KJHZ2014-16)资助
海南省重点科技计划项目(ZDXM20130078)资助
