期刊文献+

Hadoop云平台下基于资源预估的作业调度算法 被引量:4

Job scheduling algorithm based on data-aware in Hadoop
下载PDF
导出
摘要 为解决Hadoop云平台下作业无法满足时间约束的问题,提出一种基于资源预估的作业调度算法。通过建立资源预估模型计算作业所需资源,结合作业间的资源竞争关系对完成时间进行判定,最后根据作业的数据本地性改进延迟调度策略。实验结果表明,该算法能够满足作业对时间约束的需求,提升系统的资源利用率。 In order to satisfy the deadline constraints in Hadoop, this paper proposed a job scheduling algorithm based on data- aware. This algorithm proposed a resource estimation model and computed job' s deadline, then optimized the delay strategy ac- cording to resource locality. Through analyzing the result of experiments, this algorithm can satisfy the deadline constraints and improve the efficient in Haoop.
出处 《计算机应用研究》 CSCD 北大核心 2016年第8期2311-2314,共4页 Application Research of Computers
基金 四川省科技厅基金资助项目(2012FZ0081)
关键词 HADOOP 云平台 时间约束 资源预估 作业调度 Hadoop cloud computing deadline constraints data-aware job scheduling
  • 相关文献

参考文献15

  • 1Kambatla K, Pathak A, Pucha H. Towards optimizing Hadoop provisio- ning in the cloud [ C ]// Proc of the 1 st Workshop on Hot Topics in Cloud Computing. Berkeley : USENIX Association ,2009. 被引量:1
  • 2林闯,苏文博,孟坤,刘渠,刘卫东.云计算安全:架构、机制与模型评价[J].计算机学报,2013,36(9):1765-1784. 被引量:321
  • 3孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169. 被引量:2378
  • 4Chang H, Kodialam M, Kompella R R, et al. Scheduling in mapreduce- like systems for fast completion time [ C ]//Proc of IEEE INFOCOM. 2011:3074-3082. 被引量:1
  • 5栾亚建,黄翀民,龚高晟,赵铁柱.Hadoop平台的性能优化研究[J].计算机工程,2010,36(14):262-263. 被引量:51
  • 6Lee D, Kim J S, Maeng S. Large-scale incremental processing with MapReduce[ J]. Future Generation Computer Systems,2013,36 (7) :66-79. 被引量:1
  • 7Tang Huo,Zhou Junqing,Li Kenli, et al. A MapReduce task schedu- ling algorithm for deadline constraints [ J ]. Cluster Computing, 2013,16(4) :651-662. 被引量:1
  • 8Dai Wei, Bassiouni M. An improved task assignment scheme for Ha- doop running in the clouds [ J]. Journal of Cloud Computing Ad- vances Systems & Applications ,2013,2 ( 1 ) :1-16. 被引量:1
  • 9Ezerra A, Herundez P, Espinosa A, et al. Job scheduling for optimizing data locality in Hadoop clusters [ C ]//Proc of the 20th European MPI Use Group Meeting. New York: ACM Press,2013:271-276. 被引量:1
  • 10Guo Zhenhua, Fox G, Zhou Mo, et al. Improving resource utilization in MapReduce [ C ]//Proc of IEEE International Conference on Cluster Computing. [ S. 1. ] :IEEE Press,2012:402-410. 被引量:1

二级参考文献311

  • 1林闯,汪洋,李泉林.网络安全的随机模型方法与评价技术[J].计算机学报,2005,28(12):1943-1956. 被引量:92
  • 2樊亚军,刘久文.TPM安全芯片设计与实现[J].信息安全与通信保密,2007,29(6):136-137. 被引量:5
  • 3张旻晋 桂文明 苏递生 等.从终端到网络的可信计算技术.信息技术快报,2006,4(2):21-34. 被引量:7
  • 4Dean J,Ghemawat S.MapReduce:Simplified Data Processing on Large Cluster[C] //Proc.of OSDI'04.Boston,MA,USA:[s.n.] ,2004. 被引量:1
  • 5Hadoop Distributed Filesystem[EB/OL].(2008-12-13).http://hadoop.apache.org/hdfs/. 被引量:1
  • 6IBM Research.Cloud Analytics:Do We Really Need to Reinvent the Storage Stack?[Z].2009. 被引量:1
  • 7Apache Hadoop[EB/OL].(2009-09-12).http://hadoop.apache.org/. 被引量:1
  • 8DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[ J]. Communication ACM ,2008,51 (1) :107-113. 被引量:1
  • 9ISARD M, BUDIU M, YU Yuan, et al. Dryad: distributed data-parallel programs from sequential building blocks [ C ]//Proc of ACM SIGOPS/EuroSys European Conference on Computer Systems. New York: ACM Press,2007:59-72. 被引量:1
  • 10Hadoop [ EB/OL ]. (2011 - 12-18 ) [ 2012- 03-12 ]. http ://hadoop. apache. org. 被引量:1

共引文献2769

同被引文献30

引证文献4

二级引证文献16

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部