摘要
传统大数据平台软件历经十余年发展,已日趋成熟。近年来,以容器化为主要特征的云原生架构成为基础设施建设的首选方案。尤其在高性能计算技术融入云原生环境的趋势下,新一代大数据平台设计也面临着不同的挑战。这些挑战涉及云原生环境下的作业调度、高性能网络的容器化适配、存算分离架构下的存储管理等。针对这些问题,提出一组高性能云原生大数据平台关键技术,包括多模式负载容器化调度技术、容器化RDMA数据交换技术、云原生存算分离架构等,并在此基础上研发了OMBD大数据平台。OMBD能够有效适配高性能云原生环境的特点,以体系化的技术方案使得多模式大数据作业在带有高性能网卡的容器化集群中实现有效调度和高效执行。实验数据和真实应用效果证明,OMBD是一款具有实用性和高效性的生产级大数据平台。
After more than ten years,the traditional big data platform software has become increasingly mature.In recent years,the cloud-na⁃tive architecture featuring containerization has become the preferred solution for infrastructure construction.Especially under the trend of inte⁃grating high-performance computing technology into the cloud-native environment,the design of a new generation of big data platforms is fac⁃ing different challenges.These challenges involve job scheduling in a cloud-native environment,containerized adaptation of high-perfor⁃mance networks,and storage management under a storage-compute separation architecture.In response to these problems,this paper propos⁃es a set of key technologies for a high-performance cloud-native big data platform,including the multi-mode workload containerized schedul⁃ing technology,the containerized RDMA data exchange technology,and the cloud-native storage-compute separation architecture.And on this basis,developed the OMBD big data platform.OMBD can effectively adapt to the characteristics of high-performance cloud native envi⁃ronment,and realize effective scheduling and efficient execution of multi-mode big data jobs in containerized clusters with high-performance network cards with systematic technical solutions.Experimental data and real-world application results prove that OMBD is a practical and effi⁃cient production-grade big data platform.
作者
林健
黄林
黄进军
谢冬鸣
洪志刚
LIN Jian;HUANG Lin;HUANG Jinjun;XIE Dongming;HONG Zhigang(Oriental Mind(Wuhan)Computing Technology Co.,Ltd.,Wuhan 430200,China)
出处
《软件导刊》
2024年第3期99-106,共8页
Software Guide
基金
武汉东湖新技术开发区第十三批“3551光谷人才计划”支持项目(M165)
关键词
大数据
云原生
高性能计算
容器化
平台软件
big data
cloud-native
high-performance computing
containerization
platform software
