摘要
新的科研和工程活动正在以越来越快的速度产出海量并且多样化的数据,这些数据带来巨大的计算需求并且迅速增长,数据传输时间占作业处理时间的比例越来越大。本文面向中国科学院科学计算环境,针对计算环境中几百MB到GB量级的数据文件传输耗时长、网络不稳定和数据共享等问题,设计和实现基于微服务架构的文件传输和共享服务,提供一组简单易用的REST开发接口,包括文件传输、管理、共享、统计等服务。该服务的目标是提升作业数据文件的传输效率和在线分享能力,为多学科领域的开发人员提供简单易用的数据开发服务,为用户提供简单灵活的数据传输和共享服务。理论分析和实验表明,该服务简单易用并具备良好的可扩展性,能够根据负载变化而动态扩展文件传输微服务。
New research and engineering activities are generating diverse data at a faster rate than before.These data brings massive computing demands and grows rapidly.Compared to whole time of a job,the proportion of time spent on transferring files is getting bigger.Based on Scientific Computing Environment (SCE)of Chinese Academy of Sciences,we designed and implemented file transfer and sharing services atop micro- service architecture to solve transfer time-consuming,networks instability,data sharing and other issuesfor files whose sizes are from hundreds of MB to several GB.In addition,a set of REST API composing of file transfer,management,sharing and statistics services are provided.Our goal is to promote file transfer and data online sharing capabilities in SCE,provide easy-to-use development services for developers in diverse research areas and disciplines,and provide simple and flexible file transfer and sharing services for computing users.Our analysis and experiments show that services proposed in this paper are easy-to- use for developers and users,and has good scalability that the file transfer micro-service could be extended dynamically taking accounts of load change.
作者
曹荣强
肖海力
王小宁
卢莎莎
刘芳
王彦棡
Cao Rongqiang;Xiao Haili;Wang Xiaoning;Lu Shasha;Liu Fang;Wang Yangang(Computer Network Information Center,Chinese Academy of Sciences,BeO'ing 100190,China)
出处
《科研信息化技术与应用》
2018年第3期26-33,共8页
E-science Technology & Application
基金
国家重点研发计划高性能计算专项课题"大规模并行计算的工具库和领域相关基础软件包"(2017YFB0202202)
国家自然基金委青年基金项目(61702476)
关键词
高性能计算
数据传输
数据共享
REST接口
微服务
high performance computing
data transfer
data sharing
RESTful API
micro-service
