摘要
随着数值模式时空分辨率的提升,数据量急剧增加,长序列数据很难直接通过文件拷贝或者网络传输方式为用户提供数据服务。为此,笔者设计实现了一种分布式管理平台,该平台根据用户定制的数据需求,运用预报要素、空间范围、时间尺度等约束条件,抽取或根据区域参数裁剪指定气象要素,生成精简数据进行用户服务。该平台集成了搜索引擎、格点数据解码、内存数据库技术以及分布式框架,实现跨操作系统的统一接口调用和数据快速获取,有效解决用户访问长时间序列历史资料的难题。实验测试显示,该平台在格点数据管理规模和访问效率方面均表现出色。特别是在北京2022年冬季奥运会和冬残奥会气象保障服务中,该平台发挥了重要作用,展现了其实际应用的价值和潜力。
With the rapid development of numerical weather prediction services,the resolution and forecasting lead time of meteorological models have significantly improved,leading to an exponential growth in the volume of forecast data output.As a national meteorological model research and operational centre,CMA Earth System Modeling and Prediction Center(CEMC)currently produces daily gridded data outputs of 0.76 TB,with an annual output reaching 155.12 TB.Given the enormous data volumes,researchers’preferences for data access are evolving.Wagemann predicts that future scientific users increasingly prefer cloud platforms or other interfaces for data access rather than solely relying on downloads.To address these issues,this paper proposes a lightweight distributed parallel processing framework for gridded data management,aiming to streamline data management processes and enhance data access speed.The core design philosophy revolves around leveraging search engine technology for rapid metadata retrieval and gridded data decoding techniques for efficient data acquisition.To mitigate performance penalties from repetitive decoding,the framework decodes gridded data files once and supports multiple retrievals and extractions,significantly accelerating data access.Additionally,it supports cross-platform data access,facilitating easier data acquisition for researchers.The framework adopts a three-tier architecture:the data layer stores data,the algorithm layer implements core search and cataloguing algorithms,and the business layer interfaces directly with user needs.The framework implements crucial functions such as gridded data cataloguing,extraction,and clipping.During cataloguing,users invoke the cataloguing interface and input parameters(e.g.,original data file paths,index names,index types),and the system automatically parses file metadata and generates indexes.For data extraction,users call the retrieval interface with specific parameters to obtain designated data.Moreover,the framework supports precise extraction of spe
作者
贾晓振
胡江凯
王大鹏
梁晨
JIA Xiaozhen;HU Jiangkai;WANG Dapeng;LIANG Chen(CMA Earth System Modeling And Prediction Center,Beijing 100080)
出处
《气象科技》
2024年第6期797-806,共10页
Meteorological Science and Technology
关键词
分布式架构
搜索引擎
数值预报业务
内存数据库
数据检索
跨平台
distributed architecture
search engine
numerical prediction business
in-memory database
data retrieval
cross-platform
