近年来,随着计算机技术的发展及其在互联网、传感器和科学数据分析等领域的广泛应用,数据量爆炸性地增长,海量数据给传统的数据管理和分析带来新的挑战,学界和业界广泛采用分布式文件系统和MapReduce编程模型来应对这一挑战.介绍了HaoLa...近年来,随着计算机技术的发展及其在互联网、传感器和科学数据分析等领域的广泛应用,数据量爆炸性地增长,海量数据给传统的数据管理和分析带来新的挑战,学界和业界广泛采用分布式文件系统和MapReduce编程模型来应对这一挑战.介绍了HaoLap(Hadoop based OLAP),一种基于Hadoop分布式文件系统(HDFS)和MapReduce编程模型的海量数据OLAP系统.本研究吸取了MOLAP的经验:采用元数据存储多维模型以及HDFS存储事实数据,采用编码完成维和事实数据的映射,采用MapReduce完成OLAP运算.描述了HaoLap的关键技术,包括系统结构、维定义和编码、事实数据存储和编码、OLAP算法和服务接口.介绍了HaoLap在科学数据分析的应用案例,并与主流非关系数据管理系统进行性能对比.实验结果表明,尽管数据装载性能略显不足,但HaoLap的OLAP性能要优于HBase,Hive,HadoopDB等主流非关系数据管理系统.展开更多
智能公交系统(Advanced Public Transportation Systems,APTS)数据具有海量、类型多样等大数据的典型特征,对其进行分析和挖掘可能获得丰富的公交出行特征和规律.构建基于APTS大数据的公交出行多维分析框架,在计算乘客出行时空信息(上...智能公交系统(Advanced Public Transportation Systems,APTS)数据具有海量、类型多样等大数据的典型特征,对其进行分析和挖掘可能获得丰富的公交出行特征和规律.构建基于APTS大数据的公交出行多维分析框架,在计算乘客出行时空信息(上车、下车和换乘)的基础上,建立包含4个维度(乘客、时间、空间和行为)的公交出行数据模型,系统提出基于5种联机分析处理方法的公交出行分析内容.应用APTS大数据对模型和方法进行了实验和验证,研究结果表明,该方法能够便捷地分析不同维度、不同粒度的公交出行信息,不仅能够应用于公交乘客出行行为的研究,还能够为城市公交系统的规划和管理提供决策支持.展开更多
To improve the effectiveness of dam safety monitoring database systems, the development process of a multi-dimensional conceptual data model was analyzed and a logic design wasachieved in multi-dimensional database mo...To improve the effectiveness of dam safety monitoring database systems, the development process of a multi-dimensional conceptual data model was analyzed and a logic design wasachieved in multi-dimensional database mode. The optimal data model was confirmed by identifying data objects, defining relations and reviewing entities. The conversion of relations among entities to external keys and entities and physical attributes to tables and fields was interpreted completely. On this basis, a multi-dimensional database that reflects the management and analysis of a dam safety monitoring system on monitoring data information has been established, for which factual tables and dimensional tables have been designed. Finally, based on service design and user interface design, the dam safety monitoring system has been developed with Delphi as the development tool. This development project shows that the multi-dimensional database can simplify the development process and minimize hidden dangers in the database structure design. It is superior to other dam safety monitoring system development models and can provide a new research direction for system developers.展开更多
文摘近年来,随着计算机技术的发展及其在互联网、传感器和科学数据分析等领域的广泛应用,数据量爆炸性地增长,海量数据给传统的数据管理和分析带来新的挑战,学界和业界广泛采用分布式文件系统和MapReduce编程模型来应对这一挑战.介绍了HaoLap(Hadoop based OLAP),一种基于Hadoop分布式文件系统(HDFS)和MapReduce编程模型的海量数据OLAP系统.本研究吸取了MOLAP的经验:采用元数据存储多维模型以及HDFS存储事实数据,采用编码完成维和事实数据的映射,采用MapReduce完成OLAP运算.描述了HaoLap的关键技术,包括系统结构、维定义和编码、事实数据存储和编码、OLAP算法和服务接口.介绍了HaoLap在科学数据分析的应用案例,并与主流非关系数据管理系统进行性能对比.实验结果表明,尽管数据装载性能略显不足,但HaoLap的OLAP性能要优于HBase,Hive,HadoopDB等主流非关系数据管理系统.
文摘智能公交系统(Advanced Public Transportation Systems,APTS)数据具有海量、类型多样等大数据的典型特征,对其进行分析和挖掘可能获得丰富的公交出行特征和规律.构建基于APTS大数据的公交出行多维分析框架,在计算乘客出行时空信息(上车、下车和换乘)的基础上,建立包含4个维度(乘客、时间、空间和行为)的公交出行数据模型,系统提出基于5种联机分析处理方法的公交出行分析内容.应用APTS大数据对模型和方法进行了实验和验证,研究结果表明,该方法能够便捷地分析不同维度、不同粒度的公交出行信息,不仅能够应用于公交乘客出行行为的研究,还能够为城市公交系统的规划和管理提供决策支持.
基金supported by the National Natural Science Foundation of China (Grant No. 50539010, 50539110, 50579010, 50539030 and 50809025)
文摘To improve the effectiveness of dam safety monitoring database systems, the development process of a multi-dimensional conceptual data model was analyzed and a logic design wasachieved in multi-dimensional database mode. The optimal data model was confirmed by identifying data objects, defining relations and reviewing entities. The conversion of relations among entities to external keys and entities and physical attributes to tables and fields was interpreted completely. On this basis, a multi-dimensional database that reflects the management and analysis of a dam safety monitoring system on monitoring data information has been established, for which factual tables and dimensional tables have been designed. Finally, based on service design and user interface design, the dam safety monitoring system has been developed with Delphi as the development tool. This development project shows that the multi-dimensional database can simplify the development process and minimize hidden dangers in the database structure design. It is superior to other dam safety monitoring system development models and can provide a new research direction for system developers.
