随着智能电网、全球能源互联网的建设与相关技术的发展,现代电力系统中电力大数据的格局已经形成,如何对高维海量数据进行深度挖掘以实现数据的充分利用,成为当前电力工作者们关心的问题。该文针对电力大数据环境下高精度和实时性的负...随着智能电网、全球能源互联网的建设与相关技术的发展,现代电力系统中电力大数据的格局已经形成,如何对高维海量数据进行深度挖掘以实现数据的充分利用,成为当前电力工作者们关心的问题。该文针对电力大数据环境下高精度和实时性的负荷预测展开了研究,提出了基于Spark平台和并行随机森林回归算法(Spark platform and parallel random forest regression,SP-RFR)的短期电力负荷预测方法,通过3次弹性分布式数据集(resilient distributed datasets,RDD)转换实现单机随机森林算法的并行化改进,并在Spark分布式集群环境下实现部署。结合某区域实际电力负荷数据设计试验,进行模型训练和回归预测,通过试验证明,对同等的数据集,基于Spark平台的并行随机森林回归算法预测精度高于单机负荷预测算法;并行随机森林算法受离群数据干扰较小,且随着数据集的增大,并行随机森林算法表现出良好的鲁棒性;与单机算法在运行时间上相比,随着数据集的增大,基于分布式集群的方法优势明显。该文提出的方法能够有效地在分布式环境中进行电力负荷预测,为负荷预测提供了一种新思路。展开更多
Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm...Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.展开更多
文摘随着智能电网、全球能源互联网的建设与相关技术的发展,现代电力系统中电力大数据的格局已经形成,如何对高维海量数据进行深度挖掘以实现数据的充分利用,成为当前电力工作者们关心的问题。该文针对电力大数据环境下高精度和实时性的负荷预测展开了研究,提出了基于Spark平台和并行随机森林回归算法(Spark platform and parallel random forest regression,SP-RFR)的短期电力负荷预测方法,通过3次弹性分布式数据集(resilient distributed datasets,RDD)转换实现单机随机森林算法的并行化改进,并在Spark分布式集群环境下实现部署。结合某区域实际电力负荷数据设计试验,进行模型训练和回归预测,通过试验证明,对同等的数据集,基于Spark平台的并行随机森林回归算法预测精度高于单机负荷预测算法;并行随机森林算法受离群数据干扰较小,且随着数据集的增大,并行随机森林算法表现出良好的鲁棒性;与单机算法在运行时间上相比,随着数据集的增大,基于分布式集群的方法优势明显。该文提出的方法能够有效地在分布式环境中进行电力负荷预测,为负荷预测提供了一种新思路。
基金This work is supported by the Science Research Projects of Hunan Provincial Education Department(Nos.18A174,18C0262)the National Natural Science Foundation of China(No.61772561)+2 种基金the Key Research&Development Plan of Hunan Province(Nos.2018NK2012,2019SK2022)the Degree&Postgraduate Education Reform Project of Hunan Province(No.209)the Postgraduate Education and Teaching Reform Project of Central South Forestry University(No.2019JG013).
文摘Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.
