Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm...Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.展开更多
股票市场的情绪可以在一定程度上反映投资者的行为并影响其投资决策。市场新闻作为一种非结构性数据,能够体现并引导市场的大环境情绪,与股票价格一同成为至关重要的市场参考数据,能够为投资者的投资决策提供有效帮助。文中提出了一种...股票市场的情绪可以在一定程度上反映投资者的行为并影响其投资决策。市场新闻作为一种非结构性数据,能够体现并引导市场的大环境情绪,与股票价格一同成为至关重要的市场参考数据,能够为投资者的投资决策提供有效帮助。文中提出了一种可以准确、快速地建立针对海量新闻数据的多维情绪特征向量化方法,利用支持向量机(Support Victor Machine,SVM)模型来预测金融新闻对股票市场的影响,并通过bootstrap来减轻过拟合问题。在沪深股指上进行实验的结果表明,相比于传统模型,所提方法能够将预测准确度提高约8%,并在3个月的回测实验中获得了6.52%的超额收益,证明了其有效性。展开更多
基金This work is supported by the Science Research Projects of Hunan Provincial Education Department(Nos.18A174,18C0262)the National Natural Science Foundation of China(No.61772561)+2 种基金the Key Research&Development Plan of Hunan Province(Nos.2018NK2012,2019SK2022)the Degree&Postgraduate Education Reform Project of Hunan Province(No.209)the Postgraduate Education and Teaching Reform Project of Central South Forestry University(No.2019JG013).
文摘Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.
文摘股票市场的情绪可以在一定程度上反映投资者的行为并影响其投资决策。市场新闻作为一种非结构性数据,能够体现并引导市场的大环境情绪,与股票价格一同成为至关重要的市场参考数据,能够为投资者的投资决策提供有效帮助。文中提出了一种可以准确、快速地建立针对海量新闻数据的多维情绪特征向量化方法,利用支持向量机(Support Victor Machine,SVM)模型来预测金融新闻对股票市场的影响,并通过bootstrap来减轻过拟合问题。在沪深股指上进行实验的结果表明,相比于传统模型,所提方法能够将预测准确度提高约8%,并在3个月的回测实验中获得了6.52%的超额收益,证明了其有效性。
