摘要
Kafka是一种高吞吐量消息中间件,但该算法存在负载均衡导致消费者消息处理效率下降的问题。本文提出一种改进的Kafka负载均衡算法,其中协调者基于消费者和分区的对应关系,根据不同的负载均衡场景优化调整消费者数目,按照业务负载倒序的方式更新消费者和分区的对应关系,优先处理负载较大的业务分区,提高消息传输效率,并且搭建Fabric联盟链验证分析算法的性能。实验结果表明,本算法在中央处理器(CPU)资源消耗比其他算法低5%的情况下,共识速度提升了2%~7%,并且在6个Kafka节点中3个宕机的情况下仍然能共识上链,提升了Kafka负载均衡算法的效率和稳定性。
Kafka is a kind of message middleware with high throughput.However,this algorithm has the problem that load balancing can reduce the efficiency of consumer message processing.The coordinator in this algorithm optimi-zes and adjusts the number of consumers based on the correspondence between consumers and partitions according to different load balancing scenarios,and then updates the correspondence between consumers and partitions in a reverse order of business load.This algorithm prioritizes the business partitions with heavy load and improves the ef-ficiency of message transmission.And build the Fabric consortium chain to verify the performance of the algorithm.Experimental results show that the proposed algorithm improves the consensus speed by 2%-7%when the central processing unit(CPU)resource consumption is 5%lower than that of other algorithm,and the consensus link can still be available even when three out of six Kafka nodes are out of service,which improves the efficiency and sta-bility of the Kafka load balancing algorithm.
作者
苏玉钊
孙恩昌
杨睿哲
李萌
张延华
司鹏搏
张卉
SU Yuzhao;SUN Enchang;YANG Ruizhe;LI Meng;ZHANG Yanhua;SI Pengbo;ZHANG Hui(Faculty of Information Technology,Beijing University of Technology,Beijing 100124)
出处
《高技术通讯》
CAS
2023年第1期42-49,共8页
Chinese High Technology Letters
基金
国家自然科学基金(61901011)
北京市教育委员会科技计划一般项目(KM202110005021,KM202010005017)资助。
