Spark任务间消息传递方法研究 夏立斌.pdf

计算机工程与应用
Computer Engineering and Applications
ISSN 1002-8331,CN 11-2127/TP
存取快速的特点和 MPI 的多种高性能通信机制，
解决了 Spark 编程模型表达能力不足的缺陷，同时为 MPI 提供了面向数据的 DAG 计算方式。通过对 Spark 内
部的运行环境和调度系统进行修改，使得 MPI 在 Spark 中得以无缝融合，为高性能计算和大数据任务提供了一
个统一的内存计算系统。测试结果表明，在数值计算和迭代算法上相比 Spark 至少有 50%的性能提升。
关键词：Spark；MPI；科学计算；内存计算；迭代算法
文献标志码: A 中图分类号:TP311 doi：.1002--0182
Exploring Message Passing Method between Spark Tasks
XIA Libin1,2, LIU Xiaoyu1,2, SUN Wei1, JIANG Xiaowei1,2, SUN Gongxing1
1. Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China
2. University of Chinese Academy of Sciences, Beijing 100049, China
Abstract：Engineering problems and scientific research are facing dual challenges of big data processing and
high-performance computing tasks. Spark, a distributed processing framework based on in-memory computing
technology, has been widely used in academia and industry. However, its MapReduce-like programming model fails
to communicate between tasks, causing numerical algorithms in scientific computing cannot be efficiently imple-
mented. In response to the above problems, a compu