Publications of Torsten Hoefler

Torsten Hoefler, Andrew Lumsdaine and Jack Dongarra:

Towards Efficient MapReduce Using MPI

(In Recent Advances in Parallel Virtual Machine and Message Passing Interface, 16th European PVM/MPI Users' Group Meeting, presented in Helsinki, Finland, Springer, Sep. 2009)

Abstract

MapReduce is an emerging programming paradigm for data-parallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show how anticipated MPI-2.2 and MPI-3 features, such as MPI Reduce local and nonblocking collective operations, can be used to implement and optimize MapReduce with a performance improvement of up to 25% on 127 cluster nodes. Finally, we discuss additional features that would enable MPI to more efficiently support all MapReduce applications.

Documents

download article:

download slides:

BibTeX

@inproceedings{hoefler-map-reduce,
  author={Torsten Hoefler and Andrew Lumsdaine and Jack Dongarra},
  title={{Towards Efficient MapReduce Using MPI}},
  year={2009},
  month={Sep.},
  booktitle={Recent Advances in Parallel Virtual Machine and Message Passing Interface, 16th European PVM/MPI Users' Group Meeting},
  location={Helsinki, Finland},
  publisher={Springer},
  source={http://www.unixer.de/~htor/publications/},
}


serving: 18.119.142.210:19444	© Torsten Hoefler