Efficient Communication Between Parallel Programs with InterComm
Abstract
We present the design and implementation of InterComm, a framework to
couple parallel components that enables efficient communication in the
presence of complex data distributions within a coupled application.
Multiple parallel libraries and languages may be used in different modules
of a single application. The ability to couple such modules is required
in many emerging application areas, such as complex simulations that
model physical phenomena at multiple scales and resolutions, and remote
sensing image data analysis applications.
The complexity of the communication algorithms is highly dependent on the
distribution of data across the processes in a distributed memory parallel
program. We classify the distributions into two types - one that
represents a data distribution in a compact way so that the distribution
information can be replicated, and one that explicitly describes the
location of each data element, so can be very large, requiring that the
distribution information be distributed across processes as is the
data.
InterComm builds on our previous work on the Meta-Chaos program coupling
framework. In that work, we showed that the steps required to perform the
data exchanges include locating the data to be transferred within the
local memories of each program, generating communication schedules (the
patterns of interprocessor communication) for all processes, and
transferring the data using the schedules. In this paper we describe the
new algorithms we have developed, and show how those algorithms greatly
improve on the Meta-Chaos algorithms by reducing the high cost of building
the communication schedules. We present experimental results showing the
performance of various algorithmic tradeoffs, and also compare against the
original Meta-Chaos implementation.
(UMIACS-TR-2004-04)