(j3.2006) Heterogeneous parallelism
Wed Aug 22 14:22:28 EDT 2007
Craig and Lawrie have brought up homogeneous versus heterogeneous parallelism
and how it affects co-arrays. I am trying to understand exactly what we are
talking about and what they want that is not there now.
I can think of two important sources of heterogeneity: Interconnect/memory
sharing, and different hardware.
The first is now standard, for example, clusters of SMPs or collections of
multi-core processors. Is that what the concern is? If yes, what exactly do
we want CAF to do about such hierarchical structure?
CAF explicitly chooses not to expose any such hierarchical structure of the
architecture to the programmer. To the programmer the machine is represented
as homogeneous. This is no different from the way memory is presented to all
programmers in Fortran (or any high-level language). All existing machines
have memory hierarchies and these affect performance in marvelous ways. Yet
we (and C) present memory as a linear random-access memory. Files are also
treated uniformly---there is no provision for remote verus nearby files. We
leave it up to hardware and compilers to deal with caches and reordering of
memory references, or buffering of files.
This does not mean that programmers are free to ignore memory hierarchy. Quite
the opposite, they need to be aware of it and there are lots of studies on
how to improve locality of codes. And locality seems to be the ONLY universal
answer. CAF stimulates locality: Reference variables on your image as much as
possible. It does not pretend that there is some uniform shared memory.
Beyond this, I cannot think of anything that would help portability or
programmers. In fact, the opposite, it would expose architectural details,
which is what Lawrie is arguing against!
As to the second type of heterogeneity, different processors and ISAs and
representation methods mixed together, I ask: What do you want to do about
such systems? Is this in conflict with CAF as it is or is it an extension of
it to add features to deal with heterogeneity?
If Lawrie's answer is, as for genericity, I don't know but I don't want to put
in the wrong answer, then I cannot argue with that. It is the worst kind of
argument one can pose: One that puts something down without offering a better
alternative. Sorry, programmers cannot wait for the perfect solution. In
fact, exisiting Fortran is anything but perfect. So long as it is good
enough, or, preferably, better than existing alternatives, we are in great
Aleksandar Donev, Ph.D.
Lawrence Postdoctoral Fellow @ LLNL
High Performance Computational Materials Science and Chemistry
E-mail: donev1 at llnl.gov
Phone: (925) 424-6816 Fax: (925) 423-0785
Address: P.O.Box 808, L-367, Livermore, CA 94551-9900
More information about the J3