(j3.2006) Heterogeneous parallelism

[Aleksandar Donev]

Hi,

Craig and Lawrie have brought up homogeneous versus heterogeneous parallelism 
and how it affects co-arrays. I am trying to understand exactly what we are 
talking about and what they want that is not there now.

I can think of two important sources of heterogeneity: Interconnect/memory 
sharing, and different hardware. 

The first is now standard, for example, clusters of SMPs or collections of 
multi-core processors. Is that what the concern is? If yes, what exactly do 
we want CAF to do about such hierarchical structure? 

CAF explicitly chooses not to expose any such hierarchical structure of the 
architecture to the programmer. To the programmer the machine is represented 
as homogeneous. This is no different from the way memory is presented to all 
programmers in Fortran (or any high-level language). All existing machines 
have memory hierarchies and these affect performance in marvelous ways. Yet 
we (and C) present memory as a linear random-access memory. Files are also 
treated uniformly---there is no provision for remote verus nearby files. We 
leave it up to hardware and compilers to deal with caches and reordering of 
memory references, or buffering of files.

This does not mean that programmers are free to ignore memory hierarchy. Quite 
the opposite, they need to be aware of it and there are lots of studies on 
how to improve locality of codes. And locality seems to be the ONLY universal 
answer. CAF stimulates locality: Reference variables on your image as much as 
possible. It does not pretend that there is some uniform shared memory. 
Beyond this, I cannot think of anything that would help portability or 
programmers. In fact, the opposite, it would expose architectural details, 
which is what Lawrie is arguing against!

As to the second type of heterogeneity, different processors and ISAs and 
representation methods mixed together, I ask: What do you want to do about 
such systems? Is this in conflict with CAF as it is or is it an extension of 
it to add features to deal with heterogeneity?

If Lawrie's answer is, as for genericity, I don't know but I don't want to put 
in the wrong answer, then I cannot argue with that. It is the worst kind of 
argument one can pose: One that puts something down without offering a better 
alternative. Sorry, programmers cannot wait for the perfect solution. In 
fact, exisiting Fortran is anything but perfect. So long as it is good 
enough, or, preferably, better than existing alternatives, we are in great 
shape!

Best,
Aleks

-- 
Aleksandar Donev, Ph.D.
Lawrence Postdoctoral Fellow @ LLNL
High Performance Computational Materials Science and Chemistry
E-mail: donev1 at llnl.gov
Phone: (925) 424-6816  Fax: (925) 423-0785
Address: P.O.Box 808, L-367, Livermore, CA 94551-9900
Web: http://cherrypit.princeton.edu/donev