(j3.2006) (SC22WG5.4478) Rice University Commentary on ISO/IEC JTC1/SC22/WG5 N1835 (Addition/Modification of CAF Features)

David Muxworthy d.muxworthy
Thu Jun 16 05:21:13 EDT 2011

The following message was blocked by the sc22wg5 mailserver.
David Muxworthy
From: Bill Scherer <scherer at rice.edu>
Subject: Rice University Commentary on ISO/IEC JTC1/SC22/WG5 N1835  
(Addition/Modification of CAF Features)
Date: Wed, 15 Jun 2011 19:26:49 -0500
Cc: cafcompiler-l at rice.edu
To: sc22wg5 at open-std.org, John.Reid at stfc.ac.uk

Attached, please find Rice University's Commentary on ISO/IEC JTC1/ 
SC22/WG5 N1835 (Addition/Modification of CAF Features).

Thank you for your consideration.

A Critique of ISO/IEC JTC1/SC22/WG5 N1835 (Addition/Modification of  
CAF Features)

    Laksono Adhianto, John Mellor-Crummey, Guohua Jin, Karthik  
Murthy, Dung Nguyen,
            William N. Scherer III, Scott Warren, and Chaoran Yang
{laksono, johnmc, jin, Karthik.S.Murthy, dxnguyen, scherer, scott,  

In this article, we provide commentary on the feature additions/ 
to J3/08-131r1 based on the discussion held in Sept 2010. This  
document is
available online as ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/ 
N1835.txt.  Our
commentary is based on our experiences with developing and using the  
Coarray Fortran 2.0 (Rice CAF 2.0) programming language, runtime, and

Proposal 1.

We generally support this proposal; however, we believe that a larger  
set of
intrinsics would be useful. In particular, the full set of collectives
supported by MPI seems worth considering.

Although we did not implement Rice CAF 2.0 collectives in this  
manner, having
an optional result parameter seems reasonable to us.

Proposal 2.

We agree that "raw" atomic operations are useful for development of
high-performance synchronization and concurrency routines. We suggest  
the committee consider the equivalent feature set from the Java  
language, which appears in the java.util.concurrent library, as it  
has been
very successful in that community. Specifically, it supports two key  
that are missing from this proposal:

(1) atomic swap, also known as fetch-and-store, is necessary for the
implementation of commercially important algorithms including the  
routine for the widely used MCS queue-based lock.  Although atomic  
swap can
be simulated via a looped CAS construct, this is an imperfect  
because the CAS loop can fail arbitrarily many times (starvation) before
success; atomic swap is guaranteed to complete within a bounded  
length of

(2) CAS on pointer values -- equivalent to the
java.util.concurrent.AtomicReference class -- is necessary for the
implementation of virtually all concurrent algorithms that are  
currently in
use.  In C, support for integers is sufficient because its more  
powerful cast
operations allow the programmer to cast a pointer to an integer type;
however, the equivalent functionality is not present in Fortran due  
to its
stronger typing.

We note that the restriction of types to exclude variables of type  
real seems
arbitrary; however, we have no opinion on whether they should be  
included as possible targets of the atomic instructions.

Finally, we observe that some level of protection against the so- 
called ABA
problem is desirable. The ABA problem occurs when a CAS is made  
against a
value that has changed but has then accidentally changed back to its  
value between when it was first read and when the CAS is effected.   
In this
case, it is usually wrong (algorithmically) for the CAS to succeed; this
leads to subtle corruption and difficult to track down race  
conditions in

We additionally refer the committee to the C++ atomics  
standardization work
by Hans Boehm and Lawrence Crowl [1].

Proposal 3.

3b) We generally concur that restrictions should only be present when
absolutely necessary.

3c) In our view, there is already enough confusion in the world about  
difference between global and local synchronization.  They are very  
things; combining them into a single sync statement will only serve to
increase the confusion.

3d) We see no problem with allowing functions to have side effects.  
than an IMPURE attribute that proclaims a function free of side effects,
however, we espouse a PURE attribute that is an explicit promise,  
made by the
programmer, that a function is side effect free.

3e) Fundamentally, we disagree with requiring MPI in addition to  
Fortran in
order to have a complete programming model: [Coarray] Fortran should  
stand on
its own.  There is substantial utility to having a rich set of  
and compiler support for them can greatly ease the burden on the  
(and reduce opportunities for error) when using them.  For example,  
in the
Rice CAF 2.0 implementation, we have built support in the compiler to
automatically compute sizes of data and to generate callback functions.

3f) Teams are needed for coupled codes and are very useful for linear  
applications.  Again, we disagree strongly with requiring MPI in  
addition to
Fortran in order to have a complete programming model.  This is  
true when an all-coarray Fortran program could be aesthetically  

3g) We dislike notify and query as we strongly prefer first-class  
Instead of directly synchronizing with another processor, we find it  
a far
better programming model to synchronize with an event that is logically
connected to remote data.  Further, events provide a safe  
space: If a library method notifies an event, that notification  
cannot be
picked up by a waiting operation in user code, but with direct
processor-to-processor synchronization, the same cannot be said.   
synchronization errors of this form is slow, tedious, and painful.

3h) Rather than have an intrinsic isMyLock that is specific to locks, we
propose extending imageof() from handling just copointers to also  
locks and events.  However, we note that many implementations will  
wish to
use a test-and-test-and-set lock, for which lock ownership  
information is not
normally stored with the lock.

Rather than isLocked(), we would suggest adding a trylock() function  
attempts to acquire a lock if it is unlocked and fails otherwise.
Programmers should not write their own spin loops.  Locks can  
implement their
own spins, including spin-then-yield code as appropriate.  This gains
efficiency since no traversal of data structures is necessary to find  
memory location to spin on.

On the subject of locks, we note that formal locksets allow multi-lock
locking to occur in a canonical order; this provides a degree of safety
against cyclic deadlock in multi-lock codes.  A very simple canonical  
would be the address of the lock variables.

3i) While compatibility is useful, we reiterate our stance that  
Fortran 2008
should stand on its own.  For example, the compiler can generate
multithreaded or CUDA code from a do concurrent loop.  Requiring CUDA +
OpenMP + MPI + CAF is far less aesthetically appealing than an all-CAF

Proposal 4.

This proposal is subsumed by our approach to copointers, the details  
of which
appear in Appendix II.  In particular, we observe that adding the  
attribute to a non-coarray variable makes it a coscalar by requiring  
that it
be allocated in shared memory space.

We see no need for the relocate() statement nor for the image=  
qualifier to
the allocate statement.  Functionality equivalent to relocate() can be
achieved by just reallocating the scalar and copying date from the old
location to the new.  Functionality equivalent to the image=  
qualifier can be
achieved by placing a conditional around the allocation statement:

         if (mype .eq. 4) then

We note that for caching purposes, it suffices to copy a coscalar to  
a local

In general, the heap is not symmetric; providing optimizations based  
on an
assumption otherwise seems ill advised.

Proposal 5.

We are in full agreement that asynchronous collective operations are  
and desirable.  In fact, we have used them to good effect in  
developing Rice
CAF 2.0 implementations for the High Performance Computing Challenge  
benchmarks [2].

Rice CAF 2.0 supports two variants of asynchrony for collectives.  In  
explicit model, an event variable is supplied as a parameter to the
collective.  Upon completion of the collective operation, the event is
notified.  This allows the programmer to determine when the collective
operation has completed so that subsequent code, predicated on  
completion of
the collective, may be executed.

         co_sum_async(some_coarray, some_event)  ! kick off a reduction
         ...                                     ! overlap  
computation with it
         event_wait(some_event)                  ! ensure it has  

In contrast, in the implicit model, the programmer omits the event  
and instead calls an explicit "cofence" to be sure that all pending
operations have completed:

         co_sum_async(some_coarray)  ! kick off an asynchronous  
         ...                         ! overlap computation with the  
         cofence                     ! ensure it has completed

For more details of the cofence, see Appendix I.

In addition to collectives, we have found substantial benefit in  
two other asynchronous functions:

(1) An asynchronous barrier offers the same functionality as does a
split-phased barrier.  Triggering the barrier is equivalent to a  
notify, and
waiting on the event/blocking with a cofence is equivalent to awaiting
completion of the barrier.

(2) A predicated asynchronous copy allows data to be transferred to/ 
from a
remote image as soon as it is ready, and automatically notify when  
the copy
has completed.  This is useful, for example, in a scenario where we have
initialization to perform and need data from a partner:

          copy_async(my_buffer, remote_buffer[partner], pred_event, &
          ! perform other initialization while waiting for the data
          event_wait(data_copied_event)      ! make sure we have the  
          ! proceed with computation

Here, we have overlapped the computation of our initialization with the
communication of data into my_buffer from partner's remote_buffer.

Proposal 6.

Issue A) We believe that this is a non-issue.  The allocation of  
coarray 'a'
on team c would overwrite the pointer to a on overlapping members;  
there can
be only one 'a' on any image.  This would of course be a programming  
that could be checked at runtime when trying to allocate an already- 

In the Rice CAF 2.0 implementation, coarrays are registered after  
the name duplication conflict would manifest in this stage if it had not
previously been detected.

Issue B) As detailed in Tony Skjellum's rationale for MPI libraries [3],
reindexing is crucial if support libraries are to be developed.  We  
that having ranks > 1 poses several logistical problems from a language
viewpoint.  This is precisely why we oppose having more than one rank  

However, to give the functionality of multiple dimensions, we support
topologies.  In particular, with a cartesian topology, one can write  
that appears to index multiple ranks.  The indexing is reduced by the
topology to a linearized one-dimensional index into the single  
physical rank
for the coarray.  This resolves the issues described here.

Issue C). We believe that teams are very useful for many applications,
including coupled codes and linear algebra applications to name two.   
We urge
the committee not to remove them from the Fortran 2008 specification.

Proposal 7.

This proposal is subsumed by our approach to copointers, the details  
of which
appear in Appendix II.

Proposal 8.

We agree that this proposal has appeal.  In fact, an early version of  
our CAF
2.0 implementation supported asymmetric coarrays.  But when we tried  
it, it
caused chaos with reshaping of arrays.

Suppose, for example that we have 2D arrays of different sizes.  Now  
we pass column 3 to a local subroutine, which then tries to access that
column on another image.  We see no reasonable way to handle the case  
that column does not exist on the remote image.  Further, even if the  
image *does* have a third column in the coarray, what if the columns  
are of
differing lengths? The subroutine has no good way to know the bounds  
of the
column on the remote image. A semantic problem occurs when we attempt to
access the entire column (via a ':' operator): does the colon refer  
to the
local or remote bounds?

For all of these reasons, we dropped support for asymmetric coarrays  
from our
CAF 2.0 compiler.

Proposal 9.

We note that when reading a standard, it is useful to have names that  
logically associated appear near to each other in the standard,  
including the
index and a table of intrinsics.  For this reason, we have adopted  
and event_notify in the Rice CAF 2.0 implementation.

9.1) As detailed in our memory model notes (see Appendix I), notify  
should be
a "release" operation.  Coarray operations that appear after a notify  
execute before the notify, but no coarray operations before a notify  
execute after the notify.  This is needed to make events reasonable:  
If a
programmer writes to a remote coarray then performs a notify to  
signal that
the write has completed, the write had better not be delayed until  
after the

Similarly, query should be an "acquire" operation (antisymmetric  

In general, the semantics of notify should be non-blocking.   
should occur after the communication completes, but there is no need  
to block
the caller until that time.  This would just make it harder to overlap
communication latency with computation (which is crucial for extracting
maximum performance in HPC environments).

9.2) It seems strange to separate the image number and event name  
when they
could be combined into a single parameter.  For example, the second  
below seems far more intuitive and in keeping with existing coarray  

         notify(3, some_event(i))       ! As proposed
         notify(some_event(i)[3])       ! Implemented in Rice CAF 2.0

9.3) We disagree with restricting the number of outstanding notifies  
to one.
For example, a bounded buffer implementation could take advantage of  
-- and
would require -- higher limits.

9.4) We note that image numbers should be relative to a team.  For  
in the following call, j is relative to the team some_team, not an  
image number:

         notify(some_event(i)[j at some_team])

9.5) Please don't conflate notify and query with (asynchronous)  
Point-to-point and collective operations should be kept as separate

On the subject of events, similar to the locksets we proposed earlier  
in this
document, we propose eventsets.  As implemented in Rice CAF 2.0, these
collections of events offer programmers the following convenient

         notifyall:    perform a notify on each member event
         waitall:      wait for each member event to be notified
         wantany:      wait for one member event to be notified,  
similar to
                       the socket library select() method
         waitanyfair:  wait for one of the member events that has  
                       the fewest notifications to be notified

Since it may not be obvious, the intent behind waitanyfair is that by  
it in a loop exactly N times, where N is the cardinality of the  
eventset, it
is guaranteed that each component event will have been notified  
exactly once
at the termination of the loop.


[1] Hans-J. Boehm, Lawrence Crowl. C++ Atomic Types and Operations.  
JTC1 SC22 WG21 N2427 = 07-0297 - 2007-10-03.

[2] HPC Challenge benchmark. http://icl.cs.utk.edu/hpcc.

[3] A. Skjellum, N. E. Doss, and P. V. Bangalore. Writing libraries in
MPI. In A. Skjellum and D. S.  Reese, editors, Proceedings of the  
Parallel Libraries Conference, pages 166=E2=80"173. IEEE Computer  
Society Press,
October 1993.

Appendix I: Commentary on the Fortran 2008 Memory Model

In this section, we present our views/comments on the memory model  
in the draft Fortran 2008 standard (while the memory model has not been
formally described in the F2008 standard, our views are based on  
mined from it, especially Section 8.5: Image Execution Control).

Comment #1: The draft standard does not define the consistency  
within a segment.

We recommend processor consistency for coarray reads/writes within a  

The absence of any form of consistency within a segment allows  
compiler/hardware reorderings; this forces the programmer to introduce
numerous memory fences in the program for correctness, making it  
harder to
add optimization.

Comment #2: The current memory model effects a difficult programming  

We believe that the average programmer should not be exposed to the
intricacies of the memory model (such as needing to use sync_memory) in
order to write correct code.
The current memory model supports a "performance-first" approach by  
aggressive compiler/hardware optimizations which re-order within a  
segment or
between segments which are not ordered via image control
constructs. Programmers have to use sync_memory to avoid subtle race
conditions in most places especially when asynchronous operations are
employed. We believe that average programmers should not need to  
learn the
intricacies of the memory model to obtain correct code.

Comment #3: The current memory model lacks predicated fences.

We believe that predicated fences, such as our cofence, are a necessary

The current memory fences (sync_memory) are not sufficiently flexible to
provide the required performance tuning that advanced programmers  
would need.
As currently described, sync_memory acts as a barrier for all memory and
coaarray operations. However, advanced programmers need constructs to
separately capture the local/global completion of coarray operations,
especially asynchronous ones.

The cofence construct allows programmers to control the local  
completion of
put, get, and implicitly synchronized asynchronous operations. The  
API is as follows:


Cofence takes two optional arguments.  The first specifies which  
of implicit asynchronous operations i.e. "put/get" to allow  
downwards, and
the second argument specifies which category of implicit asynchronous
operations to allow upwards. Depending upon the argument values  
passed, the
cofence allows puts, gets, or both to pass across the cofence in the
specified direction.

Let us consider a collective asynchronous broadcast operation to  
the use of cofences in tuning performance.

         ! process p is performing a broadcast
         broadcast_async(buffer, p)
         cofence(DOWNWARD=GET, UPWARD=PUT_GET)

         ! after the cofence, buffer can be safely overwritten
         buf = ...

         ! wait for global completion of the broadcast

In the above code sample, process p is performing an asynchronous
broadcast. Once p sends the broadcast data to its children (i.e. the
broadcast is locally complete in p) p does not need to participate in  
remainder the broadcast. Processor p can thus overlap useful work,  
such as
preparing the next iteration of buffer, with waiting for the  
broadcast to

While capturing this local completion, it is performance efficient to  
other "get" memory operations to be performed later (allowed to pass
downwards) or "put/get" memory operations to be performed earlier  
relative to the cofence.  A full memory barrier would not allow these

The broadcast is globally complete when all participating processes  
the broadcast data. Global completion is important for process p if
activities after the broadcast in p are dependent directly/ 
transitively on
the assumption that the other processes have received the broadcast  

Comment #4: It is not clearly stated (but it is implied) that functions
should not have side effects. This should be clarified in the standard.

Appendix II: Copointers in Rice CAF 2.0

CAF 2.0 adds global pointers to the Fortran language in support of  
data decompositions, distributed linked data structures, and parallel  
coupling. The definition and use of these new "copointers" is as  
similar as
possible to ordinary Fortran pointers: they are declared with new  
analogous to 'pointer' and 'target', manipulated with the existing '=>'
pointer assignment statement, and inspected with the existing pointer
intrinsics. Accessing data via copointers is as similar as possible to
existing coarray accesses, with implicit access to the local image and
explicit access to remote images using a square-bracket notation. CAF  
copointers may point to values of any type, including coarrays; we  
that copointers to coarrays will be especially valuable for parallel  
coupling in systems like the Community Earth System Model. Copointers  
can be
implemented easily and efficiently in existing CAF compilers; we have  
begun adding them to our prototype CAF 2.0 compiler.

The rest of this note explains the copointer concept in more detail,  
describes how copointers are declared, created, copied, dereferenced,  
inspected. It closes by mentioning a few nonobvious semantic details and
sketching an implementation strategy.

The approach here is tutorial rather than formal and terminology is  
for the
most part programmer- oriented rather than compatible with the Fortran
standard documents. For instance, we usually say "variable" rather  
than the
standards' "entity" and "points to" rather than "is associated with".  
But not


Copointers are typed "global pointers" which can point to storage on any
processor ("image") in a parallel computer. Each copointer points to a
specific typed block of storage (a Fortran "entity") allocated on a  
image. Despite the"co" in their name, copointers are not distributed  
images like coarrays; each copointer is a small scalar value residing  
on a
single image. Apart from their global reach, the semantics of  
copointers is
nearly identical to the semantics of ordinary Fortran pointers:  
variables and copointer components of derived types may be declared,  
set to
point to other entities, copied, dereferenced, sectioned via  
subscripting to
yield copointers to subentities, and examined via the existing Fortran
pointer intrinsics. It may be helpful to think of a copointer as a  
pair <i,p>
where 'i' is an image number and 'p' is an ordinary Fortran pointer  
valid on
'i', although the implementation may be different.

Cotargets are entities which may become the destination of a  
copointer. Such
entities must be declared with the 'cotarget' attribute, just as  
destinations of ordinary pointers must be declared with the 'target'
attribute. If a CAF2 implementation relies on special "shared memory"  
for efficient communication between images, then it will allocate  
with the 'cotarget' attribute in such a region. Cotarget entities are  
in all
other respects ordinary entities and may be used locally without  

Copointer values may be freely copied, even from one image to  
another, and
each new copy points to the same specific storage block on the same  
image as does the original copointer. Creating and copying copointers  
cheap, purely local operations. So is dereferencing a copointer that  
to point to the image doing the dereferencing. Dereferencing a copointer
which points to a different image requires the same sort of  
communication as
a corresponding off-image coarray reference.


Copointer and cotarget entities are declared with the usual Fortran
declaration syntax augmented with new 'copointer' and 'cotarget'
attributes. For instance, to declare an integer array and a copointer  
can point to it, we write

         integer, dimension(10), cotarget :: a1
         integer, dimension(:), copointer :: p1

This makes 'a1' an array of 10 integers allocated in shared memory  
and 'p1' a
copointer variable of compatible type. Copointers may point to  
entities of
any type, subject to the limitations of Fortran's attribute syntax as
explained in the next paragraph. In particular CAF 2.0 allows  
copointers to
coarrays, providing an expressive and efficient mechanism for model  
in large parallel codes. The 'copointer' and 'cotarget' attributes  
may be
combined with other Fortran attributes just as 'pointer' and 'target'  
be. For instance,

         type(t), dimension(:,:), save, contiguous, copointer :: p2

declares a copointer entity 'p2' which points to two-dimensional  
arrays of
elements of derived type 't', which retains its association across  
invocations, and which can only be associated with contiguous cotarget

Declaring cotargets needs no further explanation. To describe how  
types are declared we must first consider a key syntactic feature of
Fortran's existing type declarations: namely, that the textual order  
in which
an entity's attributes are given is insignificant. This feature both  
potential ambiguities and limits the set of data types which can be
expressed. For instance, both of the following declarations specify type
"pointer to array of integer":

         integer, pointer, dimension(:) :: p3
         integer, dimension(:), pointer :: p4

Since the order of appearance of 'pointer' and 'dimension' does not  
the ambiguity in interpretation is resolved by a rule we can write as
"pointer < dimension"; that is, 'pointer' has lower syntactic  
priority than
'dimension' and so is applied later during type formation, giving
"pointer(dimension1(integer))" as the specified type. Because of this  
there is no way to express the type "array of pointer to integer" in
Fortran. However, the missing type can be simulated by wrapping a  
pointer in
a derived type:

         type :: t; integer, pointer :: p; end type
         type(t), dimension(:) :: a2
         ! initialize a2 =E2=80=A6
         a2(1)%p = 0

We can now describe the precise syntactic intepretation of  
'copointer' in CAF
2.0 by the following rules:

         pointer < copointer < codimension < dimension

These precedence relations are consistent with the existing syntax of  
2008 and give an unambiguous interpretation of every possible  
combination of
these four attributes in a type declaration. For instance, both of the
following declarations specify a copointer to a coarray of corank 1,  
rank 2,
and element type integer:

         integer, dimension(:,:), codimension(:), copointer :: p5
         integer, copointer :: p6(:,:)[*]

In each declaration, the three attributes 'copointer', 'codimension',  
'dimension' occur and are interpreted in that order to give the type

Like Fortran 2008's, CAF 2.0's attribute interpretation rules resolve
ambiguity at the cost of limiting the set of types which can be directly
expressed. For instance, "array of copointer" can't be expressed but  
can be
simulated with derived types just as shown above for "array of pointer".


Copointers are created and copied via Fortran's existing 'allocate' and
pointer assignment statements in the same way as ordinary pointers.  
There are
four cases to consider.

(1) A copointer is created when an 'allocate' statement is executed  
with a
copointer variable as its argument. The allocated storage comes from the
current image's shared memory region so that it can be accessed from any
other image. A copointer to that storage is created and stored in the
argument variable.

(2) A copointer is created when a pointer assignment statement's  
right hand
side (RHS) is a plain data reference; a new copointer to the RHS is  
to the variable on the left hand side (LHS). (In Fortran terminology,  
the LHS
entity "becomes copointer associated with" the RHS data ref.) The RHS  
have the 'cotarget' attribute. The RHS may be either a reference to  
data on this image or a reference to remote data on another image; in  
case, a copointer is created which points to the RHS data. Of course,  
for an
RHS to reference remote data it must be a coarray reference or a
copointer-dereference expression (next section). For instance, the  
two statements both create copointers, one pointing to a local array  
and one
pointing to an array on another image:

         integer, dimension(:), copointer :: p7, p8  ! copointer to  
array of integer
         integer, dimension(10), cotarget :: a3[*]   ! coarray of  
array of integer
         p7 => a3                                    ! copointer to  
a3's local array
         p8 => a3[9]                                 ! copointer to  
a3 on image 9

(3) When a pointer assignment statement's RHS is an ordinary (i.e.  
pointer, the local pointer cannot be copied as-is into the LHS  
because its
type is not correct. Instead, the pointer is converted into a  
copointer and
assigned to the LHS; this is a form of copointer creation. For instance:

         integer, dimension(:), pointer :: r  ! pointer to array of  
         r => a3                              ! creates pointer to  
a3's local array
         p7 => r                              ! converts local  
pointer to copointer

(4) A copointer is copied when a pointer assignment statement's RHS is
already a copointer. Given the previous declarations, the following  
copies an existing copointer:

         p7 => p8


Copointers may be "dereferenced" to get a data reference that can be  
used in
either RHS or LHS contexts. In general the data reference is remote, so
loading from it and storing into it require communication with another
image. For this reason, CAF 2.0 requires copointers to be explicitly
dereferenced via a new "co-dereference operator" ([ ]) to indicate this
communication cost in the source code. This is in contrast to Fortran's
implicit dereferencing of ordinary pointers. For instance, the  
introduced variable 'p7' is a copointer to array of integer, so 'p7 '  
is just
an array of integer, and the following assignments copy integers and  
arrays between this image and some other image:

         integer :: k
         integer, dimension(10) :: a4
         k = p7[ ](1)
         a4 = p7[ ]
         p7[ ](1) = a4(1)
         p7[ ] = a4

For additional expressiveness, CAF 2.0 allows a copointer to be  
implicitly when it is known that the copointer points to local data.  
indicates in the source code that the dereference operation requires no
communication. The result of an implicit dereference is undefined if the
copointer points to another image. For instance, if the value of 'p7'  
is a
copointer to this image we can write:

         k = p7(1)
         a4 = p7
         p7(1) = a4(1)
         p7 = a4


CAF 2.0 extends the pointer-related intrinsic procedures of Fortran  
2008 to
work with copointers as well. For instance, 'associated(p7)' returns a
boolean indicating whether 'p7' is associated with a target, and 'p7 =>
null()' sets 'p7' to disassociated status.

In addition, CAF 2.0 provides a new intrinsic function 'imageof' which
returns the image number to which an associated copointer points. It is
undefined if the copointer is disassociated.


Here are a few related details of CAF 2.0 semantics.

(1) A copointer value may be implicitly converted into an ordinary  
when it is known that the copointer points to local data. The result is
undefined if it points to another image. For instance, if the value  
of 'p7'
is a copointer to this image we can write:

         r => p7

(2) Fortran 2008 forbids associating an ordinary pointer with a  
remote data
reference (a coindexed object, i.e.  all or part of a coarray).  
CAF 2.0 forbids associating an ordinary pointer with the result of
dereferencing a copointer. Thus the following statement is incorrect:

         r => p7[ ]	! not allowed, even though RHS is type-compatible
                         ! with 'r' ("array of integer")

(3) As mentioned above, CAF 2.0 allows all possible combinations of  
the four
type-determining attributes. In addition to our new attributes, this  
Fortran 2008's use of existing attributes by allowing "pointer to
coarray". CAF 2.0 also eliminates Fortran 2008's restrictions on nesting
coarrays and on embedding coarrays within arrays.


CAF 2.0's copointers can be easily and efficiently implemented so that
copointer dereferencing is no more expensive than a corresponding  
reference, and typically cheaper. To add copointers to a compiler which
already implements coarrays, one has only to factor the code  
generation for a
coarray reference into two parts: a generalized address calculation to
determine which bytes are needed from which image, followed by a
communication operation to obtain those bytes across the  
interconnect. Then
the code for a copointer dereference is just the communication code,  
a copointer's representation essentially caches the result of an address

Specifically, our prototype CAF 2.0 compiler represents a copointer  
value as
a pair <i,p> where 'i' is an image number and 'p' is an ordinary Fortran
pointer valid on image 'i'. Our prototype dereferences a copointer to  
storage by sending its pointer 'p' to the image 'i' that created it,
dereferencing the pointer normally on 'i', and receiving the fetched  
bytes in
reply. This is about the same communication cost as a corresponding  
reference. On a machine whose interconnect hardware supports one-sided
communication, the CAF 2.0 runtime could decode 'p' and use the  
addresses, strides, and lengths to initiate low level hardware  

Our prototype's representation does make an assumption about the  
Fortran compiler's storage allocator: the allocator must tolerate our  
and storing of pointers beyond its reach. For instance, the allocator  
not do reference counting or garbage collection, nor storage  
compaction by
moving blocks and updating pointers, because the allocator cannot see  
copies of pointers on other images. The Fortran language does not  
any of this, and in fact all commonly used Fortran compilers satisfy our
assumption. However, a simple change of representation would permit
implementing CAF 2.0 on an allocator which doesn't satisfy the
assumption. The pointer component 'p' is replaced by an opaque handle  
which can be looked up somehow on image 'i' to yield a corresponding  
instead of sending 'p' to the remote image, one would send 'h'  
instead at the
same cost, and the rest of the implementation would be unchanged.

Bill Scherer
Research Scientist
Department of Computer Science
Rice University
Houston, Texas, USA

More information about the J3 mailing list