[J3] FORM TEAM statement NEW_INDEX= specifier & failed images

John Reid John.Reid at stfc.ac.uk
Sun May 12 07:30:00 EDT 2019


Nathan Weeks via J3 wrote:
> Hi all,
> Thanks for the helpful clarification (and identifying where the standard 
> is unclear). I'll note that this issue impacts the first failed-images 
> example in section C.6.8 of the Fortran 2018 standard, so there is 
> motivation for clarification in the standard itself.

I think we were a bit hasty in choosing to assign failed images to new 
teams in a processor-dependent manner. We definitely want the C.6.8 
example to work. It was always a design objective that following an 
image failure, it would be possible to form a new team of active images 
and continue the calculation there. We don't want any failed images in 
the team because we want to be able to test for newly failed images.

Anyway,, para 5 says

5 Successful execution of a CHANGE TEAM statement performs an implicit 
synchronization of all images of the new team that is identified by 
team-value. All active images of the new team shall execute the same 
CHANGE TEAM statement. On each image of the new team, execution of the 
segment following the CHANGE TEAM statement is delayed until all other 
images of that team have executed the same statement the same number of 
times in the original team.

It is clearly expected that all images of the team are active. The 
adjective "active" is not used in the first and third sentences. It 
should be deleted from the second, for consistency.

To go back to your question:

"What happens in the case where an image specifies both NEW_INDEX= and
STAT= in a FORM TEAM statement, and the image index specified for
NEW_INDEX= turns out to be greater than the number of images in the
new team due to image failure during the execution of FORM TEAM?",

I think this is an error condition. Note that in C.6.8 the NEW_INDEX 
values are carefully set.



> One drawback of modifying the standard to assign stopped/failed images 
> to new teams in a processor-dependent manner when FORM TEAM includes 
> NEW_INDEX= is that the resulting team wouldn't be able to use collective 
> routines (although I suppose the program could check if FORM TEAM sets 
> STAT=STAT_FAILED_IMAGE, and in that case execute a FORM TEAM without 
> NEW_INDEX= to cull non-active images, before trying FORM TEAM with 
> NEW_INDEX= again).  But I suppose an advantage is that it would give 
> applications additional flexibility?
> I'm particularly interested in this issue as I'm implementing a 
> prototype of OpenCoarrays that includes failed images support for 
> teams.  FWIW, I have a research poster that advertises a (somewhat 
> dated) prototype implementation (where FORM TEAM excludes NEW_INDEX=):
> https://lib.dr.iastate.edu/cs_conf/49/
> (the Docker image, which emits a lot of debugging info, is based on 
> OpenCoarrays ~2.4.0 + failed images extensions).
> Given the current implementation (MPI + ULFM under the hood), it would 
> be easier to either (1) just not create the team, or (2) exclude 
> non-active images from new teams created with FORM TEAM (..., 
> NEW_INDEX=...), and consider NEW_INDEX= to behave (in a 
> processor-dependent way) like the MPI_Comm_split() "key" value, used 
> simply to determine ordering of the images in the resulting team.  But I 
> don't think it would be insurmountably difficult to implement a 
> failed/stopped-images-fill-in-gaps (where necessary?) approach (though I 
> admittedly haven't through about it too hard).
> --
> Nathan
> On Thu, May 9, 2019 at 10:13 AM <j3-request at mailman.j3-fortran.org 
> <mailto:j3-request at mailman.j3-fortran.org>> wrote:
>     Message: 3
>     Date: Thu, 9 May 2019 14:12:57 +0000
>     From: Bill Long <longb at cray.com <mailto:longb at cray.com>>
>     To: General J3 interest list <j3 at mailman.j3-fortran.org
>     <mailto:j3 at mailman.j3-fortran.org>>
>     Subject: Re: [J3] FORM TEAM statement NEW_INDEX= specifier & failed
>              images
>     Message-ID: <A2E9865F-EB94-4DA4-81E0-B8F2E75C6E62 at cray.com
>     <mailto:A2E9865F-EB94-4DA4-81E0-B8F2E75C6E62 at cray.com>>
>     Content-Type: text/plain; charset="utf-8"
>      > On May 8, 2019, at 8:07 PM, Malcolm Cohen via J3
>     <j3 at mailman.j3-fortran.org <mailto:j3 at mailman.j3-fortran.org>> wrote:
>      >
>      > <<<
>      > I?m not convinced that Jon?s example is unreasonable.
>      > >>>
>      >
>      > It?s definitively Not Standard Conforming!
>     That, I think, we can agree.  The way the standard is written, the
>     only way for an image to join a new team is to execute FORM TEAM. 
>     NOTE: Exactly the same issue under discussion applies to STOPPED
>     images as well as FAILED ones.  An image that is stopped will not
>     execute FORM TEAM either.  Stopped images are a bit different in
>     that data on stopped images can still be accessed. Such accesses are
>     still possible through the image index of the stopped image in the
>     parent team.
>      >
>      > <<<
>      > We currently allow the value of an image index to be larger than
>     the number of active images in a team.
>      > >>>
>      >
>      > We currently allow the user to drink hot coffee too, which is
>     about as relevant.
>      >
>      > Seriously, he suggested the image index being larger than
>     NUM_IMAGES().
>     Only if the failed (and stopped) images are not part of the new
>     teams, which they are not by the argument above. NUM_IMAGES()
>     includes failed and stopped images in a team.  If we want to require
>     that the  sum of the num_images() values for the new teams is the
>     same as num_images() for the parent [something that I think would be
>     intuitive], then the processor could assign failed and stopped
>     images  to new teams in a processor-dependent way that ?fills in?
>     gaps in the image index values if NEW_INDEX is specified, or adds
>     them at the end of the image index range if there are no gaps (or if
>     NEW_IMAGE is not specified).
>     So the question is what we really want.  Either the failed and
>     stopped images  of the parent are discarded from the new teams (the
>     current wording), or whether they are included in the new teams. 
>     Which option is more useful and less error-prone for the
>     programmer?   I think the arguments / examples presented so far
>     suggest that including the stopped and failed images is preferred.
>     Others, of course, might have the opposite view.  But this is the
>     question that should be discussed, not whether the example is
>     conforming to the current standard (since we have a clear answer for
>     that already).
>     Cheers,
>     Bill
>      > That is a horse of a completely different colour.  I assert that
>     this is rather likely to lead to disastrous outcomes, like accessing
>     data of non-existent images (but image number less than
>     num_images!), or syncing with such, etc.  It?s not that unlikely
>     that the runtime system itself has tables sized by the number of
>     images, which could lead to sync all not syncing with the images
>     whose numbers are out of range, using random garbage instead of
>     network addresses, or indeed seg faulting.
>      >
>      > <<<
>      > Otherwise, we are basically requiring that the user go through an
>     exercise to track down which images are failed
>      > >>>
>      >
>      > If only we had an intrinsic that returned the failed image numbers?
>      >
>      > <<<
>      > before executing FORM TEAM and then communicating these boundary
>     image index numbers at runtime
>      > >>>
>      >
>      > Indeed, the user had better do this if he cares about his data,
>     as the failed images Have Already Lost His Data!  Carefully forming
>     a boundary image setup without knowing what data is available is a
>     complete waste of time. (He might not even *have* any boundary data
>     left, or all the images that would have belonged in ?team 3?, which
>     might be a critical team for his algorithm to work, might have failed.)
>      >
>      > If he doesn?t care about the data why bother running the program
>     at all.
>      >
>      > Or if your concern is how he is going to recover without ?going
>     non-standard?, we would need to add that capability. There are two
>     obvious possibilities: (1) NEW_INDEX is simply inoperative when an
>     image has failed; (2) invalid NEW_INDEX is allowed, at least when
>     images have failed, but doesn?t form a team.  There are many details
>     to be worked out, but we could certainly do something about it. 
>     STAT_INVALID_NEW_INDEX would be one possibility, but something more
>     lightweight could also work.
>      >
>      > In principle I support ?doing something?, as long as we can all
>     agree on what the something is, as otherwise there is a race
>     condition anyway in between checking for failed images and executing
>      >
>      > Cheers,
>      > --
>      > ..............Malcolm Cohen, NAG Oxford/Tokyo.
>     Bill Long longb at cray.com <mailto:longb at cray.com>
>     Principal Engineer, Fortran Technical Support &   voice:  651-605-9024
>     Bioinformatics Software Development                      fax: 
>     651-605-9143
>     Cray Inc./ 2131 Lindau Lane/  Suite 1000/  Bloomington, MN  55425

More information about the J3 mailing list