(j3.2006) (j3.2005) Re: question on token replacement/concatenation

[Malcolm Cohen]

I wrote:
>>>>  3 %% .14159 %% E %% + %% 0
> ...
>>   I've asked this before, but I don't remember the answer.  What sort of
>>   lexical token is the "E"?  It doesn't seem to match anything in the list.
> 
> Before macro token concatenation, it's a name.
> 
>   <letter> [ <digit> | <letter> | _ ]...
> 
> After macro token concatenation, the 5 tokens ( "digits", "decimal",
> "name", "operator", "digits" ) combine to form one token, so the "E"
> no longer maps to a whole token but is merely one character in a token.

Dick Hendrickson said:
> OK, but this sort of limits how we can use the %% to build up things.
> The syntax for "names", and the other lexical tokens, is fairly
> restrictive.  The two cases I think might be hard to do are
> building up structure names and adding kinds to constants.
> 
> Suppose I want to build up component names, so that sometimes I
> refer to A%B and sometimes to A.  My niave attempt would be
>        A%%p1
> to get A, I'd invoke the macro with p1 as a null macro argument
> for A%B, I'd try to set p1 to %B.
> But, I can't do that because %B isn't a name, nor any other
> lexical token.

There does not seem to be a problem here.
You are not trying to build up a token: A%B is three tokens, not one,
so in any case you do not want to concatenate anything.

In other words, just

   A p1

does what you are trying to do with A%%p1.

> Similarly for constants, if I sometimes want to postfix a kind parameter
> I'd try
>        3.14%%kind
> where "kind" would sometimes be null and other times be _4 or
> _workingprecision.  Again, I can't do this.

Ugh.  Yes, we have a problem here, but the problem is because of the
problematic way we have defined 3.14_WP as being one token instead of
three.  Tokens are meant to be the basic lexical building blocks - and
this just isn't (WP has to be a name, with associated semantics etc.,
which means we're treating it as a token in its own right).

Actually, I had spotted this before (many years before) and it has been
somewhat of a sore point with users who want to write things like
  3.14 _ WP
for readability reasons (in fact the NAG compiler used to allow that until
someone else complained that we didn't detect the spaces inside the "token"
3.14_WP).

> So, is it worth trying to add some sort of "except leading underscores
> and %s are allowed in the names" condition to the concatenation
> rules?  Or are these cases too small to worry about this time?
> Or, have I misunderstand something?

W.r.t. 3.14_WP, I think that the right fix is to correct our faulty
definition of a token.  If we want that to have no "user effect" (i.e.
the compiler writers not to have to change anything) we can forbid
blanks in between the tokens of a literal constant.  Seems pretty
straightforward to do.  (If the other vendors want to allow blanks
around the underscore that's fine by me too.)

We could call a literal constant without a kind parameter an "unkinded"
literal constant... we already use them quite a lot in i/o for example.

Cheers,
-- 
........................Malcolm Cohen (malcolm at nag-j.co.jp), Nihon NAG, Tokyo.