How to handle .for loops in make ?

[Marc Espie <Marc_(_dot_)_Espie_(_at_)_liafa_(_dot_)_jussieu_(_dot_)_fr>]

I'm facing a problem in make which I'd like to finish solving, but I still
have a few options.


Background: our make does support loops in Makefile, akin to macro expansion,
e.g., you write:

.for A in a b c
zoinx-$A:
	@echo $A
.endfor

and this will expand to
zoinx-a:
	@echo a
zoinx-b:
	@echo b
zoinx-c:
	@echo c

This is handled through a mechanism similar to include files:
make's parser does have a stack of `live' Makefiles it is currently
proceeding. Those can either be real, opened files, or strings in memory.
(see parse.c)

The for expander deals with strings in memory (see for.c)
Whenever the parser sees a .for line, it passes control to the for expander.

That one builds a structure from it: for loop starting at that line, variable
name is A, variable values is a list of a, b, c.

Then the parser accumulates the lines in the loop until it sees the matching
endfor, at which point we've got a loop template:
zoinx-$A:
	@echo $A

ready to be expanded.

The expansion is done by a special form of Var_Subst (in var.c), that will
only expand a given variable.

So, for.c expands the template for each variable in turn, and pushes the
resulting string on the stack of Makefiles to parse.

Since this is a stack, the list of values is expanded in reverse:
so that we push
zoinx-c:
	@echo c

zoinx-b:
	@echo b

zoinx-a:
	@echo a


(For nested loops, only the external loop is  expanded directly.  The internal
loop will be expanded when the parse meets it)

The problem with that scheme is memory-consumption: make uses a special
Buffer structure (in buf.c) to build variable-size strings.  Starts with
a good guess of a size, and accumulate characters in the buffer, eventually
doubling its size if this doesn't fit.
Here, Var_Subst uses one separate buffer for each variable expansion, which
can be very wasteful for larger loops structure.

It would be more efficient to accumulate all expansions into the same buffer,
instead of pushing several strings on the parse stack.

The only problem lies with line number reports.  The parser needs some
extra knowledge to correctly number for loops. Right now, since each expansion
is pushed separately, it is enough to know the line number at start of loop,
since "\n" will mean business as usual.  But this is no longer enough
if the whole expansion is pushed at once.

There are a few ways to solve the problem:
1/ do nothing, assume for loops are wasteful. I'm loathe to follow that
direction, since for loops CAN be efficient.
2/ provide a debug switch (-dl for `line numbers') that gives the slower
scheme, with correct line numbers. Without that switch, for.c can take
shortcuts, and lose line numbers.
3/ ensure correct line numbers reports somehow...

`Somehow' is a bit intricate.  In the parser, Parse_FromString currently
makes the assumption that 0 never occurs in the string to parse.  We could
use 0 as a marker to `reset the line number to some value'.  Then we
build a corresponding array of line numbers.  This means a few changes to
the variant of VarSubst specific to for loops to handle this correctly, along
with a few tweaks to parse.c... There is an added benefit that we could make
sure line numbers are always correct. Right now, if you manage to put
a \n within a variable expansion, you will get erroneous line numbers...

I'd like ideas.  That scheme does work, but it looks like overkill.
Something simpler could be handy.   Baring that, I'll probably settle for 1 or
2...
-- 
	Marc Espie		
|anime, sf, juggling, unicycle, acrobatics, comics...
|AmigaOS, OpenBSD, C++, perl, Icon, PostScript...
| `real programmers don't die, they just get out of beta'