Define a recursively expanded variable using a variable whose name is computed from a simply expanded variable

Question

I've run into a difficulty in defining generic rules for a non-recursive make system.

Background

For further reading, rather than me reproducing too much existing material, see this earlier question, which covers the ground pretty well, and previously helped me when constructing this system.

For the make system I am constructing, I want to define dependencies between system components - e.g. component A depends on component B - and then leave the make system to ensure that any products of the B build process are built before they will be needed by build steps for A. It's slightly wasteful due to the granularity (some unneeded intermediates may be built), but for my use case it meets a comfortable balance point between ease-of-use and build performance.

A difficulty the system must deal with is that the order of makefile loading cannot be controlled - indeed, should not matter. However because of this, a component defined in an early-loaded makefile may depend on a component defined in a not-yet-read makefile.

To allow common patterns to be applied across all the component-defining makefiles, each component uses variables such as: $(component_name)_SRC. This is a common solution for non-recursive (but recursively included) make systems.

For information on GNU make's different types of variables, see the manual. In summary: simply expanded variables (SEVs) are expanded as a makefile is read, exhibiting behaviour similar to an imperative programming language; recursively expanded variables (REVs) are expanded during make's second phase, after all the makefiles have been read.

The Problem

The specific issue arises when trying to turn a list of depended-on components into a list of the files those components represent.

I've distilled my code down to this runnable example which leaves out a lot of the detail of the real system. I think this is sufficiently simple to demonstrate the issue without losing its substance.

rules.mk:

$(c)_src              := $(src)
$(c)_dependencies     := $(dependencies)

### This is the interesting line:
$(c)_dependencies_src := $(foreach dep, $($(c)_dependencies), $($(dep)_src))

$(c) : $($(c)_src) $($(c)_dependencies_src)
        @echo $^

Makefile:

.PHONY: foo_a.txt foo_b.txt bar_a.txt hoge_a.txt

### Bar
c            := bar
src          := bar_a.txt
dependencies :=

include rules.mk

### Foo
c            := foo
src          := foo_a.txt foo_b.txt
dependencies := bar hoge

include rules.mk

### Hoge
c            := hoge
src          := hoge_a.txt
dependencies := bar

include rules.mk

These will run to give:

$ make foo
foo_a.txt foo_b.txt bar_a.txt
$

hoge_a.txt is not included in the output, because at the time that foo_dependencies is defined as a SEV, hoge_src doesn't exist yet.

Expansion after all the makefiles have been read is a problem REVs should be able to solve and I did previously try defining $(c)_dependencies_src as a REV, but that doesn't work either because $(c) is then expanded at substitution time, not definition time, so it no longer holds the correct value.

In case anyone is wondering why I am not using target-specific variables, I am concerned that the application of the variable to all the prerequisites of the target described in the manual will cause an unwanted interaction between the rules for different components.

I'd like to know:

1. Is there a solution to this specific issue? (i.e. is there a simple way to make that line achieve what I want it to?)
2. Is there a more typical way of building a make system like this? (i.e. a single make instance, loading components from multiple makefiles and defining dependencies between those components.)
3. If there are multiple solutions, what are the trade-offs between them?

A final comment: As I've written my question, I've begun to realise that there might be a solution possible using eval to construct a REV definition, however as I couldn't find this problem covered anywhere else on SO I thought worthwhile asking the question anyway for the sake of future searchers, plus I'd like to hear more experienced users' thoughts on this or any other approaches.

Answer 1

The short answer is there's no good solution for the question you are asking. It's not possible to stop expansion of a variable partway through and defer it until later. Not only that, but because you use the variable in a prerequisite list even if you could get the value of the $(c)_dependencies_src variable to contain just the variable references you wanted, in the very next line they would be completely expanded as part of the prerequisites list so it wouldn't gain you anything.

There's only one way to postpone the expansion of prerequisites and that's to use the secondary expansion feature. You would have to do something like:

$(c)_src              := $(src)
$(c)_dependencies     := $(dependencies)

.SECONDEXPANSION
$(c) : $($(c)_src) $$(foreach dep, $$($$@_dependencies), $$($$(dep)_src))
        @echo $^

(untested). This side-steps the issue with $(c)_dependencies_src by just not defining it at all and putting it into the prerequisite list directly, but as a secondary expansion.

As I wrote in my comment above, though, I personally would not design a system that worked like this. I prefer to have systems where all the variables are created up-front using a namespace (typically prepending the target name) and then at the end, after all variables have been defined, including a single "rules.mk" or whatever that will use all those variables to construct the rules, most likely (unless all your recipes are very simple) using eval.

So, something like:

targets :=

### Bar
targets += bar
bar_c            := bar
bar_src          := bar_a.txt
bar_dependencies :=

### Foo
targets += foo
foo_c            := foo
foo_src          := foo_a.txt foo_b.txt
foo_dependencies := bar hoge

### Hoge
targets += hoge
hoge_c            := hoge
hoge_src          := hoge_a.txt
hoge_dependencies := bar

# Now build all the rules
include rules.mk

And then in rules.mk you would see something like:

define make_c
$1 : $($1_src) $(foreach dep, $($1_dependencies), $($(dep)_src))
        @echo $$^
endif

$(foreach T,$(targets),$(eval $(call make_c,$T)))

You can even get rid of the settings for target if you are careful about your variable names, by adding something like this into rules.mk:

targets := $(patsubst %_c,%,$(filter %_c,$(.VARIABLES)))

In order to allow the same components in different targets you'd just need to add more to the namespace to differentiate the two different components.