Disable paralled build for a specific target

Question

I need to disable parallel run for a single target. It is a test that verifies if program doesn't create some random or incorrectly named files. Any other file that is build in the meantime fails this test.

I found this advice on SCons FAQ:

Use the SideEffect() method and specify the same dummy file for each target that shouldn't be built in parallel. Even if the file doesn't exist, SCons will prevent the simultaneous execution of commands that affect the dummy file. See the linked method page for examples.

However, this is useless, as it would prevent parallel build of any two targets not only the test script.

Is there any way to prevent parallel build of one target while allowing it for all others?

Answer 1

We discussed this in the scons discord, and came up with an example which will setup synchronous test runners which will make sure no other tasks are running when the test is run.

This is the example SConstruct from the github example repo:

import SCons

# A bound map of stream (as in stream of work) name to side-effect
# file. Since SCons will not allow tasks with a shared side-effect
# to execute concurrently, this gives us a way to limit link jobs
# independently of overall SCons concurrency.
node_map = dict()

# A list of nodes that have to be run synchronously.
# sync node ensures the test runners are syncrhonous amongst
# themselves.
sync_nodes = list()

# this emitter will make a phony sideeffect per target
# the test builders will share all the other sideeffects making
# sure the tests only run when nothing else is running.
def sync_se_emitter(target, source, env):
    name = str(target[0])
    se_name = "#unique_node_" + str(hash(name))
    se_node = node_map.get(se_name, None)
    if not se_node:
        se_node = env.Entry(se_name)
        # This may not be necessary, but why chance it
        env.NoCache(se_node)
        node_map[se_name] = se_node
        for sync_node in sync_nodes:
            env.SideEffect(se_name, sync_node)
    env.SideEffect(se_node, target)
    return (target, source)

# here we force all builders to use the emitter, so all
# targets will respect the shared sideeffect when being built.
# NOTE: that the builders which should be synchronous must be listed
# by name, as SynchronousTestRunner is in this example
original_create_nodes = SCons.Builder.BuilderBase._create_nodes
def always_emitter_create_nodes(self, env, target = None, source = None):
    if self.get_name(env) != "SynchronousTestRunner":
        if self.emitter:
            self.emitter = SCons.Builder.ListEmitter([self.emitter, sync_se_emitter])
        else:
            self.emitter = SCons.Builder.ListEmitter([sync_se_emitter])
    return original_create_nodes(self, env, target, source)
SCons.Builder.BuilderBase._create_nodes = always_emitter_create_nodes


env = Environment()
env.Tool('textfile')
nodes = []

# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SCons.Builder.Builder(
    action=SCons.Action.Action([
        "sleep 1",
        "echo Starting test $TARGET",
        "sleep 5",
        "echo Finished test $TARGET",
        'echo done > $TARGET'],
    None))

# this emitter connects the test runners with the shared sideeffect
def sync_test_emitter(target, source, env):
    for name in node_map:
        env.SideEffect(name, target)
    sync_nodes.append(target)
    return (target, source)

env['BUILDERS']["SynchronousTestRunner"].emitter = SCons.Builder.ListEmitter([sync_test_emitter])

# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")

for i in range(50):
    nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))

for i in range(50, 76):
    node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
    env.Depends(node, "test.out")
    nodes.append(node)

for i in range(76, 100):
    node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
    env.Depends(node, "test2.out")
    nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))

env.Program('out', nodes)

Answer 2

This solution is based in dmoody256's answer. The underlying concept is the same but the code should be easier to use and it's ready to be put in the site_scons directory to not obfuscate SConstruct itself.

site_scons/site_init.py:

# Allows using functions `SyncBuilder` and `Environment.SyncCommand`.
from SyncBuild import SyncBuilder

site_scons/SyncBuild.py:

from SCons.Builder import Builder, BuilderBase, ListEmitter
from SCons.Environment import Base as BaseEnvironment

# This code allows to build some targets synchronously, which means there won't
# be anything else built at the same time even if SCons is run with flag `-j`.
#
# This is achieved by adding a different dummy values as side effect of each
# target. (These files won't be created. They are only a way of enforcing
# constraints on SCons.)
# Then the files that need to be built synchronously have added every dummy
# value from the entire configuration as a side effect, which effectively
# prevents it from being built along with any other file.
#
# To create a synchronous target use `SyncBuilder`.

__processed_targets = set()
__lock_values = []
__synchronous_nodes = []

def __add_emiter_to_builder(builder, emitter):
    if builder.emitter:
        if isinstance(builder.emitter, ListEmitter):
            if not any(x is emitter for x in builder.emitter):
                builder.emitter.append(emitter)
        else:
            builder.emitter = ListEmitter([builder.emitter, emitter])
    else:
        builder.emitter = ListEmitter([emitter])

def __individual_sync_locks_emiter(target, source, env):
    if not target or target[0] not in __processed_targets:
        lock_value = env.Value(f'.#sync_lock_{len(__lock_values)}#')
        env.NoCache(lock_value)
        env.SideEffect(lock_value, target + __synchronous_nodes)
        __processed_targets.update(target)
        __lock_values.append(lock_value)
    return target, source

__original_create_nodes = BuilderBase._create_nodes
def __create_nodes_adding_emiter(self, *args, **kwargs):
    __add_emiter_to_builder(self, __individual_sync_locks_emiter)
    return __original_create_nodes(self, *args, **kwargs)
BuilderBase._create_nodes = __create_nodes_adding_emiter

def _all_sync_locks_emitter(target, source, env):
    env.SideEffect(__lock_values, target)
    __synchronous_nodes.append(target)
    return (target, source)

def SyncBuilder(*args, **kwargs):
    """It works like the normal `Builder` except it prevents the targets from
    being built at the same time as any other target."""
    target = Builder(*args, **kwargs)
    __add_emiter_to_builder(target, _all_sync_locks_emitter)
    return target

def __SyncBuilder(self, *args, **kwargs):
    """It works like the normal `Builder` except it prevents the targets from
    being built at the same time as any other target."""
    target = self.Builder(*args, **kwargs)
    __add_emiter_to_builder(target, _all_sync_locks_emitter)
    return target
BaseEnvironment.SyncBuilder = __SyncBuilder

def __SyncCommand(self, *args, **kwargs):
    """It works like the normal `Command` except it prevents the targets from
    being built at the same time as any other target."""
    target = self.Command(*args, **kwargs)
    _all_sync_locks_emitter(target, [], self)
    return target
BaseEnvironment.SyncCommand = __SyncCommand

SConstruct (this is adapted dmoody256's test that does the same thing as the original):

env = Environment()
env.Tool('textfile')
nodes = []

# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SyncBuilder(
    action=Action([
        "sleep 1",
        "echo Starting test $TARGET",
        "sleep 5",
        "echo Finished test $TARGET",
        'echo done > $TARGET'],
    None))

# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")

for i in range(50):
    nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))

for i in range(50, 76):
    node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
    env.Depends(node, "test.out")
    nodes.append(node)

for i in range(76, 100):
    node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
    env.Depends(node, "test2.out")
    nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))

env.Program('out', nodes)

After creating site_scons/site_init.py and site_scons/SyncBuild.py, you can just use function SyncBuilder or method Environment.SyncCommand in any SConstruct or SConscript file in the project without any additional configuration.