Testing @Default value for Beam PipelineOptions ValueProvider

Question

I would like a Dataflow template with a default value for one of the PipelineOptions parameters.

Inspired by examples online I use a ValueProvider placeholder for deferred parameter setting in my PipelineOptions "sub"-interface:

  @Default.String("MyDefaultValue")
  ValueProvider<String> getMyValue();
  void setMyValue(ValueProvider<String> value);

If I specify the parameter at runtime, the template works for launching a real GCP Dataflow job. However if I try to test not including the parameter before doing this for real:

@Rule
public TestPipeline pipeline = TestPipeline.create();
...
  
@Test
public void test() {
  PipelineOptions options = PipelineOptionsFactory.fromArgs(new String[] {...}).withValidation();
  ...
  pipeline.run(options);
}

Then when my TestPipeline executes a DoFn processElement method where the parameter is needed I get

IllegalStateException: Value only available at runtime, but accessed from a non-runtime context: 
RuntimeValueProvider{propertyName=myValue, default=MyDefaultValue}
...

More specifically it fails here in org.apache.beam.sdk.options.ValueProvider:

@Override
public T get() {
  PipelineOptions options = optionsMap.get(optionsId);
  if (options == null) {
    throw new IllegalStateException(...

One could presumably be forgiven for thinking runtime is when the pipeline is running.

Anyway, does anybody know how would I unit test the parameter defaulting, assuming the top code snippet is how it should be set up and it is supported? Thank you.

Answer 1

You can also use Flex Tempaltes and avoid all the hassles with ValueProviders.

Answer 2

I had the same problem when I was generating a Dataflow template from Eclipse, my Dataflow template receives a parameter from Cloud Composer DAG.

I got the solution from the Google Cloud documentation: https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#using-valueprovider-in-your-functions