Encountered an internal AutoML error- ClientException: Message: No objects to concatenate

Question

I am trying to implement Hierarchical time series forecasting on azureautoml pipelines.

I followed this notebook for implementation https://github.com/Azure/azureml-examples/blob/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-hierarchical-timeseries/auto-ml-forecasting-hierarchical-timeseries.ipynb

While I ran training pipeline on compute instance it worked, but when I am running the same on compute cluster it breaks at hts-proportion-calculation part.

This is the error I am getting,

system error:

Encountered an internal AutoML error. Error Message/Code: ClientException. Additional Info: ClientException:
      Message: No objects to concatenate
      InnerException: None
      ErrorResponse
{
    "error": {
        "message": "No objects to concatenate"
    }
}

logs :

Loading arguments for scenario proportions-calculation
adding argument --input-medatadata
adding argument --hts-graph
adding argument --enable-event-logger
Input arguments dict is {'--input-medatadata': '/mnt/azureml/cr/j/85509be625484b6caa3c1d97b7ab2e33/cap/data-capability/wd/INPUT_automl_training_workspaceblobstore/azureml/17ca5ae7-7269-4246-888f-e781071e3f5c/automl_training', '--hts-graph': '/mnt/azureml/cr/j/85509be625484b6caa3c1d97b7ab2e33/cap/data-capability/wd/INPUT_hts_graph_workspaceblobstore/azureml/a2c1b15a-c895-41e8-b6a6-1ca37ebe9e77/hts_graph', '--enable-event-logger': None}

Unknown file to proceed outputs.txt
processing: outputs.txt with type None.
Cleaning up all outstanding Run operations, waiting 300.0 seconds
3 items cleaning up...
Cleanup took 0.001676321029663086 seconds

Traceback (most recent call last):
  File "proportions_calculation_wrapper.py", line 47, in <module>
    runtime_wrapper.run()

  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_many_models/automl_pipeline_step_wrapper.py", line 63, in run
    self._run()

  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 44, in _run
    proportions_calculation(self.arguments_dict, self.event_logger, script_run=self.step_run)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 173, in proportions_calculation
    proportion_files_list, forecasting_parameters.time_column_name, graph.label_column_name
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/azureml/train/automl/runtime/_hts/proportions_calculation.py", line 92, in calculate_time_agg_sum_for_all_files

    df = pd.concat(pool.map(concat_func, files_batches), ignore_index=True)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/core/reshape/concat.py", line 304, in concat
    sort=sort,
  File "/azureml-envs/azureml_e34d0633ffc4cb2fa25d91e3da5f59be/lib/python3.7/site-packages/pandas/core/reshape/concat.py", line 351, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate

Please let me know how can I resolve this issue ?

Answer 1

We are able to run the sample successfully using the compute cluster as given below.

from azureml.core.compute import ComputeTarget, AmlCompute
    
    # Name your cluster
    compute_name = "hts-compute"
    
    
    if compute_name in ws.compute_targets:
        compute_target = ws.compute_targets[compute_name]
        if compute_target and type(compute_target) is AmlCompute:
            print("Found compute target: " + compute_name)
    else:
        print("Creating a new compute target...")
        provisioning_config = AmlCompute.provisioning_configuration(
            vm_size="STANDARD_D16S_V3", max_nodes=20
        )
        # Create the compute target
        compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)
    
        # Can poll for a minimum number of nodes and for a specific timeout.
        # If no min node count is provided it will use the scale settings for the cluster
        compute_target.wait_for_completion(
            show_output=True, min_node_count=None, timeout_in_minutes=20
        )
    
        # For a more detailed view of current cluster status, use the 'status' property
        print(compute_target.status.serialize())

Answer 2

This error was incurred as Iteration timeout was not less than experiment timeout , but the system error & logs are a kind of misleading.

 df = pd.concat(pool.map(concat_func, files_batches), ignore_index=True)

logs was pointing to pandas "No objects to concatenate"

This error can be overcome by setting iterationtimeout value less than experimenttime out value. I had set iteration_timeout_minutes=60 which caused the error.

automl_settings = AutoMLConfig(
    task="forecasting",
    primary_metric="normalized_root_mean_squared_error",
    experiment_timeout_hours=1,
    label_column_name=label_column_name,
    track_child_runs=False,
    forecasting_parameters=forecasting_parameters,
    pipeline_fetch_max_batch_size=15,
    model_explainability=model_explainability,
    n_cross_validations="auto",  # Feel free to set to a small integer (>=2) if runtime is an issue.
    cv_step_size="auto",
    # The following settings are specific to this sample and should be adjusted according to your own needs.
    iteration_timeout_minutes=10,
    iterations=15,
)