Running AzureML experiment programatically with data, error loading data

Question

I am following this tutorial to send a script to AzureML.

Getting an error on the last line of the step that loads data:

from azureml.core import Workspace

ws = Workspace.get(name="ws_name", subscription_id='asdfadsfasdf232323', resource_group='my-rg')

def_blob_store = ws.get_default_datastore()
def_blob_store.upload_files(files = ['dummy_file.pkl'],
                       target_path = 'script-run/',
                       overwrite = True,
                       show_progress = True)

def_blob_store = ws.get_default_datastore()
input_data = Dataset.File.from_files(def_blob_store.path('script-run/dummy_file.pkl')).as_named_input('input').as_mount()

The error I get:

ImportError: Missing required package "azureml-dataset-runtime", which can be installed by running: "env_name\Scripts\python.exe" -m pip install azureml-dataset-runtime --upgrade

I'm using pipenv and installed azureml-core using the pipenv install command. I have seen that the AzureML library doesn't seem to play so well with pipenv. I did try to install as per the message above but still getting the same error. Tried

import azureml.datasets.runtime

But that doesn't work. Running this on Python 3.9.6.

By the way, at this point, I don't care so much whether I load the file from the script or access it from the Data section of AzureML. I already have this same file loaded there, so a method to access it in the ScriptRunConfig step would also work.

Answer 1

One possible way to access the input file in the ScriptRunConfig step is by keeping the file in your source directory with your script and access as root file.

Then you can use the source directory path for your file

With this I was able to execute the job. For other details, you can also check this thread.