Allocate resources for Kubeflow pipeline using pipeline params

Question

I would like to be able to create a Kubeflow pipeline that allows users to set the allocated resources for a run. The end result would be something like this:

Example of Kubeflow "Create Run" UI with ability to set resource allocation.

Definition of the pipeline params is possible; however, the syntax of the pipeline params does not match the validation regex used by Kubeflow to preprocess its YAML definition.

---

As an example, using the values paramters in the screen shot, I can hard-code the resources allocated to the pipeline by adding this to the pipeline's YAML definition:

resources:
    limits: {nvidia.com/gpu: 1}
    requests: {cpu: 16, memory: 32G}

However, what I want to do is to use the pipeline's paramaters to define these allocations for each run. Something like:

resources:
    limits: {nvidia.com/gpu: '{{inputs.parameters.gpu_limit}}'}
    requests: {cpu: '{{inputs.parameters.cpu_request}}', memory: '{{inputs.parameters.memory_request}}'}

When I use the second definition of pipeline resources, creation of the pipeline fails because Kubeflow cannot to parse these resource parameter as the input parameter syntax '{{input.parameters.parameter}}' does not match the regular expression ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$.

{
    "error_message":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'",
    "error_details":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'"
}

---

Has anyone found a workaround for this issue, or am I trying to force Kubeflow to do something it isn't built for? Defining and using pipeline parameters like I have in the second example works for other portions of the pipeline definition (eg args or commands to run in the Docker container).

Answer 1

This just can be done in current version of kubeflow pipelines. It is a limitation, but you cannot change resources from the pipeline itself.