Google Cloud Vertex AI: custom-job cannot specify arg multiple times error

Question

I have a Python project using click args where an argument should be in tuple(string,float) format and can accept multiple instances of the arg.

@click.option(
    "--metric-and-lift",
    multiple=True,
    type=(str, float),
    help="The list of key metric & corresponding expected lift to detect. Provide `[metric_name] [lift]`, \
        e.g. 1% lift in net revenue as `net_revenue 0.01`",
)

I also have a gcloud command which attempts to create & run a Vertex AI job with an image of this project and args to match the tuple(string,float) format.

gcloud ai custom-jobs create \
          --region us-central1 \
          --service-account $(USERNAME)@$(GCP_PROJECT) \
          --display-name=$(DISPLAY_NAME) \
          --project=$(GCP_PROJECT) \
          --worker-pool-spec=machine-type=m1-ultramem-40,replica-count=1,container-image-uri=$(CONTAINER_IMAGE_LOCATION) \
          --command="poetry,run,power_analysis_cli" \
          --args=--metric-and-lift=impression,0.01,--metric-and-lift=click,0.01,--metric-and-lift=install,0.01,--metric-and-lift=revenue,0.01,--metric-and-lift=spend,0.01,--metric-and-lift=net_rev,0.01

I am running into an issue with the gcloud CLI where it is erroring out due to values being specified multiple times. It looks like the command is evaluating 0.01 individually several times instead of clumping it together with the preceding string (ex. impression,0.01).

ERROR: (gcloud.ai.custom-jobs.create) argument --args: "0.01" cannot be specified multiple times

Interestingly enough, I can launch this job successfully if I append 0's to the left hand side of the float value (ex. --metric-and-lift=revenue,00.01) so that none of the floats exactly match one another.

Is there a better way of handling this case in Vertex AI? It would be great to keep the existing --metric-and-lift=$(STRING),$(FLOAT) format since that converts to the pattern python click args expects when running the gcloud Vertex AI command.

Answer 1

Sorry but if I understand it will set at 0.01 at all the proceeding argument in the command ? so completely removing it after the first argument is okay ?

I replicated the error but it changed when I removed the 0.01 on the next arguments sample:

gcloud ai custom-jobs create \ 
--args=--metric-and-lift=impression,0.01 \ 
--args=--metric-and-lift=click \
--args=--metric-and-lift=install

I removed some of the commands/flags since I cannot reproduce your data but am able to reproduce the error using just what I posted above.

Answer 2

If anyone is curious for a solution, I ended up using config files to launch the Vertex AI jobs instead of passing in --args directly into the command. Using a config file did not have the same limitation in repeating arguments.

ex.launcher.yml

workerPoolSpecs:
  machineSpec:
    machineType: ${MACHINE_TYPE}
  replicaCount: 1
  containerSpec:
    imageUri: ${CONTAINER_IMAGE}
    args:
    - --metric-and-lift
    - impression
    - "0.01"
    - click
    - "0.01"
    - install
    - "0.01"

ex. gcloud Vertex AI command

gcloud ai custom-jobs create \
      --region $(GCP_LOCATION) \
      --service-account $(SERVICE_ACCOUNT) \
      --display-name=$(DISPLAY_NAME) \
      --project=$(GCP_PROJECT) \
      --command="poetry,run" \
      --config=./scripts/launcher.yml

Furthermore, I used yq to generate the yml file programmatically before calling the gcloud command. This freed users from needing to write the launcher config file manually, which allowed the workflow to be automated.