How to make request to google Vertex AI API for MultiModal embeddings

Question

Endgoal: Retrieve Embeddings to store them in a Vector DB

I'm using Nextjs API routes (which is just code that runs in a NODE.js env). The exported POST function is just the POST method handler.

In this handler, I to make requests to this model

import { UploadImageResponseBody } from "@/types";
/* 
importing this because I intend to migrate however,
I'm having a hard time understanding the docs, 
feel free to give an example using the client with the same model if it's easier
*/
import { PredictionServiceClient } from "@google-cloud/aiplatform";


/**
 * Despite the project being in `us-east1`,
 * `us-central1` is the only one that works
 */
const LOCATION = "us-central1";
const PROJECT_ID = "...";
const MODEL = "multimodalembedding";
const RPC_CALL = "predict";
const G_CLOUD_ACCESS_TOKEN ="...";

const REQUEST_URL = `https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL}@001:${RPC_CALL}`;

type Instances = { image: { bytesBase64Encoded: string } }[];

async function getEmbedding(instances: Instances) {
  try {
    const response = await fetch(REQUEST_URL, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${G_CLOUD_ACCESS_TOKEN}`,
      },
      body: JSON.stringify({
        instances,
      }),
    });

    return response;
  } catch (error) {
    console.error(error);
    return null;
  }
}

export const POST = async (req: Request) => {

  const formData = await req.formData();
  const imageBlobs = formData.getAll("images") as Blob[];

  const base64Images = await Promise.allSettled(
    imageBlobs.map(async (imgBlob) => {
      const binaryArr = await imgBlob.arrayBuffer();
      const buffer = Buffer.from(binaryArr);
      const base64EncodedImage = buffer.toString("base64");
      const fileName = (imgBlob as File).name;
      return { base64EncodedImage, fileName };
    })
  );

  const successfulEncodings = base64Images.filter(
    (imgRes) => imgRes.status === "fulfilled"
  ) as PromiseFulfilledResult<{
    base64EncodedImage: string;
    fileName: string;
  }>[];

  const instances = successfulEncodings.map((imgRes) => {
    return {
      image: {
        bytesBase64Encoded: imgRes.value.base64EncodedImage,
      },
    };
  });


  const embeddings = await getEmbedding(instances);

  // ...rest
 

  return response;
};

I'm currently receiving the following response

{
  embeddings: Response {
    [Symbol(realm)]: null,
    [Symbol(state)]: {
      aborted: false,
      rangeRequested: false,
      timingAllowPassed: true,
      requestIncludesCredentials: true,
      type: 'default',
      status: 500,
      timingInfo: [Object],
      cacheState: '',
      statusText: 'Internal Server Error',
      headersList: [HeadersList],
      urlList: [Array],
      body: [Object]
    },
    [Symbol(headers)]: HeadersList {
      [Symbol(headers map)]: [Map],
      [Symbol(headers map sorted)]: null
    }
  }
}
getEmbeddingTime: 50.414s /* <- not part of the response,
 this is my log because i saw the `timingAllowPassed`
 flag set to true */

I had it working fine with text embeddings which leads me to believe it could be something with how I'm handling/encoding the images. I know internal server error / 500 usually means not my fault but I'm willing to bet there's something I'm doing wrong.

If it helps to see my Frontend/Client code here it is

"use client";
import { ChangeEvent, useState, EventHandler, MouseEvent } from "react";
import { Button } from "../Button";
import axios from "axios";
import ImageFileList from "./ImageFileList";

export const FileUpload: React.FC = () => {
  const [selectedImageFiles, setSelectedImageFiles] = useState<File[]>([]);

  const handleFileChange: EventHandler<ChangeEvent<HTMLInputElement>> = (
    event
  ) => {
    const { files } = event.target;

    if (!files || !files.length) return;

    const fileArr = Array.from(files);
    const MAX_BYTES_2MB = 2_000_000;

    for (const f of fileArr) {
      if (f.size > MAX_BYTES_2MB) {
        const warning = `File ${f.name} is too large. Please upload a file less than 2MB`;
        alert(warning);
      }
    }

    const acceptedFiles = fileArr.filter((f) => f.size <= MAX_BYTES_2MB);
    const newFiles = [...selectedImageFiles, ...acceptedFiles];
    setSelectedImageFiles(newFiles);
  };

  const handleUploadImagesClick: EventHandler<
    MouseEvent<HTMLButtonElement>
  > = async () => {

    if (selectedImageFiles.length === 0) {
      alert("Please select some images to upload");
      return;
    }

    const formData = new FormData();
    for (let i = 0; i < selectedImageFiles.length; i++) {
      formData.append("images", selectedImageFiles[i]);
    }

    try {
      const response = await axios.postForm("/api/images", formData);
    } catch (error) {
      console.error(error);
    }
  };

  const hasFilesSelected =
    !!selectedImageFiles && selectedImageFiles.length > 0;

  return (
    <div >
      <label >
        <span>
          {hasFilesSelected ? (
            <ImageFileList files={selectedImageFiles} />
          ) : (
            "Upload Images"
          )}
        </span>
        <input
          type="file"
          onChange={handleFileChange}
          accept=".jpg,.jpeg,.png"
        />
      </label>
      <Button
        onClick={handleUploadImagesClick}
        disabled={!hasFilesSelected}
      >
        Upload
      </Button>
    </div>
  );
};