Data Connection - downloading files from multiple URLs in one sync

Question

How can I use Data Connection to download files from a large list of URLs as part of a single sync?

I want to be able to do this in parallel, since I'll be downloading 1–2,000 new files per day. I also want all the files to be stored in a single dataset.

Answer 1

This is supported using the magritte-rest-plugin. Configre the source as a map of magritte rest sources. The format of the source is:

type: magritte-rest-v2
  sourceMap:
    source_name_1:
      source-object
    source_name_2:
      source-object2

which when filled out, might look like:

type: magritte-rest-v2sourceMap:
    my_api:
      type: magritte-restauthToken: "{{token}}"url: "https://some-api.com/"another_api:
      type: magritte-restauthToken: "{{other_token}}"url: "https://some-other_api.com/"