Strange xargs Behavior with special characters within variable

Question

I'm running into strange behavior with xargs in a bash script where the replacement is not working anymore when placing it in the middle of a variable with special characters. Please see the following:

#!/bin/bash

url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"

echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}

results in:

https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 1
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 2
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 3

Please note the 1 2 and 3 did not show up but was the literal {} in the endpoint.

I determined it was the parameter because when I removed a few characters, it worked:

#!/bin/bash

url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"


echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}

https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3

Note the 1, 2, and 3 is there.

I am on MacOS. This behavior seems to be specific to the xargs on Mac. This will not even run on linux xargs with the flags passed. Would love some suggestions how to make this portable and work on all systems. Thank you.

Answer 1

From the man page:

-I replstr
        Execute utility for each input line, replacing one or more
        occurrences of replstr in up to replacements (or 5 if no -R flag
        is specified) arguments to utility with the entire line of input.
        The resulting arguments, after replacement is done, will not be
        allowed to grow beyond replsize (or 255 if no -S flag is
        specified) bytes; this is implemented by concatenating as much of
        the argument containing replstr as possible, to the constructed
        arguments to utility, up to replsize bytes.  The size limit does
        not apply to arguments to utility which do not contain replstr,
        and furthermore, no replacement will be done on utility itself.
        Implies -x.

Your output string is longer that 255 characters, so it doesn't perform the replacement. Add a -S option that's large than the maximum possible output string.

$ echo 1 2 3 | xargs -n1 -I{} -S 500 echo "$url{}$parameters" -o {}
https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3

Also remember to quote variables to prevent word splitting and globbing.

Answer 2

I ultimately went with @Charles Duffy's solution in the comments. It is best to bypass xargs altogether and use something that is compatible both on Linux and MacOS/BSD.

url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"

while IFS= read -r line; do
  echo "${url}${line}${parameters}" -o downloads/$line
done < <(printf "1.mp4\n2.mp4\n3.mp4\n")

Returns expected value:

https://myhostname.com/endpoint/1.mp4?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o downloads/1.mp4
https://myhostname.com/endpoint/2.mp4?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o downloads/2.mp4
https://myhostname.com/endpoint/3.mp4?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o downloads/3.mp4

The details I left out was the printf "1.mp4\n2.mp4\n3.mp4\n" was actually a grep command which I was easily able to replace, and the echo was a call to curl.

I highly recommend using this method over trying to fuss with making it work with xargs. It is much more straightforward and you do not have to worry about the different versions of xargs. Utilizing process substitution was the most straightforward cross-platform compatible methodology.