Redirecting xargs input as stdin stream instead of argument

Question

Sometimes it's useful to write an xargs command that begins with xargs -I {} 'echo {} | [rest of command]' in order to redirect the argument as a pipe.

however, for large arguments, you will encounter xargs: argument line too long.

How do I tell xargs to redirect straight to the os input pipe for [rest of command] so that I avoid the above issue when using large arguments?

---

Reproducer:

# create a file with very long base64-encoded lines
seq 1 10 |
  xargs -I {} sh -c '
    openssl rand -base64 21000000 | tr -d "'"\n"'"; echo
  '> out.b64

# now, try to pipe each line into a new instance of a program
# xargs fails at doing this because the lines are too large
cat out.b64 |
  xargs -I {} sh -c "echo {} | rest-of-command"

Answer 1

Nothing you're doing calls for xargs at all, anywhere.

#!/usr/bin/env bash

for ((i=0; i<10; i++)); do
  openssl rand -base64 21000000 | tr -d '\n'
  echo
done >out.b64

while IFS= read -r line; do
  { rest-of-command; } <<<"$line"
done <out.b64

The { rest-of-command; } <<<"$line" could also be written as printf '%s\n' "$line" | rest-of-command or < <(printf '%s\n' "$line") rest-of-command. I don't recommend echo for the reasons given in Why is printf better than echo?.

xargs is a limited-purpose tool: it transports content from stdin to command-line arguments. If that's not what you mean to accomplish, it's the wrong tool for the job.

Answer 2

Edited to include a better solution

There's a better way of accomplishing what you're doing, and that's just to pipe it directly to the rest of the command inside of a for loop:

for x in {1..10}; do
    { openssl rand -base64 21000000 | tr -d '\n' ; echo; } |
        [rest of command]
done

No need to store it in a variable, dump it to a file, or read it out again.

However...

If you do need to read the string into a variable:

for x in {1..10}; do
    bigstring=$(openssl rand -base64 21000000 | tr -d '\n')
    echo "$bigstring" |
        [rest of command]
done

This is just unnecessary memory usage though. It also runs at about 25% of the speed of the first example.

---

Old solution:

Just replace xargs with a while read -r line; ...;done loop:

my-string-generator |while IFS= read -r mystring; do echo "$mystring" | [rest of command]; done

Setting IFS to an empty string prevents it from eliminating leading and trailing whitespace from the input.

You can also choose the record separator by using read -d STRING (in this case a NULL byte, \0):

my-string-generator |while IFS= read -d $'\0' -r mystring; do echo "$mystring" | [rest of command]; done

Note: Using NULL as a separator is mostly useful for strict security when dealing with filenames; while exceedingly rare to see in the wild, technically *NIX filenames can contain newline characters.