I have a C program that uses SGR ("Select Graphic Rendition") ANSI terminal escape sequences to produce color output.
However, if output is being redirected to a text file, then the escape sequences should not be output since they'd be seen as "garbage" in the generated file. To handle this, one can use the isatty(3) function: output color only if it returns true. Indeed, this is what programs like grep(1) do.
But if output is being being redirected to a Unix pipe, e.g., | less that, at least in my case, is primarily what someone would pipe the output of my program to, then the escape sequences should be output since less(1) correctly handles them and prints in color.
After some testing using isatty(3) and fstat(3), I've produced this table where C represents my C program, the "Should?" column is whether I should output color, and the remaining columns are T/F as the return value of isatty or fstat(STDOUT_FILENO):
COMMAND Should? isatty ISCHR ISFIFO ISREG
======== ======= ====== ===== ====== =====
C T T T F F
C > file F F F F >>T<<---- The interesting case.
C | less T F F T F
Hence, it would seem that I want to output color except when ISREG on the stat struct on stdout is T.
Are there any caveats with doing this? If there are, is there a better way to get what I want?
Note: I also have an option similar to grep's --color option to specify either auto or always (or none simply by not specifying any option at all). My hope in using fstat and ISREG is to make the auto case better the majority of the time.