I’m looking for recommendations for a software to store a large number of logs from daily batch jobs run in a couple of different enterprise scheduling systems. After perhaps a year, the logs no longer need to be directly accessible via these scheduling systems, but it is still important that they be stored (perhaps for three more years). The logs will come from UNIX, Windows, Oracle, z/OS, and DB2 for z/OS.
I have researched the various offerings under the vaguely-defined categories enterprise content management, enterprise output management, and document management. Many of these products appear to be focused on rich & formatted customer communications. The logs I need to store are execution logs of a technical nature, and are plain, unformatted text.
I would be glad to get recommendations of a specific solution, but also suggestions on how to narrow the search to identify those solutions that are most pertinent.
The systems management vendors all have comprehensive output management solutions that handle what you describe. CA (CA VIEW, CA DELIVER, CA SPOOL, etc), IBM (Infoprint), LRS (VPS) and many others can do this sort of thing. The more sophisticated products not only store and archive data, they can scan your logs looking for signs of errors or unusual activity and then distribute copies of just the important parts to the correct parties.
If you want to "roll your own", z/OS lets you create what amounts to a virtual printer that consumes data from a particular SYSOUT class or destination, writing the report data wherever you like. There are several ways to do it, depending on how deep your programming skills go...one of the simpler is the "External Writer" that you'll find written up in the z/OS JES2 Initialization and Tuning Guide (SA32-0991). Once you have the data, you can create files, post copies to an FTP site, write to Splunk - really the only limit is your imagination.
If you go this way, you simply change your JCL to "SYSOUT=x" (where "x" is the class or destination serviced by your external writer) and all of your data will collected by JES2 and spooled to your writer. If you want data from UNIX or other platforms, you can even define a mainframe LPD printer that lets other platforms talk to z/OS printers using the remote LP protocol, and with a little configuration, this also drives your external writer.
I'd strongly recommend one of the vendor products if you have budget - this is certainly do-able, but it's not exactly easy absent the prerequisite system programming background.