Which tools for implementing distributed logging?

462 views Asked by At

I'm trying to pick a set of tools to implement logging (starting with just error logging at first) for the different components of our system. The system consists of a set of client implementations (iOS, web MVC etc) and a backend.

I'd like to be able to store every 4xx/5xx my backend returns, including the context, as well as whatever happens on the clients when an issue occurs. Ideally this would be correctly serialized so that I can rely on the timestamps to establish order of the events, but I realize this is not a trivial problem, so I'm ok with not fully solving it at first.

Additionally, I'd like this to be as maintenance-free as possible for now. I could spin up a web app backed by a Postgres DB very quickly (note: I'm not expressing any preference as far as SQL vs NoSQL here), but I don't want to invest time in configuration management, DB growth management, security etc. I'd much rather outsource this to an external SaaS provider for now, and hand-roll this in the future if the need arises.

The problem is that I'm not super familiar with the various DBaaS/BaaS offerings out there and would love to hear your opinion. I might be missing a fantastic offering and do things the hard way out of ignorance.

I've used Parse before, and I'm also getting the impression that DynamoDB would be a good fit as well. Any other options I should look into? Thank you.

2

There are 2 answers

0
Stuart On

Recently we've been using Papertrail (http://papertrailapp.com) which is a SaaS app that takes care of the storage, archiving and the interface for searching through the log.

Implementing it was pretty easy, in each of our existing logging frameworks (log4j & log4net) we just had to add a syslog appender to start sending the log messages through. We've also setup some saved searches to send alerts for things like 5xx's.

As for being able to correlate the events, it can get pretty dodgey trying to rely on timestamps to correlate the messages especially once the volume of messages and source devices starts to build up. You're better off adding an identifier each of your messages as part of a standard format (either a user or transaction #) to help identify related messages. Once that's done the search in Papertrail makes seeing the whole picture of the transaction a snap.

0
guerilla On

Check apiOmat. It's a BaaS hosted in Germany and has generated SDKs to communicate with the server. Different programming languages like Android, iOS, JavaScript, etc. are supported, so it will be very easy to store your logs on a central place.