Is there a way to catch the openssh key based authentication failure? I am looking for an option, where some custom "error handling like alarms" to be raised whenever a ssh key based authentication fails for a certain user.
I know that the key based authentication is done by openssh, I wanted to know if there is any option, where in I can plug in my error handling/reporting code which will be called when the authentication fails.
Any pointers on how this can be done? Any way for this failure to caught by some application process?
No. OpenSSH does not support any way to plug your error handling. But the failed login attempts are recorded in error log or in audit (on RHEL/Fedora), where you can simply differentiate the user, authentication type and react accordingly.
This is very similar as the way
fail2ban
does it (it reads error logs). But I don't know what are you trying to achieve. Failed publickey authentication is quite common (most of the clients are sending public keys tests to all servers they are trying to access and therefore you can see a lot of these events).