I want to prevent hackers to break into my users' accounts. It is often said that:
The best approach it to lockout an account temporarily after
x
failed login attempts.
I understand this and it seems like a good idea. Using IP for example is a very bad idea - there is at least one whole country NAT'ed in Asia, so IP's cannot be used for anything.
Unfortunately there comes a real issue with lockout. It reveals the info whether the account exists or not. We don't want to do this, that is why we always write "email and password do not match" or something like this.
I can't lockout non-existing accounts - otherwise I would have to store info about non-existing accounts with failed login attempts. A botnet then could lead to billions of records in my database - of nonexisting accounts.
What are the possibilities to handle this issue? To prevent brute force attacks and at the same time do not reveal the information whether an account does exist or not?
My team just tackled this exact same problem and, given how ultimately simple our solution was, it was a long road to get there.
There are so many factors that need to be taken into consideration here, most of which you have already covered. Luckily for us, all the DDoS/DoS stuff is handled by our IaaS provider so we didn't have to worry about any of that. My first recommendation would be if you aren't using such a service I recommend that you do, implementing this stuff correctly is not trivial.
With all the infrastructure being handled for us, this allowed us to focus on the application itself. Our first instinct was to look at a lockout approach, however, after some discussions and even just a simple walk through of how it would work, implementation etc. we found so many potential pitfalls in it (the IP one you mentioned being one) we decided to abandon it.
We then asked the question "what exactly is it we are trying to do here?", ultimately we want to prevent brute force attacks by both bots & hackers but at the same time maintain a good UX for genuine users....when the penny dropped we actually couldn't believe how simple it was - use a Captcha.
The implementation is pretty simple, after X failed attempts we add a Captcha to the form and force the user to verify this along with their credentials. We felt this gave us the best balance between security & usability because: