I read from railscast #401 that having long living connections actually takes up the threads in rails server. And puma server can optimally take on may be 20-30 conncurrent connections.. Does that mean I will have to spin off many Heroku instances for my rails application to support (e.g. 1K - 5K people chatting?).
Or is it not affecting because all the chat connections are directly connceted to Pubnub's server?
Yes, you are correct, your end users (browser clients) will connect (publish/subscribe) through PubNub without involving your server.
In other words, you do not need to have your Rails/server act as a subscriber. Just serve up the web page with the appropriate PubNub JavaScript for you clients to chat with each other directly without involving your server.
This image from one of the examples in the links below is a good illustration
Here are some links that should be helpful: