I've used bigbluebutton in a recent project in order to build a web conferencing platform. I was unhappy with the quality of the service and noticed a delay between the voice and the video.
Technically bigbluebutton used Flex integrated with a red5 server that sends the voice to Asterisk for voice mixing (Konference) and treats the video locally by implementing a codec in java.
Bigbluebutton is very well done and the sources are really well written. As I understood that Flex it self limited to choice of media server implementation such as the impossibility to implement UDP sokets etc.
So my question is that,another way than Flex to implement video chat application in browser. Is JavaFX a good solution ?
No. Perhaps in the future (e.g. JavaFX 3.0 next year), but not now.
JavaFX currently has no in-built camera or microphone support.
To build such support yourself you would need to rely on a non-JavaFX library (such as lti-civil) and then interface it into a JavaFX application. As JavaFX is just Java and JavaFX apps controls can be embedded in Swing applets, this is all possible, so you could consider it. To get to the point of an end-to-end solution with all of the features of BigBlueButton would be a great deal of work.