Text classification & topic modelling

507 views Asked by At

For a huge set of articles, I want to get the topic models with weightage assigned to different topics & within topics, what are the weightage for different sub-topics. For example, if I feed an article which falls in both Business & Technology domain, then the program's output shuold be something like this :-

  • 0.593 Business ( 0.438 - Marketing , 0.375 - Companies, 0.062 - Office Work)
  • 0.148 Technology ( 0.500 Technology by type, 0.250 - High_technology Business Districts, 0.250 - Technology Companies)
  • 0.111 Society ( 0.333 - Organizations, 0.333 - Technology in Society, 0.333 - Labor)

What's the best open-source language processing programs available that can successfully do this stuff?

3

There are 3 answers

0
jgloves On

I would give NLTK a try, but scikit-learn, even though it has a steeper learning curve than NLTK, is probably a better bet. It's much more configurable.

http://scikit-learn.org/stable/documentation.html

0
skaz On

You can classify using the open-source NLTK Toolkit.

0
Sir Cornflakes On

There are several programs to do a part of this task, for a starter I recommend mallet. Note that any topic modeling program gives you the topics in the form you want, i.e.,

 ( 0.438 - Marketing , 0.375 - Companies, 0.062 - Office Work)

but the labels (in this example Business) you need to assign yourself. Mallet also gives you a decomposition of the text to the topics (identified by numbers, not by the labels).