Python HMM tagger _pickle.PicklingError attribute lookup estimator on nltk.tag.hmm failed

Question

I am trying to evaluate Python HMM tagger on PTB corpus, then output to file.

train_data = treebank.tagged_sents()[:3523]
test_data = treebank.tagged_sents()[3523:]
hmm_pos_tagger = hmm.HiddenMarkovModelTagger.train(train_data)
print('HMM = %0.5f' % hmm_pos_tagger.evaluate(test_data))

output = open('tagger_hmm.pickle', 'wb', -1)
pickle.dump(hmm_pos_tagger, output)
output.close()

Here is the following output:

HMM = 0.89920
...
line 56, in <module>
    pickle.dump(hmm_pos_tagger, output)
_pickle.PicklingError: Can't pickle <function HiddenMarkovModelTagger._train.<locals>.estimator at 0x05A59300>: attribute lookup estimator on nltk.tag.hmm failed

I already tried TnT & Brill tagger but there was no error. What am I doing wrong?

Answer 1

I tried to used dill instead of pickle and it worked.

dill: https://pypi.python.org/pypi/dill