why the bleu score is zero for this pair even though they are similar

Question

why I'm getting 0 score even though sentences are similar. Please refer the code below

from nltk.translate.bleu_score import sentence_bleu

score = sentence_bleu(['where', 'are', 'economic', 'networks'], ['where', 'are', 'the', 'economic', 'network'])

print(score)

score value is '0' if u run the above code

Answer 1

This has to do with how you hand the references to the method.

Check out the documentation or more specifically this sample of how to use it.

References have to be a list (a list of all reference translations to compare the hypothesis with). So in your case a list containing the one tokenized reference translation.

Use it like this and it will work:

score = sentence_bleu([['where', 'are', 'economic', 'networks']], ['where', 'are', 'the', 'economic', 'network'])
> 9.283142785759642e-155

Although I have to admit I have struggled with the exact same problem before and this is not very intuitive on NLTKs part. The documentation even states that references has to be of type list(list(str)) but then doesn't check if this is the case and instead returns a misleading result when used incorrectly.