NLTK PlainTextCorpusReader shows AssertionError when counting sentences/paras in multiple text files

240 views Asked by At

I'm using a PlainTextCorpusReader to analyse some 40 text files. The files are getting imported easily, but when I try to count the number of sentences/paragraphs/words in each of the file using .sents(), it throws an AssertionError.

The files run and form a dataframe perfectly when I avoid calculating the sentences, words. But I need this information for my analysis.

Here is the python code

nm_speeches = [ ]  #Empty dataframe to store all the content of speeches
for path in paths:
    corpusReader = nltk.corpus.PlaintextCorpusReader(folder, '.*\.txt')
    number_of_sents = len(corpusReader.sents())
    # opening the files and converting it to a dictionary

The error

AssertionError                            Traceback (most recent call last)
<ipython-input-44-00f3eb2ee014> in <module>()
      2 for path in paths:
      3     corpusReader = nltk.corpus.PlaintextCorpusReader(folder, '.*\.txt')
----> 4     number_of_sents = len(corpusReader.sents())
      5     # opening the files and converting it to a dictionary
      6     with open(path, encoding="utf-8") as speech_file:

~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in __len__(self)
    378         if len(self._offsets) <= len(self._pieces):
    379             # Iterate to the end of the corpus.
--> 380             for tok in self.iterate_from(self._offsets[-1]): pass
    381 
    382         return self._offsets[-1]

~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in iterate_from(self, start_tok)
    400 
    401             # Get everything we can from this piece.
--> 402             for tok in piece.iterate_from(max(0, start_tok-offset)):
    403                 yield tok
    404 

~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in iterate_from(self, start_tok)
    299                 self.read_block.__name__)
    300             num_toks = len(tokens)
--> 301             new_filepos = self._stream.tell()
    302             assert new_filepos > filepos, (
    303                 'block reader %s() should consume at least 1 byte (filepos=%d)' %

~\Anaconda3\lib\site-packages\nltk\data.py in tell(self)
   1366             check1 = self._incr_decode(self.stream.read(50))[0]
   1367             check2 = ''.join(self.linebuffer)
-> 1368             assert check1.startswith(check2) or check2.startswith(check1)
   1369 
   1370         # Return to our original filepos (so we don't have to throw

AssertionError: 

I expect it to count the number of sentences and successfully execute.

0

There are 0 answers