I'm using a PlainTextCorpusReader to analyse some 40 text files. The files are getting imported easily, but when I try to count the number of sentences/paragraphs/words in each of the file using .sents(), it throws an AssertionError.
The files run and form a dataframe perfectly when I avoid calculating the sentences, words. But I need this information for my analysis.
Here is the python code
nm_speeches = [ ] #Empty dataframe to store all the content of speeches
for path in paths:
corpusReader = nltk.corpus.PlaintextCorpusReader(folder, '.*\.txt')
number_of_sents = len(corpusReader.sents())
# opening the files and converting it to a dictionary
The error
AssertionError Traceback (most recent call last)
<ipython-input-44-00f3eb2ee014> in <module>()
2 for path in paths:
3 corpusReader = nltk.corpus.PlaintextCorpusReader(folder, '.*\.txt')
----> 4 number_of_sents = len(corpusReader.sents())
5 # opening the files and converting it to a dictionary
6 with open(path, encoding="utf-8") as speech_file:
~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in __len__(self)
378 if len(self._offsets) <= len(self._pieces):
379 # Iterate to the end of the corpus.
--> 380 for tok in self.iterate_from(self._offsets[-1]): pass
381
382 return self._offsets[-1]
~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in iterate_from(self, start_tok)
400
401 # Get everything we can from this piece.
--> 402 for tok in piece.iterate_from(max(0, start_tok-offset)):
403 yield tok
404
~\Anaconda3\lib\site-packages\nltk\corpus\reader\util.py in iterate_from(self, start_tok)
299 self.read_block.__name__)
300 num_toks = len(tokens)
--> 301 new_filepos = self._stream.tell()
302 assert new_filepos > filepos, (
303 'block reader %s() should consume at least 1 byte (filepos=%d)' %
~\Anaconda3\lib\site-packages\nltk\data.py in tell(self)
1366 check1 = self._incr_decode(self.stream.read(50))[0]
1367 check2 = ''.join(self.linebuffer)
-> 1368 assert check1.startswith(check2) or check2.startswith(check1)
1369
1370 # Return to our original filepos (so we don't have to throw
AssertionError:
I expect it to count the number of sentences and successfully execute.