problem arises when using mosestokenizer (https://github.com/luismsgomes/mosestokenizer, installed through pypi https://pypi.org/project/mosestokenizer/) on windows platform (tried multiple windows machines).
Although there was no new releases during last month, before it used to work fine. I believe it has something to do with communication between python and perl.
>> MosesSentenceSplitter('en')
stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "...\.venv\lib\site-packages\mosestokenizer\sentsplitter.py",
line 82, in __init__
super().__init__(argv)
File "...\.venv\lib\site-packages\toolwrapper.py",
line 64, in __init__
self.start()
File "...\.venv\lib\site-packages\toolwrapper.py",
line 102, in start
self.proc = subprocess.Popen(
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
In order to reproduce it all it takes is to create some moses* class object:
from mosestokenizer import MosesSentenceSplitter
splitsents = MosesSentenceSplitter('en') # and here error appear.
Installing perl fixed the error.
Also, as @ikegami mentioned, This should have
$| = 1
, otherwise you will get stdbuf warnings.