I'm using mmaction2 to test some action recognition and action detection models. I run mani tests for many of the available action recognition models on kinetics400 and everything worked properly.
Then I tried to run tests on some of the available action detection models (slowest, lfb) on AVA dataset. Unfortunately in these cases, at the end of all the test epochs, the execution fails.
(openmmlabGPU) daitan3@daitan3-workstation:~/decondDrive/pythonProjects/mmaction2$ python tools/test.py configs/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.py https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb_20220906-4c5b9f25.pth --dump result_lfb_nl.pkl >> result_lfb_nl.txt
Downloading: "https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb_20220906-4c5b9f25.pth" to /home/daitan3/.cache/torch/hub/checkpoints/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb_20220906-4c5b9f25.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 138M/138M [00:15<00:00, 9.03MB/s]
.
.
.
.
11/17 12:18:09 - mmengine - INFO - Epoch(test) [50140/50252] eta: 0:00:03 time: 0.0450 data_time: 0.0016 memory: 1873
11/17 12:18:10 - mmengine - INFO - Epoch(test) [50160/50252] eta: 0:00:03 time: 0.0356 data_time: 0.0016 memory: 1873
11/17 12:18:10 - mmengine - INFO - Epoch(test) [50180/50252] eta: 0:00:02 time: 0.0317 data_time: 0.0016 memory: 1873
11/17 12:18:11 - mmengine - INFO - Epoch(test) [50200/50252] eta: 0:00:01 time: 0.0447 data_time: 0.0017 memory: 1873
11/17 12:18:12 - mmengine - INFO - Epoch(test) [50220/50252] eta: 0:00:01 time: 0.0438 data_time: 0.0016 memory: 1873
11/17 12:18:13 - mmengine - INFO - Epoch(test) [50240/50252] eta: 0:00:00 time: 0.0400 data_time: 0.0017 memory: 1873
==> 1.19164 seconds to Reading GT results
==> 19.6223 seconds to Reading Detection results
Killed
(openmmlabGPU) daitan3@daitan3-workstation:~/decondDrive/pythonProjects/mmaction2$ Process ForkPoolWorker-9:
Process ForkPoolWorker-10:
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header)
File "/home/daitan3/miniconda3/envs/openmmlabGPU/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
(openmmlabGPU) daitan3@daitan3-workstation:~/decondDrive/pythonProjects/mmaction2$
I tried to avoid the dump and the output redirecting but the result is the same.
The test has been executed under tux. Outside tux it just result in freezing the system that needs to be restarted.
The system is an Ubuntu 20.04 running on an i7 with 16 GB RAM and an NVIDIA TITAN RTX 24 Gb
Can someone help me in finding the problem?