I am trying to train a YOLO model on a custom dataset using YOLOv8 Ultralytics, using M1 MacBook Pro MPS acceleration which is supported by PyTorch but I am getting the following error while executing the train command.
Command:
!yolo task=detect mode=train model=yolov8s.pt data={HOME}/datasets/data.yaml epochs=25 device=mps
Error Message:
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/cli.py", line 56, in cli
func(cfg)
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 79, in decorated_main
return task_function(cfg_passthrough)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 207, in train
model.train(**cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/model.py", line 193, in train
self.trainer.train()
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
self._do_train(int(os.getenv("RANK", -1)), world_size)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 301, in _do_train
self.loss, self.loss_items = self.criterion(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 77, in criterion
return self.compute_loss(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 168, in __call__
targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Is there any way to get around this issue and use MPS for training?
I have tried looking for this issue on PyTorch's GitHub conversation, but got no simple solution.
Edit: I have set HYDRA_FULL_ERROR to 1, and this is the updated error traceback:
Error executing job with overrides: ['task=detect', 'mode=train', 'model=yolov8s.pt', 'data=/Users/aditya_nemiwal/Documents/C++/Python/WiDS_Week3_3/datasets/data.yaml', 'epochs=25', 'device=mps']
Traceback (most recent call last):
File "/opt/homebrew/bin/yolo", line 8, in <module>
sys.exit(cli())
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main
_run_hydra(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
raise ex
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
return func()
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/opt/homebrew/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/opt/homebrew/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/cli.py", line 56, in cli
func(cfg)
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 79, in decorated_main
return task_function(cfg_passthrough)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 207, in train
model.train(**cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/model.py", line 193, in train
self.trainer.train()
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
self._do_train(int(os.getenv("RANK", -1)), world_size)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 301, in _do_train
self.loss, self.loss_items = self.criterion(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 77, in criterion
return self.compute_loss(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 168, in __call__
targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.