aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [9,0,0], thread: [124,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [9,0,0], thread: [125,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [9,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [9,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
1/300 2.34G 4 9.454 4.133 67 640: 54%|█████▍ | 86/158 [00:16<00:13, 5.37it/s]
Traceback (most recent call last):
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/tal.py", line 74, in forward
return self._forward(pd_scores, pd_bboxes, anc_points, gt_labels, gt_bboxes, mask_gt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/tal.py", line 102, in _forward
mask_pos, align_metric, overlaps = self.get_pos_mask(
^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/tal.py", line 124, in get_pos_mask
align_metric, overlaps = self.get_box_metrics(pd_scores, pd_bboxes, gt_labels, gt_bboxes, mask_in_gts * mask_gt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/tal.py", line 143, in get_box_metrics
bbox_scores[mask_gt] = pd_scores[ind[0], :, ind[1]][mask_gt] # b, max_num_obj, h*w
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/icislab/volume1/lishendegou/yolov12-main/train-11.py", line 9, in <module>
model.train(data='/icislab/volume1/lishendegou/yolov12-main/ultralytics/cfg/datasets/pear.yaml',
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/engine/model.py", line 808, in train
self.trainer.train()
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/engine/trainer.py", line 207, in train
self._do_train(world_size)
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/engine/trainer.py", line 381, in _do_train
self.loss, self.loss_items = self.model(batch)
^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/nn/tasks.py", line 111, in forward
return self.loss(x, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/nn/tasks.py", line 321, in loss
return self.criterion(preds, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/loss.py", line 233, in __call__
_, target_bboxes, target_scores, fg_mask, _ = self.assigner(
^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/yolov12-main/ultralytics/utils/tal.py", line 75, in forward
except torch.OutOfMemoryError:
^^^^^^^^^^^^^^^^^^^^^^
File "/icislab/volume1/lishendegou/anaconda3/envs/yolo12/lib/python3.11/site-packages/torch/__init__.py", line 1938, in __getattr__
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'OutOfMemoryError'
最新发布