Traceback (most recent call last):
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/torch/serialization.py", line 965, in save
_save(
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/torch/serialization.py", line 1266, in _save
zip_file.write_record(name, storage, num_bytes)
RuntimeError: [enforce fail at inline_container.cc:857] . PytorchStreamWriter failed writing file data/228: file write failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/liuhao/anaconda3/envs/nnunet/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 266, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 207, in run_training
nnunet_trainer.run_training()
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1387, in run_training
self.on_epoch_end()
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1142, in on_epoch_end
self.save_checkpoint(join(self.output_folder, 'checkpoint_latest.pth'))
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1176, in save_checkpoint
torch.save(checkpoint, filename)
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/torch/serialization.py", line 964, in save
with _open_zipfile_writer(f) as opened_zipfile:
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/torch/serialization.py", line 798, in __exit__
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:659] . unexpected pos 317681600 vs 317681488
Exception in thread Thread-3 (results_loop):
Traceback (most recent call last):
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Exception in thread Thread-2 (results_loop):
Traceback (most recent call last):
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/home/liuhao/anaconda3/envs/nnunet/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message