在使用nnunet训练时,出现如下报错jzuser@vpc87-3:~/Work_dir/Gn/pystudy/NnuNet$ nnUNetv2_train 3 2d 0
nnUNetv2_train 3 2d 1
nnUNetv2_train 3 2d 2
nnUNetv2_train 3 2d 3
nnUNetv2_train 3 2d 4
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e727680>)
2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428fe062c0>)
2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428f79c100>)
2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428fe062c0>)
2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428f79c100>)
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>)
2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>)
2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>)
2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>)
2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>)
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>)
2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>)
2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>)
2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>)
2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>)
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>)
2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c2a95b80>)
2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>)
2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c2a95b80>)
2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>)
2025-08-27 09:23:09.655113: unpacking dataset...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 581, in save
format.write_array(fid, arr, allow_pickle=allow_pickle,
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 754, in write_array
array.tofile(fp)
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 17, in _convert_to_npy
np.save(npz_file[:-4] + "_seg.npy", a['seg'])
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 579, in save
with file_ctx as fid:
OSError: [Errno 28] No space left on device
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1203, in run_training
self.on_train_start()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 788, in on_train_start
unpack_dataset(self.preprocessed_dataset_folder, unpack_segmentation=True, overwrite_existing=False,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 33, in unpack_dataset
p.starmap(_convert_to_npy, zip(npz_files,
File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
OSError: [Errno 28] No space left on device
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training
nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args
nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__
maybe_mkdir_p(self.output_folder)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p
os.makedirs(directory, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_1'
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training
nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args
nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__
maybe_mkdir_p(self.output_folder)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p
os.makedirs(directory, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_2'
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training
nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args
nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__
maybe_mkdir_p(self.output_folder)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p
os.makedirs(directory, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_3'
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training
nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args
nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__
maybe_mkdir_p(self.output_folder)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p
os.makedirs(directory, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_4'
jzuser@vpc87-3:~/Work_dir/Gn/pystudy/NnuNet$ nnUNetv2_train 3 2d 0
nnUNetv2_train 3 2d 1
nnUNetv2_train 3 2d 2
nnUNetv2_train 3 2d 3
nnUNetv2_train 3 2d 4
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 11:19:51.857624: unpacking dataset...
2025-08-27 11:20:40.435668: unpacking done...
2025-08-27 11:20:40.436964: do_dummy_2d_data_aug: False
2025-08-27 11:20:40.438331: Creating new 5-fold cross-validation split...
2025-08-27 11:20:40.441395: Desired fold for training: 0
2025-08-27 11:20:40.441591: This split has 104 training and 27 validation cases.
2025-08-27 11:20:40.498886: Unable to plot network architecture:
2025-08-27 11:20:40.500076: No module named 'hiddenlayer'
2025-08-27 11:20:40.552532:
2025-08-27 11:20:40.553642: Epoch 0
2025-08-27 11:20:40.554312: Current learning rate: 0.01
Exception in background worker 0:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 2:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 1:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 3:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 4:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
using pin_memory on device 0
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 11:20:48.364002: unpacking dataset...
2025-08-27 11:20:51.509973: unpacking done...
2025-08-27 11:20:51.510635: do_dummy_2d_data_aug: False
2025-08-27 11:20:51.511523: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json
2025-08-27 11:20:51.511727: The split file contains 5 splits.
2025-08-27 11:20:51.511775: Desired fold for training: 1
2025-08-27 11:20:51.511804: This split has 105 training and 26 validation cases.
2025-08-27 11:20:51.521027: Unable to plot network architecture:
2025-08-27 11:20:51.521092: No module named 'hiddenlayer'
2025-08-27 11:20:51.526808:
2025-08-27 11:20:51.526880: Epoch 0
2025-08-27 11:20:51.526956: Current learning rate: 0.01
Exception in background worker 1:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 2:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 0:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
using pin_memory on device 0
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 11:20:58.407072: unpacking dataset...
2025-08-27 11:21:01.513495: unpacking done...
2025-08-27 11:21:01.514885: do_dummy_2d_data_aug: False
2025-08-27 11:21:01.517129: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json
2025-08-27 11:21:01.517678: The split file contains 5 splits.
2025-08-27 11:21:01.517827: Desired fold for training: 2
2025-08-27 11:21:01.517945: This split has 105 training and 26 validation cases.
2025-08-27 11:21:01.529522: Unable to plot network architecture:
2025-08-27 11:21:01.529716: No module named 'hiddenlayer'
2025-08-27 11:21:01.540922:
2025-08-27 11:21:01.541166: Epoch 0
2025-08-27 11:21:01.541448: Current learning rate: 0.01
Exception in background worker 2:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 3:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
using pin_memory on device 0
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 11:21:08.460438: unpacking dataset...
2025-08-27 11:21:11.615700: unpacking done...
2025-08-27 11:21:11.616486: do_dummy_2d_data_aug: False
2025-08-27 11:21:11.618074: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json
2025-08-27 11:21:11.618454: The split file contains 5 splits.
2025-08-27 11:21:11.618557: Desired fold for training: 3
2025-08-27 11:21:11.618626: This split has 105 training and 26 validation cases.
2025-08-27 11:21:11.628197: Unable to plot network architecture:
2025-08-27 11:21:11.628319: No module named 'hiddenlayer'
2025-08-27 11:21:11.635873:
2025-08-27 11:21:11.636014: Epoch 0
2025-08-27 11:21:11.636152: Current learning rate: 0.01
Exception in background worker 1:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 3:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 2:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
using pin_memory on device 0
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Using device: cuda:0
/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
/home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn(
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}}
2025-08-27 11:21:18.424697: unpacking dataset...
2025-08-27 11:21:21.510880: unpacking done...
2025-08-27 11:21:21.511596: do_dummy_2d_data_aug: False
2025-08-27 11:21:21.513083: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json
2025-08-27 11:21:21.513473: The split file contains 5 splits.
2025-08-27 11:21:21.513583: Desired fold for training: 4
2025-08-27 11:21:21.513662: This split has 105 training and 26 validation cases.
2025-08-27 11:21:21.521894: Unable to plot network architecture:
2025-08-27 11:21:21.522025: No module named 'hiddenlayer'
2025-08-27 11:21:21.530473:
2025-08-27 11:21:21.530618: Epoch 0
2025-08-27 11:21:21.530759: Current learning rate: 0.01
Exception in background worker 2:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 97, in load_case
seg = np.load(entry['data_file'][:-4] + "_seg.npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 4:
No data left in file
Exception in background worker 1:
mmap length is greater than file size
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 477, in load
return format.open_memmap(file, mode=mmap_mode,
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 965, in open_memmap
marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order,
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/_core/memmap.py", line 289, in __new__
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
ValueError: mmap length is greater than file size
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 3:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Exception in background worker 0:
No data left in file
Exception in background worker 5:
No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
Traceback (most recent call last):
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__
return self.generate_train_batch()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load
raise EOFError("No data left in file")
EOFError: No data left in file
using pin_memory on device 0
Traceback (most recent call last):
File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training
nnunet_trainer.run_training()
File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message如何解决
最新发布