tensorflow报错解决 Variable conv1/conv1_1/weights already exist, or was not created,

本文主要介绍了在使用TensorFlow时遇到的'Variable already exists'错误的解决过程。作者通过调整variable_scope的reuse参数,解决了训练集和验证集训练操作中变量冲突的问题。在模型定义时,正确设置reuse=True和reuse=False,确保了变量的正确共享和避免了冲突。

首先我还是要喷一下坑了我好久的垃圾文章,在大多数与我标题相关的文章当中,99%的文章只写了一句话,在程序的开头加上tf.reset_default_graph(),而且连具体位置都没写,查过stackoverflow之后,才知道应该加载import tensorflow as tf。

但是问题来了,应该有不少同学跟我一样的情况,加了之后并没有作用,那么进入正题,在我尝试多次之后,终于发现并解决了我程序里的bug,这里写一下,给大家做一个思路上的参考。

我使用slim自己复现了一个cnn模型,在我的训练模型得文件当中,我定义了两个操作,第一个是训练集的训练操作,一个是验证集的训练操作,代码如下(以下是bug解决之后的代码,先放在这里以下两行是正确的的代码!!!!

with tf.variable_scope("VGG_19"):
            pre,soft_max,predictions=VGG_19(inputs=train_images,is_training=True,NUM_CLASSES=NUM_CLASSES)
            tf.get_variable_scope().reuse_variables()
            _,_,val_pre=VGG_19(inputs=val_images,is_training=False,NUM_CLASSES=NUM_CLASSES)

 

这里关注reuse参数,这是我一开始根据报错,加到函数里的,一开始,我在第一个操作里,将reuse设置为True

pre,soft_max,predictions=VGG_19(inputs=train_images,reuse=True,is_training=T

WARNING:tensorflow:From /root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term 2025-07-26 19:47:00.316548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-07-26 19:47:00.379323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:39:00.0 name: NVIDIA GeForce RTX 4090 computeCapability: 8.9 coreClock: 2.52GHz coreCount: 128 deviceMemorySize: 23.55GiB deviceMemoryBandwidth: 938.86GiB/s 2025-07-26 19:47:00.379583: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2025-07-26 19:47:00.379632: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory 2025-07-26 19:47:00.380958: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-26 19:47:00.381316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-26 19:47:00.381386: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory 2025-07-26 19:47:00.381440: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory 2025-07-26 19:47:00.381492: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory 2025-07-26 19:47:00.381501: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2025-07-26 19:47:00.381919: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2025-07-26 19:47:00.396214: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2000000000 Hz 2025-07-26 19:47:00.405365: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f45c0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-07-26 19:47:00.405415: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-07-26 19:47:00.409166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-07-26 19:47:00.409199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] WARNING:tensorflow:From /root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. loaded ./checkpoint/decom_net_train/model.ckpt loaded ./checkpoint/illumination_adjust_net_train/model.ckpt No restoration pre model! (480, 640, 3) (680, 720, 3) (415, 370, 3) Start evalating! 0 Traceback (most recent call last): File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call return fn(*args) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Restoration_net/de_conv6_1/biases [[{{node Restoration_net/de_conv6_1/biases/read}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "evaluate.py", line 92, in <module> restoration_r = sess.run(output_r, feed_dict={input_low_r: decom_r_low, input_low_i: decom_i_low}) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run run_metadata_ptr) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run feed_dict_tensor, options, run_metadata) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run run_metadata) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Restoration_net/de_conv6_1/biases [[node Restoration_net/de_conv6_1/biases/read (defined at /root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/variables.py:256) ]] Original stack trace for 'Restoration_net/de_conv6_1/biases/read': File "evaluate.py", line 28, in <module> output_r = Restoration_net(input_low_r, input_low_i) File "/root/Python/KinD-master/KinD-master/model.py", line 70, in Restoration_net conv6=slim.conv2d(up6, 256,[3,3], rate=1, activation_fn=lrelu,scope='de_conv6_1') File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, **current_args) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/layers/layers.py", line 1191, in convolution2d conv_dims=2) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, **current_args) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/layers/layers.py", line 1089, in convolution outputs = layer.apply(inputs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 1695, in apply return self.__call__(inputs, *args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 547, in __call__ outputs = super(Layer, self).__call__(inputs, *args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 758, in __call__ self._maybe_build(inputs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 2131, in _maybe_build self.build(input_shapes) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 172, in build dtype=self.dtype) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 460, in add_weight **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 447, in add_weight caching_device=caching_device) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 743, in _add_variable_with_custom_getter **kwargs_for_getter) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1573, in get_variable aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1316, in get_variable aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 551, in get_variable return custom_getter(**custom_getter_kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/layers/layers.py", line 1793, in layer_variable_getter return _model_variable_getter(getter, *args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/layers/layers.py", line 1784, in _model_variable_getter aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, **current_args) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/variables.py", line 328, in model_variable aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, **current_args) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tf_slim/ops/variables.py", line 256, in variable aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 520, in _true_getter aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 939, in _get_single_variable aggregation=aggregation) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 259, in __call__ return cls._variable_v1_call(*args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call shape=shape) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 198, in <lambda> previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2614, in default_variable_creator shape=shape) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1666, in __init__ shape=shape) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1854, in _init_from_args self._snapshot = array_ops.identity(self._variable, name="read") File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 282, in identity ret = gen_array_ops.identity(input, name=name) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3901, in identity "Identity", input=input, name=name) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal op_def=op_def) File "/root/Python/conda_lit/kind/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1791, in __init__ self._traceback = tf_stack.extract_stack()
07-27
Epoch 26: val_binary_accuracy improved from 0.88639 to 0.88671, saving model to ./modelsave/SUMIMO_simulation_CDLD_16QAM_MCS10_SNR5_LN_GN_group2_sparse_no_ITP WARNING:absl:Found untraced functions such as complex_conv2d_7_layer_call_fn, complex_conv2d_7_layer_call_and_return_conditional_losses, conv2d_layer_call_fn, conv2d_layer_call_and_return_conditional_losses, _jit_compiled_convolution_op while saving (showing 5 of 232). These functions will not be directly callable after loading. 3375/3375 - 254s - loss: 0.2509 - binary_accuracy: 0.8884 - val_loss: 0.2549 - val_binary_accuracy: 0.8867 - 254s/epoch - 75ms/step Epoch 27/100 Epoch 27: val_binary_accuracy improved from 0.88671 to 0.88770, saving model to ./modelsave/SUMIMO_simulation_CDLD_16QAM_MCS10_SNR5_LN_GN_group2_sparse_no_ITP WARNING:absl:Found untraced functions such as complex_conv2d_7_layer_call_fn, complex_conv2d_7_layer_call_and_return_conditional_losses, conv2d_layer_call_fn, conv2d_layer_call_and_return_conditional_losses, _jit_compiled_convolution_op while saving (showing 5 of 232). These functions will not be directly callable after loading. 2025-11-05 13:33:34.482240: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at save_restore_v2_ops.cc:282 : NOT_FOUND: ./modelsave/SUMIMO_simulation_CDLD_16QAM_MCS10_SNR5_LN_GN_group2_sparse_no_ITP/variables/variables_temp/part-00000-of-00001.index; No such file or directory Traceback (most recent call last): File "3-train_simulation_CDLD_16QAM_MCS10_SNR5p1_5p4_5p6_LN_GN_group2_sparse_5PRB_no_ITP_ear_stop_clip.py", line 145, in <module> callbacks=callback_list) # list,这个list中的回调函数将会在训练过程中的适当时机被调用,从而实现实时保存训练模型以及训练参数的目的 File "/home/ai/miniconda3/envs/venv/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/ai/miniconda3/envs/venv/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 55, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.NotFoundError: ./modelsave/SUMIMO_simulation_CDLD_16QAM_MCS10_SNR5_LN_GN_group2_sparse_no_ITP/variables/variables_temp/part-00000-of-00001.index; No such file or directory [Op:MergeV2Checkpoints]
最新发布
11-07
在使用nnunet训练时,出现如下报错jzuser@vpc87-3:~/Work_dir/Gn/pystudy/NnuNet$ nnUNetv2_train 3 2d 0 nnUNetv2_train 3 2d 1 nnUNetv2_train 3 2d 2 nnUNetv2_train 3 2d 3 nnUNetv2_train 3 2d 4 Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None 2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e727680>) 2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428fe062c0>) 2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428f79c100>) 2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428fe062c0>) 2025-08-27 09:22:59.147944: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428f79c100>) ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( 2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>) 2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>) 2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>) 2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>) 2025-08-27 09:23:04.645125: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c1613700>) This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} 2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>) 2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>) 2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>) 2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x70428e2d0ec0>) 2025-08-27 09:23:07.150209: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c34041c0>) These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>) 2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c2a95b80>) 2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>) 2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c2a95b80>) 2025-08-27 09:23:09.655113: failed to log: (<class 'OSError'>, OSError(28, 'No space left on device'), <traceback object at 0x7042c141d100>) 2025-08-27 09:23:09.655113: unpacking dataset... multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 581, in save format.write_array(fid, arr, allow_pickle=allow_pickle, File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 754, in write_array array.tofile(fp) OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 17, in _convert_to_npy np.save(npz_file[:-4] + "_seg.npy", a['seg']) File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 579, in save with file_ctx as fid: OSError: [Errno 28] No space left on device """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1203, in run_training self.on_train_start() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 788, in on_train_start unpack_dataset(self.preprocessed_dataset_folder, unpack_segmentation=True, overwrite_existing=False, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 33, in unpack_dataset p.starmap(_convert_to_npy, zip(npz_files, File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value OSError: [Errno 28] No space left on device Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_1' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_2' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_3' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_4' jzuser@vpc87-3:~/Work_dir/Gn/pystudy/NnuNet$ nnUNetv2_train 3 2d 0 nnUNetv2_train 3 2d 1 nnUNetv2_train 3 2d 2 nnUNetv2_train 3 2d 3 nnUNetv2_train 3 2d 4 Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 11:19:51.857624: unpacking dataset... 2025-08-27 11:20:40.435668: unpacking done... 2025-08-27 11:20:40.436964: do_dummy_2d_data_aug: False 2025-08-27 11:20:40.438331: Creating new 5-fold cross-validation split... 2025-08-27 11:20:40.441395: Desired fold for training: 0 2025-08-27 11:20:40.441591: This split has 104 training and 27 validation cases. 2025-08-27 11:20:40.498886: Unable to plot network architecture: 2025-08-27 11:20:40.500076: No module named 'hiddenlayer' 2025-08-27 11:20:40.552532: 2025-08-27 11:20:40.553642: Epoch 0 2025-08-27 11:20:40.554312: Current learning rate: 0.01 Exception in background worker 0: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 2: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 1: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 3: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 4: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file using pin_memory on device 0 Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__ item = self.__get_next_item() File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 11:20:48.364002: unpacking dataset... 2025-08-27 11:20:51.509973: unpacking done... 2025-08-27 11:20:51.510635: do_dummy_2d_data_aug: False 2025-08-27 11:20:51.511523: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json 2025-08-27 11:20:51.511727: The split file contains 5 splits. 2025-08-27 11:20:51.511775: Desired fold for training: 1 2025-08-27 11:20:51.511804: This split has 105 training and 26 validation cases. 2025-08-27 11:20:51.521027: Unable to plot network architecture: 2025-08-27 11:20:51.521092: No module named 'hiddenlayer' 2025-08-27 11:20:51.526808: 2025-08-27 11:20:51.526880: Epoch 0 2025-08-27 11:20:51.526956: Current learning rate: 0.01 Exception in background worker 1: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 2: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 0: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file using pin_memory on device 0 Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__ item = self.__get_next_item() File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 11:20:58.407072: unpacking dataset... 2025-08-27 11:21:01.513495: unpacking done... 2025-08-27 11:21:01.514885: do_dummy_2d_data_aug: False 2025-08-27 11:21:01.517129: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json 2025-08-27 11:21:01.517678: The split file contains 5 splits. 2025-08-27 11:21:01.517827: Desired fold for training: 2 2025-08-27 11:21:01.517945: This split has 105 training and 26 validation cases. 2025-08-27 11:21:01.529522: Unable to plot network architecture: 2025-08-27 11:21:01.529716: No module named 'hiddenlayer' 2025-08-27 11:21:01.540922: 2025-08-27 11:21:01.541166: Epoch 0 2025-08-27 11:21:01.541448: Current learning rate: 0.01 Exception in background worker 2: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 3: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file using pin_memory on device 0 Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__ item = self.__get_next_item() File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 11:21:08.460438: unpacking dataset... 2025-08-27 11:21:11.615700: unpacking done... 2025-08-27 11:21:11.616486: do_dummy_2d_data_aug: False 2025-08-27 11:21:11.618074: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json 2025-08-27 11:21:11.618454: The split file contains 5 splits. 2025-08-27 11:21:11.618557: Desired fold for training: 3 2025-08-27 11:21:11.618626: This split has 105 training and 26 validation cases. 2025-08-27 11:21:11.628197: Unable to plot network architecture: 2025-08-27 11:21:11.628319: No module named 'hiddenlayer' 2025-08-27 11:21:11.635873: 2025-08-27 11:21:11.636014: Epoch 0 2025-08-27 11:21:11.636152: Current learning rate: 0.01 Exception in background worker 1: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 3: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 2: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file using pin_memory on device 0 Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__ item = self.__get_next_item() File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-27 11:21:18.424697: unpacking dataset... 2025-08-27 11:21:21.510880: unpacking done... 2025-08-27 11:21:21.511596: do_dummy_2d_data_aug: False 2025-08-27 11:21:21.513083: Using splits from existing split file: /home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_preprocessed/Dataset003_Liver/splits_final.json 2025-08-27 11:21:21.513473: The split file contains 5 splits. 2025-08-27 11:21:21.513583: Desired fold for training: 4 2025-08-27 11:21:21.513662: This split has 105 training and 26 validation cases. 2025-08-27 11:21:21.521894: Unable to plot network architecture: 2025-08-27 11:21:21.522025: No module named 'hiddenlayer' 2025-08-27 11:21:21.530473: 2025-08-27 11:21:21.530618: Epoch 0 2025-08-27 11:21:21.530759: Current learning rate: 0.01 Exception in background worker 2: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 97, in load_case seg = np.load(entry['data_file'][:-4] + "_seg.npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 4: No data left in file Exception in background worker 1: mmap length is greater than file size Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 477, in load return format.open_memmap(file, mode=mmap_mode, File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 965, in open_memmap marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order, File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/_core/memmap.py", line 289, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start) ValueError: mmap length is greater than file size Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 3: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Exception in background worker 0: No data left in file Exception in background worker 5: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file Traceback (most recent call last): File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer item = next(data_loader) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in __next__ return self.generate_train_batch() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/data_loader_2d.py", line 18, in generate_train_batch data, seg, properties = self._data.load_case(current_key) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case data = np.load(entry['data_file'][:-4] + ".npy", 'r') File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 460, in load raise EOFError("No data left in file") EOFError: No data left in file using pin_memory on device 0 Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1211, in run_training train_outputs.append(self.train_step(next(self.dataloader_train))) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__ item = self.__get_next_item() File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message如何解决
08-28
jzuser@vpc87-3:~/Work_dir/Gn/pystudy/NnuNet$ nnUNetv2_train 3 2d 0 nnUNetv2_train 3 2d 1 nnUNetv2_train 3 2d 2 nnUNetv2_train 3 2d 3 nnUNetv2_train 3 2d 4 Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None ####################################################################### Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211. ####################################################################### /home/jzuser/.local/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:62: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn( This is the configuration used by this training: Configuration name: 2d {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 12, 'patch_size': [512, 512], 'median_image_size_in_voxels': [512.0, 512.0], 'spacing': [0.767578125, 0.767578125], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': [2, 2, 2, 2, 2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2, 2, 2, 2, 2], 'num_pool_per_axis': [7, 7], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} These are the global plan.json settings: {'dataset_name': 'Dataset003_Liver', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 0.767578125, 0.767578125], 'original_median_shape_after_transp': [432, 512, 512], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 5420.0, 'mean': 99.48007202148438, 'median': 101.0, 'min': -983.0, 'percentile_00_5': -15.0, 'percentile_99_5': 197.0, 'std': 37.13840103149414}}} 2025-08-22 00:17:29.967345: unpacking dataset... multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 15, in _convert_to_npy np.save(npz_file[:-3] + "npy", a['data']) File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 581, in save format.write_array(fid, arr, allow_pickle=allow_pickle, File "/home/jzuser/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 754, in write_array array.tofile(fp) OSError: Not enough free space to write 491212848 bytes """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 195, in run_training nnunet_trainer.run_training() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1203, in run_training self.on_train_start() File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 788, in on_train_start unpack_dataset(self.preprocessed_dataset_folder, unpack_segmentation=True, overwrite_existing=False, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 33, in unpack_dataset p.starmap(_convert_to_npy, zip(npz_files, File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value OSError: Not enough free space to write 491212848 bytes Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_1' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_2' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_3' Using device: cuda:0 /home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py:152: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. self.grad_scaler = GradScaler() if self.device.type == 'cuda' else None Traceback (most recent call last): File "/home/jzuser/.local/bin/nnUNetv2_train", line 8, in <module> sys.exit(run_training_entry()) File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 252, in run_training_entry run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 180, in run_training nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 65, in get_trainer_from_args nnunet_trainer = nnunet_trainer(plans=plans, configuration=configuration, fold=fold, File "/home/jzuser/.local/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 159, in __init__ maybe_mkdir_p(self.output_folder) File "/home/jzuser/.local/lib/python3.10/site-packages/batchgenerators/utilities/file_and_folder_operations.py", line 88, in maybe_mkdir_p os.makedirs(directory, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/home/jzuser/Work_dir/Gn/pystudy/NnuNet/nnUNet_results/Dataset003_Liver/nnUNetTrainer__nnUNetPlans__2d/fold_4'
08-23
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值