解决一下(nnUNet) root@autodl-container-54694c8faa-39a99183:/autodl-fs/data/nnU-Net/nnUNet#
nnUNetv2_train 4 2d 0
############################
INFO: You are using the old nnU-Net default plans. We have updated our recommendations. Please consider using those instead! Read more here: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/resenc_presets.md
############################
Using device: cuda:0
#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
2025-05-12 19:33:23.235735: Using torch.compile...
2025-05-12 19:33:23.251931: do_dummy_2d_data_aug: False
2025-05-12 19:33:23.261743: Using splits from existing split file: /root/autodl-fs/nnU-Net/nnUNet/nnUNet_preprocessed/Dataset004_Hippocampus/splits_final.json
2025-05-12 19:33:23.262319: The split file contains 5 splits.
2025-05-12 19:33:23.262368: Desired fold for training: 0
2025-05-12 19:33:23.262403: This split has 208 training and 52 validation cases.
using pin_memory on device 0
using pin_memory on device 0
This is the configuration used by this training:
Configuration name: 2d
{'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 366, 'patch_size': [56, 40], 'median_image_size_in_voxels': [50.0, 35.0], 'spacing': [1.0, 1.0], 'normalization_schemes': ['ZScoreNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.PlainConvUNet', 'arch_kwargs': {'n_stages': 4, 'features_per_stage': [32, 64, 128, 256], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2]], 'n_conv_per_stage': [2, 2, 2, 2], 'n_conv_per_stage_decoder': [2, 2, 2], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}
These are the global plan.json settings:
{'dataset_name': 'Dataset004_Hippocampus', 'plans_name': 'nnUNetPlans', 'original_median_spacing_after_transp': [1.0, 1.0, 1.0], 'original_median_shape_after_transp': [36, 50, 35], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'ExperimentPlanner', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 486420.21875, 'mean': 22360.326171875, 'median': 362.88250732421875, 'min': 0.0, 'percentile_00_5': 28.0, 'percentile_99_5': 277682.03125, 'std': 60656.1328125}}}
2025-05-12 19:33:25.892359: Unable to plot network architecture: nnUNet_compile is enabled!
2025-05-12 19:33:25.901765:
2025-05-12 19:33:25.902487: Epoch 0
2025-05-12 19:33:25.902790: Current learning rate: 0.01
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler
compiled_fn = compiler_fn(gm, self.fake_example_inputs())
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/__init__.py", line 1388, in __call__
from torch._inductor.compile_fx import compile_fx
File "/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 21, in <module>
from . import config, metrics, overrides, pattern_matcher
File "/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/pattern_matcher.py", line 19, in <module>
from .lowering import lowerings as L
File "/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 3868, in <module>
import_submodule(kernel)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1304, in import_submodule
importlib.import_module(f"{mod.__name__}.{filename[:-3]}")
File "/root/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/kernel/conv.py", line 22, in <module>
from ..utils import (
ImportError: cannot import name 'is_ones' from 'torch._inductor.utils' (/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/utils.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/miniconda3/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
File "/autodl-fs/data/nnU-Net/nnUNet/nnunetv2/run/run_training.py", line 267, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/autodl-fs/data/nnU-Net/nnUNet/nnunetv2/run/run_training.py", line 207, in run_training
nnunet_trainer.run_training()
File "/autodl-fs/data/nnU-Net/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1371, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
File "/autodl-fs/data/nnU-Net/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 989, in train_step
output = self.network(data)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
result = inner_convert(frame, cache_size, hooks)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1792, in RETURN_VALUE
self.output.compile_subgraph(
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 541, in compile_subgraph
self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised ImportError: cannot import name 'is_ones' from 'torch._inductor.utils' (/root/miniconda3/lib/python3.10/site-packages/torch/_inductor/utils.py)
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
Exception in thread Thread-2 (results_loop):
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Exception in thread Thread-1 (results_loop):
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/lib/python3.10/threading.py", line 953, in run
self.run()
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/lib/python3.10/threading.py", line 953, in run
File "/root/miniconda3/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/root/miniconda3/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise e
File "/root/miniconda3/lib/python3.10/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message