/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
The following values were not passed to `accelerate launch` and had defaults used instead:
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in `--num_processes=1`.
`--num_machines` was set to a value of `1`
`--mixed_precision` was set to a value of `'no'`
`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
/home/ustc/anaconda3/lib/python3.12/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
/home/ustc/anaconda3/lib/python3.12/site-packages/torch/nn/utils/weight_norm.py:134: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
WeightNorm.apply(module, name, dim)
/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
[rank0]: return _target_(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/modules/astral_quantization/default_model.py", line 22, in __init__
[rank0]: self.tokenizer = WhisperProcessor.from_pretrained(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/processing_utils.py", line 1079, in from_pretrained
[rank0]: args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/processing_utils.py", line 1143, in _get_arguments_from_pretrained
[rank0]: args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/feature_extraction_utils.py", line 384, in from_pretrained
[rank0]: feature_extractor_dict, kwargs = cls.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/feature_extraction_utils.py", line 510, in get_feature_extractor_dict
[rank0]: resolved_feature_extractor_file = cached_file(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/utils/hub.py", line 266, in cached_file
[rank0]: file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/transformers/utils/hub.py", line 381, in cached_files
[rank0]: raise OSError(
[rank0]: OSError: ./checkpoints/hf_cache does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/./checkpoints/hf_cache/tree/main' for available files.
[rank0]: The above exception was the direct cause of the following exception:
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/train_v2.py", line 345, in <module>
[rank0]: main(args)
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/train_v2.py", line 314, in main
[rank0]: trainer = Trainer(
[rank0]: ^^^^^^^^
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/train_v2.py", line 75, in __init__
[rank0]: self._init_models(train_cfm=train_cfm, train_ar=train_ar)
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/train_v2.py", line 106, in _init_models
[rank0]: self._init_main_model(train_cfm=train_cfm, train_ar=train_ar)
[rank0]: File "/home/ustc/桌面/seed-vc-main-v2/train_v2.py", line 116, in _init_main_model
[rank0]: self.model = hydra.utils.instantiate(cfg).to(self.device)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
[rank0]: return instantiate_node(
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 342, in instantiate_node
[rank0]: value = instantiate_node(
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
[rank0]: return _call_target(_target_, partial, args, kwargs, full_key)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ustc/anaconda3/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
[rank0]: raise InstantiationException(msg) from e
[rank0]: hydra.errors.InstantiationException: Error in call to target 'modules.astral_quantization.default_model.AstralQuantizer':
[rank0]: OSError("./checkpoints/hf_cache does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/./checkpoints/hf_cache/tree/main' for available files.")
[rank0]: full_key: content_extractor_narrow
[rank0]:[W514 21:22:23.005946238 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
W0514 21:22:24.285000 132883023812096 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 3125405 closing signal SIGTERM
W0514 21:22:24.285000 132883023812096 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 3125406 closing signal SIGTERM
W0514 21:22:24.285000 132883023812096 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 3125407 closing signal SIGTERM
E0514 21:22:24.500000 132883023812096 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 3125404) of binary: /home/ustc/anaconda3/bin/python
Traceback (most recent call last):
File "/home/ustc/anaconda3/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/ustc/anaconda3/lib/python3.12/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/home/ustc/anaconda3/lib/python3.12/site-packages/accelerate/commands/launch.py", line 1204, in launch_command
multi_gpu_launcher(args)
File "/home/ustc/anaconda3/lib/python3.12/site-packages/accelerate/commands/launch.py", line 825, in multi_gpu_launcher
distrib_run.run(args)
File "/home/ustc/anaconda3/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/home/ustc/anaconda3/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ustc/anaconda3/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train_v2.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-05-14_21:22:24
host : ustc-SYS-740GP-TNRT
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3125404)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
分析原因
最新发布