Traceback (most recent call last):
[rank0]: File "/data3/workspace/chenzh/Face-SVD/train.py", line 1668, in <module>
[rank0]: main(**OmegaConf.load(args.config))
[rank0]: File "/data3/workspace/chenzh/Face-SVD/train.py", line 1409, in main
[rank0]: discr_fake_pred = attr_discriminator(vt_hat_latents)
[rank0]: File "/data1/miniconda3/envs/face_svd/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/data1/miniconda3/envs/face_svd/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/data1/miniconda3/envs/face_svd/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1632, in forward
[rank0]: inputs, kwargs = self._pre_forward(*inputs, **kwargs)
[rank0]: File "/data1/miniconda3/envs/face_svd/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1523, in _pre_forward
[rank0]: if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
[rank0]: RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
[rank0]: making sure all `forward` function outputs participate in calculating loss.
[rank0]: If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
[rank0]: Parameters which did not receive grad for rank 0: backbone.encoder_block3.0.norm.bias, backbone.encoder_block3.0.norm.weight, backbone.encoder_block3.0.conv.weight, backbone.encoder_block2.2.attn.to_out.0.bias, backbone.encoder_block2.2.attn.to_out.0.weight, backbone.encoder_block2.2.attn.to_v.weight, backbone.encoder_block2.2.attn.to_k.weight, backbone.encoder_block2.2.attn.to_q.weight, backbone.encoder_block2.2.norm.bias, backbone.encoder_block2.2.norm.weight, backbone.encoder_block2.2.resnet.conv2.bias, backbone.encoder_block2.2.resnet.conv2.weight, backbone.encoder_block2.2.resnet.norm2.bias, backbone.encoder_block2.2.resnet.norm2.weight, backbone.encoder_block2.2.resnet.conv1.bias, backbone.encoder_block2.2.resnet.conv1.weight, backbone.encoder_block2.2.resnet.norm1.bias, backbone.encoder_block2.2.resnet.norm1.weight, backbone.encoder_block2.1.attn.to_out.0.bias, backbone.encoder_block2.1.attn.to_out.0.weight, backbone.encoder_block2.1.attn.to_v.weight, backbone.encoder_block2.1.attn.to_k.weight, backbone.encoder_block2.1.attn.to_q.weight, backbone.encoder_block2.1.norm.bias, backbone.encoder_block2.1.norm.weight, backbone.encoder_block2.1.resnet.conv2.bias, backbone.encoder_block2.1.resnet.conv2.weight, backbone.encoder_block2.1.resnet.norm2.bias, backbone.encoder_block2.1.resnet.norm2.weight, backbone.encoder_block2.1.resnet.conv1.bias, backbone.encoder_block2.1.resnet.conv1.weight, backbone.encoder_block2.1.resnet.norm1.bias, backbone.encoder_block2.1.resnet.norm1.weight, backbone.encoder_block2.0.norm.bias, backbone.encoder_block2.0.norm.weight, backbone.encoder_block2.0.conv.weight, backbone.encoder_block1.1.attn.to_out.0.bias, backbone.encoder_block1.1.attn.to_out.0.weight, backbone.encoder_block1.1.attn.to_v.weight, backbone.encoder_block1.1.attn.to_k.weight, backbone.encoder_block1.1.attn.to_q.weight, backbone.encoder_block1.1.norm.bias, backbone.encoder_block1.1.norm.weight, backbone.encoder_block1.1.resnet.conv2.bias, backbone.encoder_block1.1.resnet.conv2.weight, backbone.encoder_block1.1.resnet.norm2.bias, backbone.encoder_block1.1.resnet.norm2.weight, backbone.encoder_block1.1.resnet.conv1.bias, backbone.encoder_block1.1.resnet.conv1.weight, backbone.encoder_block1.1.resnet.norm1.bias, backbone.encoder_block1.1.resnet.norm1.weight, backbone.encoder_block1.0.attn.to_out.0.bias, backbone.encoder_block1.0.attn.to_out.0.weight, backbone.encoder_block1.0.attn.to_v.weight, backbone.encoder_block1.0.attn.to_k.weight, backbone.encoder_block1.0.attn.to_q.weight, backbone.encoder_block1.0.norm.bias, backbone.encoder_block1.0.norm.weight, backbone.encoder_block1.0.resnet.conv2.bias, backbone.encoder_block1.0.resnet.conv2.weight, backbone.encoder_block1.0.resnet.norm2.bias, backbone.encoder_block1.0.resnet.norm2.weight, backbone.encoder_block1.0.resnet.conv1.bias, backbone.encoder_block1.0.resnet.conv1.weight, backbone.encoder_block1.0.resnet.norm1.bias, backbone.encoder_block1.0.resnet.norm1.weight, backbone.conv1x1.bias, backbone.conv1x1.weight
[rank0]: Parameter indices which did not receive grad for rank 0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67