关于在循环中backward()的注意事项

许多人会在训练时遇到这样一个报错：

trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)

一般原因是在每个epoch中调用loss.backward()时 触发这个错误。此时一般不要按照报错代码中官方给出的推荐一样去做retain_graph=True，因为这样每次循环都会增大memory,导致out of menmory. 而是应该去寻找自己代码的问题，做detach()。参考python - RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed) - Stack Overflow

我的问题和他不太一样，更为简单易懂，但是原理差不多。触发这个的场景是在执行cw attack的时候。 报错代码大致如下

target_feature = model(target_image)

def forward(self,images,labels):
    images = images.clone().detach().to(self