[debug] Expected to have finished reduction in the prior iteration before starting a new one.

Harry嗷

已于 2022-07-16 18:05:32 修改

阅读量1.2k

点赞数 2

分类专栏： BUG解决文章标签： pytorch 人工智能 python

于 2022-01-11 21:48:36 首次发布

本文链接：https://blog.youkuaiyun.com/qq_41683065/article/details/122442379

版权

BUG解决专栏收录该内容

27 篇文章

订阅专栏

本文解决了PyTorch训练过程中遇到的一个常见问题：在前向传播时未使用某些模块的所有输出，导致运行时错误。通过确保所有定义的模块输出被利用，或者移除未使用的模块，可以有效避免此错误。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

问题描述

训练网络时出现错误：

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.

原因分析

参考该issue

I met the same issue.
But i solved it.
The reason is that in my model class, I define a fpn module with 5 level output feature maps in the init function,
but in forward function I only use 4 of them.
When I use all of them, the problem was solved.
This is my supposed conclusion: you should use all output of each module in forward function.

简单来说，就是在网络中定义了module，但没有使用
当然，这个module可以使最简单的网络层，也可以是更复杂的结构

解决方案

找到定义但没有使用的module，要么删掉，要么使用

update 20220716:

If finding the unused params becomes hard, there is a way to find all the pramas unused by check their gradient:

        for name, param in runner.model.named_parameters():
            if param.grad is None:
                print(name)

This code should be added between loss.backward() and optimizer.step()