pymongo 批量插入数据

最新推荐文章于 2025-09-13 08:20:44 发布

原创最新推荐文章于 2025-09-13 08:20:44 发布 · 4.1k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#python #mongo

python 专栏收录该内容

3 篇文章

订阅专栏

在使用pymongo的insert_many批量插入数据时遇到错误，原因是所有插入值相同。问题出在创建字典列表时，由于Python的引用特性，所有字典引用了同一内存地址。通过使用`copy()`方法创建字典副本，避免了值的同步，成功实现批量插入。

pymongo 批量插入数据

使用python+mongodb开发一个简单的计算机运行情况监控程序，为了方便后期测试，打算模拟批量往mongodb中插入测试数据，在使用insert_many时出现，如下错误提示：

pymongo.errors.BulkWriteError: batch op errors occurred：

经过查资料发现是pymongo在处理批量插入时会检测值是否一致，如果所有的值都一样，就会报错，为此重新看代码，在代码中加入print函数，发现使用字典生成列表时，出现列表内的字典内容全部一样，代码如下：

def funInsert():
    DictMonitorList = []
        i = 0
        while i < 10:
            DictMonitor['host'] = '192.168.1.' + str(randint(1, 10)) 
            DictMonitor['boot_start'] = randint(1000,20000)
            DictMonitor['cpu_usage'] = randint(1,100)
            DictMonitor['ram'] = 80123
            DictMonitorList.append(DictMonitor)
            print(DictMonitorList)
    result = tblmonitor.insert_many(DictMonitorList)
    print(result)

输出结果如下：

[{'host': '192.168.1.7', 'boot_start': 14773, 'cpu_usage': 68, 'ram': 80123}]
插入成功0条
[{'host': '192.168.1.6', 'boot_start': 2069, 'cpu_usage': 96, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 2069, 'cpu_usage': 96, 'ram': 80123}]
插入成功1条
[{'host': '192.168.1.5', 'boot_start': 18948, 'cpu_usage': 52, 'ram': 80123}, {'host': '192.168.1.5', 'boot_start': 18948, 'cpu_usage': 52, 'ram': 80123}, {'host': '192.168.1.5', 'boot_start': 18948, 'cpu_usage': 52, 'ram': 80123}]
插入成功2条
[{'host': '192.168.1.9', 'boot_start': 10814, 'cpu_usage': 56, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 10814, 'cpu_usage': 56, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 10814, 'cpu_usage': 56, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 10814, 'cpu_usage': 56, 'ram': 80123}]
插入成功3条
[{'host': '192.168.1.7', 'boot_start': 13935, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.7', 'boot_start': 13935, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.7', 'boot_start': 13935, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.7', 'boot_start': 13935, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.7', 'boot_start': 13935, 'cpu_usage': 35, 'ram': 80123}]
插入成功4条
[{'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 4494, 'cpu_usage': 46, 'ram': 80123}]
插入成功5条
[{'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 12683, 'cpu_usage': 3, 'ram': 80123}]
插入成功6条
[{'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.8', 'boot_start': 16117, 'cpu_usage': 46, 'ram': 80123}]
插入成功7条
[{'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}, {'host': '192.168.1.10', 'boot_start': 10338, 'cpu_usage': 95, 'ram': 80123}]
插入成功8条
[{'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}, {'host': '192.168.1.1', 'boot_start': 14840, 'cpu_usage': 38, 'ram': 80123}]
插入成功9条

根据以上结果，发现随机函数起作用了，但最终DictMonitorList[]中的值都是最后一次随机生成的DictMonitor{}的值。
原来python在处理字典、列表、变量的赋值时，一般对相同的变量名，不是真正的赋值，而是把变量的地址记录下来。这样在就出现了每个给DictMonitorList[]列表中加入DictMonitor{}时记录的都是DictMonitor{}的内存地址，每个更新DictMonitor{}时列表调取DictMonitor{}都是最新的值。所以造成了以上的结果。
将列表添加代码改成如下：

DictMonitorList.append(DictMonitor.copy())

copy（）方法是在内存中开辟一个新空间，这样值之间就不会出现同步的情况。执行结果如下：

[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}]
插入成功0条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}]
插入成功1条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}]
插入成功2条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}]
插入成功3条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}]
插入成功4条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 14434, 'cpu_usage': 35, 'ram': 80123}]
插入成功5条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 14434, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 6872, 'cpu_usage': 9, 'ram': 80123}]
插入成功6条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 14434, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 6872, 'cpu_usage': 9, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 17537, 'cpu_usage': 21, 'ram': 80123}]
插入成功7条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 14434, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 6872, 'cpu_usage': 9, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 17537, 'cpu_usage': 21, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 10711, 'cpu_usage': 31, 'ram': 80123}]
插入成功8条
[{'host': '192.168.1.8', 'boot_start': 6558, 'cpu_usage': 48, 'ram': 80123}, {'host': '192.168.1.9', 'boot_start': 14226, 'cpu_usage': 54, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 12776, 'cpu_usage': 46, 'ram': 80123}, {'host': '192.168.1.3', 'boot_start': 18060, 'cpu_usage': 34, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 16387, 'cpu_usage': 10, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 14434, 'cpu_usage': 35, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 6872, 'cpu_usage': 9, 'ram': 80123}, {'host': '192.168.1.2', 'boot_start': 17537, 'cpu_usage': 21, 'ram': 80123}, {'host': '192.168.1.6', 'boot_start': 10711, 'cpu_usage': 31, 'ram': 80123}, {'host': '192.168.1.4', 'boot_start': 15664, 'cpu_usage': 83, 'ram': 80123}]
插入成功9条
<pymongo.results.InsertManyResult object at 0x000001E521B22D48>

插入成功OK

参考：1、https://blog.youkuaiyun.com/Sheldomcooper/article/details/82258006