【已解决】Python Pickle报错:OSError: [Errno 22] Invalid argument

在macOS 10.13.4上使用Python 3.6.3的Pickle模块时,遇到OSError: [Errno 22] Invalid argument错误。原因是尝试dump的对象超过2GB。解决方法是通过bytearray来写入大对象。参考了StackOverflow等资源。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

问题描述

在Python代码中使用Pickle模块dump一个对象,报错:

>>> with open(path, 'wb') as f:
>>>    pickle.dump(self, f)

Traceback (most recent call last):
  File "<stdin>"
C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\python.exe C:\Users\wsco\Desktop\HREM-main\train.py Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias'] - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). GraphLoss( (base_loss): TripletLoss() (gnn_loss): TripletLoss() (gnn): TransformerEncoder( (layers): ModuleList( (0): TransformerEncoderLayer( (self_attn): MultiheadAttention( (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True) ) (linear1): Linear(in_features=1024, out_features=1024, bias=True) (dropout): Dropout(p=0.1, inplace=False) (linear2): Linear(in_features=1024, out_features=1024, bias=True) (norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (dropout1): Dropout(p=0.1, inplace=False) (dropout2): Dropout(p=0.1, inplace=False) ) ) ) (adj_model): AdjacencyModel( (adj_learning): AdjacencyLearning( (mlp_t2i): Sequential( (0): Linear(in_features=10, out_features=10, bias=True) (1): ReLU(inplace=True) (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=10, out_features=1, bias=True) ) (mlp_i2t): Sequential( (0): Linear(in_features=10, out_features=10, bias=True) (1): ReLU(inplace=True) (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=10, out_features=1, bias=True) ) ) ) ) Traceback (most recent call last): File "C:\Users\wsco\Desktop\HREM-main\train.py", line 271, in <module> main() File "C:\Users\wsco\Desktop\HREM-main\train.py", line 83, in main train(opt, train_loader, model, epoch) File "C:\Users\wsco\Desktop\HREM-main\train.py", line 143, in train for i, train_data in enumerate(train_loader): File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\site-packages\torch\utils\data\dataloader.py", line 368, in __iter__ return self._get_iterator() File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\site-packages\torch\utils\data\dataloader.py", line 314, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\site-packages\torch\utils\data\dataloader.py", line 927, in __init__ w.start() File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__ reduction.dump(process_obj, to_child) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) OSError: [Errno 22] Invalid argument Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\wsco\anaconda3\envs\py3.9_torch11.3.1_cuda11.3\lib\multiprocessing\spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) _pickle.UnpicklingError: pickle data was truncated 进程已结束,退出代码1 如何解决这个报错
最新发布
08-07
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值