tensorflow.python.framework.errors_impl.DataLossError: truncated record at 40109222 [Op:IteratorGetN

PinkAir

于 2020-10-25 08:43:23 发布

阅读量1.6k

点赞数

CC 4.0 BY-SA版权

分类专栏： tf/keras debug

本文链接：https://blog.youkuaiyun.com/PinkAir/article/details/109269620

在使用TensorFlow时遇到DataLossError：truncated record，可能由不完整数据、多次读取同一记录或磁盘问题引起。训练过程中罕见的文件损坏可以通过转换操作来解决。相关问题在TensorFlow的GitHub issue中有所讨论。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Error:tensorflow.python.framework.errors_impl.DataLossError: truncated record at 40109222 [Op:IteratorGetNextSync]

There are three aspects that may raise this error:

1.It is main due to the incomplete data.

2.A result of several reading of the same record, and its also dependent on the disk

3.The tf.data.experimental.ignore_errors transformation could be a solution for the rare corruption that happens during the training.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.apply(tf.data.experimental.ignore_errors())

iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
    while True:
        print(sess.run(next_element))

Reference:

https://github.com/tensorflow/tensorflow/issues/13463

https://github.com/tensorflow/tensorflow/issues/25700#issuecomment-462959084