how can I ues Dataset to shuffle a large whole dataset?

weixin_30724853

于 2018-06-26 09:32:00 发布

阅读量70

点赞数

CC 4.0 BY-SA版权

文章标签：大数据

原文链接：http://www.cnblogs.com/crackpotisback/p/9227523.html

本文介绍了一种适用于大规模数据集的外部内存洗牌方法。该方法首先使用MapReduce/Spark/Beam等工具对整个数据集进行一次随机洗牌，并将其拆分成大致相等大小的文件（碎片）。随后，在每个训练周期内，通过随机洗牌文件名列表并使用tf.data API将不同碎片中的记录混合，最后再对组合后的数据集进行洗牌。

The Dataset.shuffle() implementation is designed for data that could be shuffled in memory; we're considering whether to add support for external-memory shuffles, but this is in the early stages. In case it works for you, here's the usual approach we use when the data are too large to fit in memory:

Randomly shuffle the entire data once using a MapReduce/Spark/Beam/etc. job to create a set of roughly equal-sized files ("shards").
In each epoch:

Randomly shuffle the list of shard filenames, using Dataset.list_files(...).shuffle(num_shards).
Use dataset.interleave(lambda filename: tf.data.TextLineDataset(filename), cycle_length=N) to mix together records from N different shards.
Use dataset.shuffle(B) to shuffle the resulting dataset. Setting B might require some experimentation, but you will probably want to set it to some value larger than the number of records in a single shard.

转载于:https://www.cnblogs.com/crackpotisback/p/9227523.html