Sample by a Hash Function (Scala)

最新推荐文章于 2023-08-03 02:46:35 发布

原创最新推荐文章于 2023-08-03 02:46:35 发布 · 765 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Scala

本文介绍了一种使用MurmurHash3函数进行大数据随机采样的方法，通过哈希函数对数据的键进行处理，实现一致的数据文件采样，特别适用于信用卡数据等场景。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

It’s really common in Big Data ad hoc analysis we need to down sample the data. However for most of the cases, we need to down sample based on some hash function of a Key of the data. For example, to process credit card data, we want to perform the sampling consistently across all the data files which contend account ID as the key. The pseudo code for this is

if Hash(account_ID) % 100 < 5: Keep
else: Drop

In Spark, we can use the MurmurHash3 function in Scala to do the sampling:

import
scala.util.hashing.{MurmurHash3=>MH3}
.....
val
seed=12345
val
rate=0.05
val
sample=data.filter(line=>(MH3.stringHash(line.take(10),seed)&0x7FFFFFFF)<(rate*0x7FFFFFFF))

Here I take the first 10 characters as the key and applied the MH3 hash. Since MH3.stringHash returns an Int, I apply a binary “and” to drop the 1st bit and applied the sample rate comparison.