java查重工具类_Java各种高性能近似成员过滤器(布隆过滤器等查重过滤)

最新推荐文章于 2025-01-19 19:58:16 发布

月姨的设计笔记

最新推荐文章于 2025-01-19 19:58:16 发布

阅读量303

点赞数

CC 4.0 BY-SA版权

文章标签： java查重工具类

本文链接：https://blog.youkuaiyun.com/weixin_33134401/article/details/114860124

本文介绍了一种在Java中实现的快速近似成员过滤器，包括Bloom过滤器的各种变体以及Cuckoo过滤器等。这些过滤器用于高效地判断元素是否可能存在于集合中，特别适用于内存有限的场景。文中还提供了一个使用示例，演示了如何从大量已知密码创建过滤器，并通过该过滤器查询密码。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Fast Approximate Membership Filters in Java

68747470733a2f2f7472617669732d63692e636f6d2f4661737446696c7465722f6661737466696c7465725f6a6176612e7376673f6272616e63683d6d6173746572

The following filter types are currently implemented:

Bloom filter: the 'standard' algorithm

Blocked Bloom filter: faster than regular Bloom filters, but need a bit more space

Counting Bloom filter: allow removing entries, but need 4 times more space

Succinct counting Bloom filter: about half the space of regular counting Bloom filters; faster lookup but slower add / remove

Succinct counting blocked Bloom filter: same lookup speed as blocked Bloom filter

Cuckoo filter: 8 and 16 bit variants; uses cuckoo hashing to store fingerprints

Cuckoo+ filter: 8 and 16 bit variants, need a bit less space than regular cuckoo filters

Golomb Compressed Set (GCS): needs less space than cuckoo filters, but lookup is slow

Minimal Perfect Hash filter: needs less space than cuckoo filters, but lookup is slow

Xor filter: 8 and 16 bit variants; needs less space than cuckoo filters, with faster lookup

Xor+ filter: 8 and 16 bit variants; compressed xor filter

Reference: Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters, Journal of Experimental Algorithmics (to appear).

Other Xor Filter Implementations

Password Lookup Tool

Included is a tool to build a filter from a list of known password (hashes), and a tool to do lookups. That way, the password list can be queried locally, without requiring a large file. The filter is only 650 MB, instead of the original file which is 11 GB. At the cost of some false positives (unknown passwords reported as known, with about 1% probability).

Generate the Password Filter File

Download the latest SHA-1 password file that is ordered by hash, for example the file pwned-passwords-sha1-ordered-by-hash-v4.7z (~10 GB) from https://haveibeenpwned.com/passwords with about 550 million passwords.

If you have enough disk space, you can extract the hash file (~25 GB), and convert it as follows:

mvn clean install

cat hash.txt | java -cp target/fastfilter*.jar org.fastfilter.tools.BuildFilterFile filter.bin

Converting takes about 2-3 minutes (depending on hardware). To save disk space, you can extract the file on the fly (Mac OS X using Keka):

/Applications/Keka.app/Contents/Resources/keka7z e -so

pass.7z | java -cp target/fastfilter*.jar org.fastfilter.tools.BuildFilterFile filter.bin

Both will generate a file named filter.bin (~630 MB).

Check Passwords

java -cp target/fastfilter*.jar org.fastfilter.tools.PasswordLookup filter.bin

Enter a password to see if it's in the list. If yes, it will (for sure) either show "Found", or "Found; common", which means it was seen 10 times or more often. Passwords not in the list will show "Not found" with more than 99% probability, and with less than 1% probability "Found" or "Found; common".

Internally, the tool uses a xor+ filter (see above) with 8 bits per fingerprint. Actually, 1024 smaller filters (segments) are made, the segment id being the highest 10 bits of the key. The lowest bit of the key is set to either 0 (regular) or 1 (common), and so two lookups are made per password. Because of that, the false positive rate is twice of what it would be with just one lookup (0.0078 instead of 0.0039). A regular Bloom filter with the same guarantees would be ~760 MB. For each lookup, one filter segment (so, less than 1 MB) are read from the file.