Bloom Filter Brief Intro

Jason Davies

Bloom Filters

Everyone is always raving about bloom filters. But what exactly are they, and what are they useful for?

Operations

The basic bloom filter supports two operations: test and add.

Test is used to check whether a given element is in the set or not. If it returns:

• false then the element is definitely not in the set.
• true then the element is probably in the set. The false positive rate is a function of the bloom filter's size and the number and independence of the hash functions used.

Add simply adds an element to the set. Removal is impossible without introducing false negatives, but extensions to the bloom filter are possible that allow removal e.g. counting filters.

Applications

The classic example is using bloom filters to reduce expensive disk (or network) lookups for non-existent keys.

If the element is not in the bloom filter, then we know for sure we don't need to perform the expensive lookup. On the other hand, if itis in the bloom filter, we perform the lookup, and we can expect it to fail some proportion of the time (the false positive rate).

Bloomfilter.js

I wrote a very fast bloom filter implementation in JavaScript called bloomfilter.js. It uses the non-cryptographic Fowler–Noll–Vo hash function for speed. We can get away with using a non-cryptographic hash function as we only care about having a uniform distribution of hashes.

The implementation also uses JavaScript typed arrays if possible, as these are faster when performing low-level bitwise operations.

Interactive Demonstration

Below you should see an interactive visualisation of a bloom filter, powered by bloomfilter.js.

You can add any number of elements (keys) to the filter by typing in the textbox and clicking "Add". Then use the second textbox to see if an element is probably there or definitely not!

Key:

Definitely not there.

Explanation

The bloom filter essentially consists of a bit vector of length m, represented by the central column.

To add an item to the bloom filter, we feed it to k different hash functions and set the bits at the resulting positions. In this example, I've setm to50 andk to 3. Note that sometimes the hash functions produce overlapping positions, so less thank positions may be set.

To test if an item is in the filter, again we feed it to the k hash functions. This time, we check to see if any of the bits at these positions are not set. If any are not set, it means the item is definitely not in the set. Otherwise, it is probably in the set.

Implementation Notes

Astute readers will note that I only mentioned one hash function, Fowler–Noll–Vo, but bloom filters require several. It turns out thatyou can produce k hash functions using only two to start off with. In fact, I just perform an additional round of FNV as the second hash function, and this works great. Unfortunately I can't use the 64-bit trick in the linked post as JavaScript only supports bitwise operations on 32 bits.

• Follow (22)
• Subscribe

A Bloom filter is a data structure optimized for fast, space-efficient set membership tests. Bloom filters have the unusual property of requiring constant time to add an element to the set or test for membership, regardless of the size of the elements or the number of elements already in the set. No other constant-space set data structure has this property.

It works by storing a bit vector representing the set S' = {h[i](x) | x in S, i = 1, …, k}, where h[1], …, h[k] := {0, 1} -> [n lg(1/ε) lg e] are hash functions. Additions are simply setting k bits to 1, specifically those at h[1](x), …, h[k](x). Checks are implemented by performing those same hash functions and returning true if all of the resulting positions are 1.

Because the set stored is a proper superset of the set of items added, false positives may occur, though false negatives cannot. The false positive rate can be specified.

Bloom filters offer the following advantages:

• Space: Approximately n * lg(1/ε), where ε is the false positive rate and n is the number of elements in the set.
  • Example: There are approximately 170k words in the English language. If we consider that to be our set (therefore n = 1.7E5), and we wish to search a corpus for them with a 1% false positive rate, the filter would require about (1.7E5 * lg(1 / 0.01)) ≈ 162 KB. Contrast this with a hashtable, which would require (1.7E5 elements * 32 bits per element) ≈ 664 KB. Obviously explicit string storage would be significantly more.
• Precision: Arbitrary precision, where increasing precision requires more space (following the above size equation) but not more time.
  • Example: If we wanted to reduce our false positive rate in the above example from one percent to one permille the space requirement would go from about 162 KB to about 207 KB.
• Time: O(k) where k is the number of hash functions. The optimal number of hash functions (though a different number can be supplied by the user if desired) is ceiling(lg(1/ε))
  • Example: In keeping with our above example, if the accepted false positive rate is 0.001, k = 10.

This implementation uses Dillinger & Manolios double hashing to provide all but the first two hash functions. By default the first hash function is the type's GetHashCode() method. This implementation also includes default secondary hash functions for strings ( Jenkin's "One at a time" method) and integers ( Wang's method).

Bloom filters are due to Burton H. Bloom, as described in the Communications of the ACM in July 1970. The full paper is available here.

Last edited Nov 11, 2014 at 9:38 PM by JustinRussell, version 6

See Also

• Bloom filter on Wikipedia.
• László Kozma’s cuckoo hashing visualisation.