Fast Thread-Safe Map Cache

Recently I came across an interesting programming problem I want to share with you. In short this post discusses thread-safe and fast java implementation of a cache for some special case.

To give you some background, this problem arose in scope of my pet project PUNKSearch (local network indexer and searcher based on Lucene ). PUNKSearch shows online/offline status of the hosts displayed in search results. These statuses are cached for several minutes to avoid “dogpile effect ” on remote hosts and increase overall responsiveness of the application. Cache is organized as HashMap for which keys are IPs and values are pairs of Date and Boolean wrapped in some extra object named HostStatus. The Date is used to determine if we need to refresh the status for the host and Boolean defines whatever host is online. Obtaining status is a Slow Process (timeout usually is 5 seconds).

Since PUNKSearch is a web application this cache must be thread-safe. Another property we want it to have is speed. Locking the whole cache for the period some thread tries to determine if a host is online is no way.

Lets try to construct such a cache.
Try 1: The simple not thread-safe implementation (cache renewal is dropped for the sake of clearness )

01	class MapCacheSimpleFastBroken {

02

03	private Map<String, Object> cache       = new HashMap<String, Object>();

04	private SlowProcess         slowProcess = new SlowProcess();

05

06	public Object get(String key) {

07	Object value = cache.get(key);

08	if (value == null ) {

09	value = slowProcess.get(key);

10	cache.put(key, value);

11

}

12	return value;

13

}

14

}

This class obviously not thread-safe. What can we do to make it thread-safe? Of course, make the method “synchronized”!

Try 2: The simple thread-safe implementation (cache renewal is dropped for the sake of clearness )

01	class MapCacheSimpleSlowSafe {

02

03	private Map<String, Object> cache       = new HashMap<String, Object>();

04	private SlowProcess         slowProcess = new SlowProcess();

05

06	public synchronized Object get(String key) {

07	Object value = cache.get(key);

08	if (value == null ) {

09	value = slowProcess.get(key);

10	cache.put(key, value);

11

}

12	return value;

13

}

14

}

This implementation is thread-safe , good. But it is extremely slow ! How can we avoid locking the whole cache object? What if we lock only required portion of the map? The map key in particular. Lets try.

Try 3: The fine-grained want-to-be thread-safe implementation (cache renewal is dropped for the sake of clearness )

01	class MapCacheTrickyFastBroken {

02

03	private Map<String, Object> cache       = new HashMap<String, Object>();

04	private SlowProcess         slowProcess = new SlowProcess();

05	private static final Object STUB        = new Object();

06

07	public Object get(String key) throws InterruptedException {

08	String lock = "" ;

09	// sync on the whole cache to: 1) get cached data, 2) insert stub if necessary

10	synchronized (cache) {

11	Object value = cache.get(key);

12	if (value != null && value != STUB) { // "null" only if this is the first time we see this key

13	return value;

14	} else if (value == null ) { // we see the key for the first time, add the stub to create a key in map keySet

15	cache.put(key, STUB);

16

}

17	// init lock with the key for the required object

18	for (String cacheKey : cache.keySet()) {

19	if (cacheKey.equals(key)) {

20	lock = cacheKey;

21

break ;

22

}

23

}

24

}

25	// grab the lock only on the part of the cache map, so online checks for different hosts can go simultaneously

26	// can't use "key" argument as lock here, since we want to sync on cache's key object

27	synchronized (lock) {

28	Object value = cache.get(key);

29	// maybe object was already updated by other thread while we were waiting for lock?

30	// call the slow process only if that was not happened

31	if (value == STUB) {

32	value = slowProcess.get(key);

33	cache.put(key, value);

34

}

35	return value;

36

}

37

}

38

}

This implementation seems to be ok. We lock the whole map for a short period of time to verify if we have required object in the cache and possibly modify the map. If the object was not found we put a stub into the map to sync against its key later.
Bad news. This implementation has a serious flaw. Can you find it? Look under the cut for the explanation and the correct implementation.

Indeed, the Try 3 is broken. Assume two threads want to obtain an object for the same key “foo”.
Now imagine first thread has left the 33 line and the second thread just fall into “else” branch at the line 14 (i.e. it is at the line 15 now). Obviously the second thread drops all that hard work done by the first thread and have to call the slow process again. Not good. Additionally notice the crap we have at lines 18-23 to obtain the key object. What we can do to avoid these 2 issues? Lets try.

Last Try: The fine-grained thread-safe implementation (cache renewal is dropped for the sake of clearness )

01	class MapCacheTrickyFastSafe {

02

03	private ConcurrentHashMap<String, Object> cache       = new ConcurrentHashMap<String, Object>();

04	private Map<String, String>               keys        = new HashMap<String, String>();

05	private SlowProcess                       slowProcess = new SlowProcess();

06	private static final Object               STUB        = new Object();

07

08	public Object get(String key) throws InterruptedException {

09	String lock = "" ;

10	// sync on the whole cache to: 1) get cached data, 2) insert stub if necessary

11	synchronized (cache) {

12	Object value = cache.putIfAbsent(key, STUB);

13	if (value != null && value != STUB) { // "null" only if this is the first time we see this key

14	return value;

15	} else if (value == null ) { // we see the key for the first time, store it

16	keys.put(key, key);

17

}

18	// init lock with the stored reference to the key for the required object

19	lock = keys.get(key);

20

}

21	// grab the lock only on the part of the cache map, so online checks for different hosts can go simultaneously

22	// can't use "key" argument as lock here, since we want to sync on cache's key object

23	synchronized (lock) {

24	Object value = cache.get(key);

25	// maybe object was already updated by other thread while we were waiting for lock?

26	// call the slow process only if that was not happened

27	if (value == STUB) {

28	value = slowProcess.get(key);

29	cache.put(key, value);

30

}

31	return value;

32

}

33

}

34

}

Obviously, we added the “keys” map to search for required lock object quickly. This solves that crap 18-23 lines of Try 3.

What about thread safety? It is putIfAbsent () method of ConcurrentHashMap which helps us a lot.
The method is equal to the following code snippet (except that the action is performed atomically ):

1	if (!map.containsKey(key))

2	return map.put(key, value);

3

else

4	return map.get(key);

Finally, we have fast and thread-safe cache implementation!
Please note, however, all implementations lack renewal of cached objects and this was done by intention for this post. Moreover, implementations suffer from memory leak. Can you see it? That simple — all distinct keys create respective objects in cache and that objects never dropped from the cache (may be replaced, yes). So the map is growing constantly. For the production implementation one may want to implement periodical cleanup of the map from very old objects using TimerTask or as side effect of the get() method call.

I hope this post was not extra boring and more than that I hope the last try is indeed thread-safe :) Have I missed something? Let me know ;)