Storing float array as value in redis performance test

Background

I am looking for a method to store float array as hash value, given that this float array is a 128 length float array, with an array item like 0.123456789. Which serialization and deserialization could I use?

Result

case 1. float array has very low accuracy, like 0.1f

• When storing float array as binary value for hset, 10000 times, the size of hash key is 6.675 MB. And time consumption is 80623ms

127.0.0.1:6379> debug object binary_value_test_binary
Value at:0x7f76e8661a30 refcount:1 encoding:hashtable serializedlength:549107 lru:5480008 lru_seconds_idle:134
127.0.0.1:6379> memory usage binary_value_test_binary
(integer) 6999488

• json value, size is 6.673 MB, time consumption is 80171ms.

127.0.0.1:6379> memory usage “binary_value_test_json”
(integer) 6996768
127.0.0.1:6379> debug object “binary_value_test_json”
Value at:0x7f76e8661860 refcount:1 encoding:hashtable serializedlength:288892 lru:5481000 lru_seconds_idle:264

case 2. float array has a higher accuracy, like this:

new float[]{ 0.060549404472112656f,
                -0.13050515949726105f,
                0.029610045254230499f,
                0.13573043048381805f,
                0.09077821671962738f,
                ...

127.0.0.1:6379> debug object “binary_value_test_json”
Value at:0x7f76e85b6e90 refcount:1 encoding:hashtable serializedlength:11508892 lru:5482608 lru_seconds_idle:4394
127.0.0.1:6379> debug object “binary_value_test_binary”
Value at:0x7f76e20e7f10 refcount:1 encoding:hashtable serializedlength:5508892 lru:5482691 lru_seconds_idle:4322
127.0.0.1:6379> memory usage “binary_value_test_json”
(integer) 15956768
127.0.0.1:6379> memory usage “binary_value_test_binary”
(integer) 6996776

Json mem size: 15.21 MB, time takes: 89570 ms
Binary mem size: 6.67 MB, time takes: 82886 ms

case 3. self defined float array de/serialization, based on the memory size, this method is the same as Java de/serialization.
mem size: 6.67 MB, time takes: 82280ms

127.0.0.1:6379> memory usage “binary_value_test_binary”
(integer) 6996776
127.0.0.1:6379> memory usage “binary_value_test_bytes”
(integer) 6996776

Conclusions

1. Redis float array as value, using Java object serialization would have a steady memory consumption regardless of accuracy. While Json serialization would increase giantly when accuracy is higher.
2. Time consumption does not vary much. When accuracy is higher, json requires slightly more time to serialize than binary.
3. ByteBuffer is the method used by Java object serialization.

Code

Code here is not for thorough purpose, it only shows idea what is used (like reactiveRedisTemplate configurations) and how the conclusion is gained.

    private <T> Jackson2JsonRedisSerializer<T> configureJackson2JsonRedisSerializer(Class<T> t) {
        ObjectMapper objectMapper = new ObjectMapper()
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        Jackson2JsonRedisSerializer<T> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer<>(t);
        jackson2JsonRedisSerializer.setObjectMapper(objectMapper);
        return jackson2JsonRedisSerializer;
    }

    /**
     * String, String, Object for hash.
     * @param connectionFactory connection
     * @return ReactiveRedisOperations that is for String and AiVec
     */
    @Bean
    ReactiveRedisOperations<String, float[]> redisOperationsMatrix(ReactiveRedisConnectionFactory connectionFactory) {
        Jackson2JsonRedisSerializer<float[]> serializer = configureJackson2JsonRedisSerializer(float[].class);
        RedisSerializationContext.RedisSerializationContextBuilder<String, float[]> builder =
                newSerializationContext(new StringRedisSerializer());
        RedisSerializationContext<String, float[]> context = builder.hashValue(serializer).build();
        return new ReactiveRedisTemplate<>(connectionFactory, context);
    }

    /**
     * Redis string key and field, binary as value.
     * @param connectionFactory
     * @return
     */
    @Bean
    ReactiveRedisOperations<String, float[]> redisBinaryOperations(ReactiveRedisConnectionFactory connectionFactory) {
        JdkSerializationRedisSerializer serializer =  new JdkSerializationRedisSerializer();
        RedisSerializationContext.RedisSerializationContextBuilder<String, float[]> builder =
                newSerializationContext(new StringRedisSerializer());
        RedisSerializationContext<String, float[]> context = builder.hashValue(serializer).build();
        return new ReactiveRedisTemplate<>(connectionFactory, context);
    }




    @Bean
    ReactiveRedisOperations<String, float[]> redisByteOperations(ReactiveRedisConnectionFactory connectionFactory) {
        JdkSerializationRedisSerializer jdkSerializationRedisSerializer =  new JdkSerializationRedisSerializer(serializer, deserializer);
        RedisSerializationContext.RedisSerializationContextBuilder<String, float[]> builder =
                newSerializationContext(new StringRedisSerializer());
        RedisSerializationContext<String, float[]> context = builder.hashValue(jdkSerializationRedisSerializer).build();
        return new ReactiveRedisTemplate<>(connectionFactory, context);
    }

    static Converter<Object, byte[]> serializer = RedisConfig::floatArray2ByteArray;

    static Converter<byte[], Object> deserializer = RedisConfig::byteArray2FloatArray;

    static byte[] floatArray2ByteArray(Object o){
        float[] values = (float[])(o);
        ByteBuffer buffer = ByteBuffer.allocate(4 * values.length);
        for (float value : values){
            buffer.putFloat(value);
        }
        return buffer.array();
    }

    static Object byteArray2FloatArray(byte[] values) {
        float[] array = new float[values.length/4];
        for (int i = 0; i< values.length/4; i++) {
            array[i] = ByteBuffer.wrap(values, 4*i, 4).getFloat();

        }
        return array;
    }

 @Test
    void test_mem_size_binary() {
        ReactiveHashOperations<String, String, float[]> ops = redisBinaryOperations.opsForHash();
        int fields = 10000;
        float[] floats = new float[128];
        Arrays.fill(floats, 0.1f);
        Boolean result = true;
        long a = System.currentTimeMillis();
        for (int i = 0; i< fields; i++) {
            result = ops.put("binary_value_test_binary", "field"+i, getFloat()).block();
        }
        long r = a - System.currentTimeMillis();
        log.error("binary time consumption: {}", r);
        Assert.assertTrue(result);
    }

    @Test
    void test_mem_size_json() {
        ReactiveHashOperations<String, String, float[]> ops = redisOperationsMatrix.opsForHash();
        int fields = 10000;
//        float[] floats = new float[128];
//        Arrays.fill(floats, 0.1f);
        Boolean result = true;
        long a = System.currentTimeMillis();
        for (int i = 0; i< fields; i++) {
            result = ops.put("binary_value_test_json", "field"+i, getFloat()).block();
        }
        long r = a - System.currentTimeMillis();
        log.error("json time consumption: {}", r);
        Assert.assertTrue(result);
    }

    @Test
    void test_mem_size_byte() {
        ReactiveHashOperations<String, String, float[]> ops = redisByteOperations.opsForHash();
        int fields = 10000;
//        float[] floats = new float[128];
//        Arrays.fill(floats, 0.1f);
        Boolean result = true;
        long a = System.currentTimeMillis();
        for (int i = 0; i< fields; i++) {
            result = ops.put("binary_value_test_bytes", "field"+i, getFloat()).block();
        }
        long r = a - System.currentTimeMillis();
        log.error("json time consumption: {}", r);
        Assert.assertTrue(result);
    }



    private static float[] getFloat() {
        return new float[]{ 0.060549404472112656f,
                -0.13050515949726105f,
                0.029610045254230499f,
                0.13573043048381805f,
                0.09077821671962738f,
                0.11595767736434937f,
                -0.12132424116134644f,
                0.067579329013824463f,
                -0.11920245736837387f,
                0.0092990463599562645f,
                -0.061574112623929977f,
                0.081587500870227814f,
                -0.098826110363006592f,
                -0.012550449930131435f,
                -0.12075814604759216f,
                -0.048236701637506485f,
                0.087256878614425659f,
                -0.10304808616638184f,
                -0.077081143856048584f,
                0.0023754285648465157f,
                -0.088484294712543488f,
                -0.044642534106969833f,
                -0.0086963847279548645f,
                -0.17266415059566498f,
                0.025830376893281937f,
                -0.10278327763080597f,
                -0.048227414488792419f,
                0.078207708895206451f,
                0.0060340459458529949f,
                -0.097168870270252228f,
                -0.05961267277598381f,
                0.13065017759799957f,
                0.12461724132299423f,
                0.06871052086353302f,
                0.041623726487159729f,
                -0.038920365273952484f,
                -0.013832301832735538f,
                -0.19563817977905273f,
                0.018485559150576591f,
                -0.011079077608883381f,
                0.093613907694816589f,
                -0.0570094995200634f,
                0.056410878896713257f,
                0.036739233881235123f,
                -0.15818977355957031f,
                -0.013796190731227398f,
                0.022542010992765427f,
                -0.063153348863124847f,
                -0.13398076593875885f,
                -0.021926335990428925f,
                0.070352151989936829f,
                -0.096275709569454193f,
                -0.066654443740844727f,
                -0.12282037734985352f,
                -0.11608711630105972f,
                -0.074962794780731201f,
                0.22098256647586823f,
                -0.020336128771305084f,
                0.004638939630240202f,
                0.011781537905335426f,
                0.078441217541694641f,
                0.093886949121952057f,
                0.24695055186748505f,
                0.094254061579704285f,
                -0.10100533068180084f,
                0.067084647715091705f,
                -0.029616802930831909f,
                -0.11150515079498291f,
                -0.019073788076639175f,
                0.037771318107843399f,
                -0.060800932347774506f,
                0.11242368072271347f,
                0.15660306811332703f,
                0.030853740870952606f,
                -0.015274095349013805f,
                0.13472853600978851f,
                0.1739380806684494f,
                0.024676920846104622f,
                0.074140824377536774f,
                -0.012744366191327572f,
                -0.11592743545770645f,
                -0.14305759966373444f,
                0.0097575774416327477f,
                0.04766475036740303f,
                -0.13701018691062927f,
                0.13264086842536926f,
                -0.081548646092414856f,
                0.031657211482524872f,
                0.032816994935274124f,
                0.12177889794111252f,
                -0.032813519239425659f,
                0.0044775740243494511f,
                -0.054822161793708801f,
                0.07960096001625061f,
                0.12672607600688934f,
                0.047396760433912277f,
                -0.13493023812770844f,
                -0.058519076555967331f,
                0.068193525075912476f,
                0.066171750426292419f,
                0.016757212579250336f,
                0.069702669978141785f,
                -0.022635810077190399f,
                0.047671616077423096f,
                -0.072486370801925659f,
                0.054383318871259689f,
                0.04323108121752739f,
                0.045912269502878189f,
                0.087737560272216797f,
                -0.017834894359111786f,
                0.063143074512481689f,
                -0.11565015465021133f,
                -0.12990054488182068f,
                -0.0057650241069495678f,
                -0.020651161670684814f,
                -0.1082155779004097f,
                0.093308813869953156f,
                0.096329562366008759f,
                -0.10402931272983551f,
                0.028056671842932701f,
                0.082366921007633209f,
                0.028657946735620499f,
                0.011522756889462471f,
                -0.090141892433166504f,
                -0.053435511887073517f,
                -0.0046332459896802902f,
                -0.13690854609012604f,
                0.12587848305702209f};

    }