简介
1.简介
Leveldb是一个google实现的非常高效的kv数据库，目前的版本1.2能够支持billion级别的数据量了。 在这个数量级别下还有着非常高的性能，主要归功于它的良好的设计。特别是LSM算法。
2特点
LevelDB 是单进程的服务，性能非常之高，在一台4核Q6600的CPU机器上，每秒钟写数据超过40w，而随机读的性能每秒钟超过10w。 此处随机读是完全命中内存的速度，如果是不命中 速度大大下降
3局限
LevelDB 只是一个 C/C++ 编程语言的库, 不包含网络服务封装, 所以无法像一般意义的存储服务器(如 MySQL)那样, 用客户端来连接它. LevelDB 自己也声明, 使用者应该封装自己的网络服务器.
4实现介绍
leveldb中有一系列参数会与读写的效率有关，将相关的配置以及编译常量统一修改成可运行时配置参数，测试选取最佳配置值。
应用场景
- 内存占用大的服务
-
leveldb是持久化型数据库
- 写多于读
-
写操作要大大快于读操作，而顺序读写操作则大大快于随机读写操作。官方网站报道其随机写性能达到40万条记录每秒，而随机读性能达到6万条记录每秒
涉及项目
- ActiveMQ
限制和弊端
- key和value数据尺寸不能太大，在KB级别，如果存储较大的key或者value，将对leveld的读写性能都有较大的影响。每个key所对应的内容不宜太大，超过32KB性能就会下降很快
- 一次只允许一个进程访问一个特定的数据库
- 没有内置的C/S架构，但开发者可以使用LevelDB库自己封装一个server，不具备“分布式”集群架构能力
- 百度开源的分布式文件系统BFS(开源地址:https://github.com/baidu/bfs)提 供了mount工具，可以将整个分布式文件系统直接挂载到本地目录，从而可以像操作本地文件一样来操作分布式文件系统中的文件，我们可以利用分布式文件系统本身提供的数据高可靠特性来保证leveldb中数据的安全。
- 不支持索引
- 不支持事务
java调用
原生leveldb是基于C++开发，java语言无法直接使用；iq80对leveldb使用JAVA 语言进行了“逐句”重开发，经过很多大型项目的验证（比如ActiveMQ），iq80开发的JAVA版leveldb在性能上损失极少（10%）。对于JAVA开发人员来说，我们直接使用即可，无需额外的安装其他lib。
java maven依赖
<dependency>
<groupId>org.iq80.leveldb</groupId>
<artifactId>leveldb</artifactId>
<version>0.7</version>
</dependency>
<dependency>
<groupId>org.iq80.leveldb</groupId>
<artifactId>leveldb-api</artifactId>
<version>0.7</version>
</dependency>
备注：最新版0.10, 2017年12月
测试demo
package cn.demo.leveldb;
import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Map;
import org.iq80.leveldb.DB;
import org.iq80.leveldb.DBFactory;
import org.iq80.leveldb.DBIterator;
import org.iq80.leveldb.Options;
import org.iq80.leveldb.ReadOptions;
import org.iq80.leveldb.Snapshot;
import org.iq80.leveldb.WriteBatch;
import org.iq80.leveldb.WriteOptions;
import org.iq80.leveldb.impl.Iq80DBFactory;
import org.junit.Test;
/**
* 各接口测试
* @author lym
*
*/
public class LevelDBDemoTest {
private static final String PATH = "/data/leveldb";
private static final Charset CHARSET = Charset.forName("utf-8");
private static final File FILE = new File(PATH);
@Test
public void putTest() {
DBFactory factory = new Iq80DBFactory();
// 默认如果没有则创建
Options options = new Options();
File file = new File(PATH);
DB db = null;
try {
db = factory.open(file, options);
byte[] keyByte1 = "key-01".getBytes(CHARSET);
byte[] keyByte2 = "key-02".getBytes(CHARSET);
// 会写入磁盘中
db.put(keyByte1, "value-01".getBytes(CHARSET));
db.put(keyByte2, "value-02".getBytes(CHARSET));
String value1 = new String(db.get(keyByte1), CHARSET);
System.out.println(value1);
System.out.println(new String(db.get(keyByte2), CHARSET));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
@Test
public void readFromSnapshotTest() {
DBFactory factory = new Iq80DBFactory();
File file = new File(PATH);
Options options = new Options();
DB db = null;
try {
db = factory.open(file, options);
// 读取当前快照，重启服务仍能读取，说明快照持久化至磁盘，
Snapshot snapshot = db.getSnapshot();
// 读取操作
ReadOptions readOptions = new ReadOptions();
// 遍历中swap出来的数据，不应该保存在memtable中。
readOptions.fillCache(false);
// 默认snapshot为当前
readOptions.snapshot(snapshot);
DBIterator it = db.iterator(readOptions);
while (it.hasNext()) {
Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
.next();
String key = new String(entry.getKey(), CHARSET);
String value = new String(entry.getValue(), CHARSET);
System.out.println("key: " + key + " value: " + value);
if (key.equals("key-01")) {
System.out.println("".equals(value));
}
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
@Test
public void snapshotTest() {
DBFactory factory = new Iq80DBFactory();
Options options = new Options();
DB db = null;
try {
db = factory.open(FILE, options);
db.put("key-04".getBytes(CHARSET), "value-04".getBytes(CHARSET));
// 只能之前到getSnapshot之前put的值，之后的无法获取，即读取期间数据的变更，不会反应出来
Snapshot snapshot = db.getSnapshot();
db.put("key-05".getBytes(CHARSET), "value-05".getBytes(CHARSET));
ReadOptions readOptions = new ReadOptions();
readOptions.fillCache(false);
readOptions.snapshot(snapshot);
DBIterator it = db.iterator(readOptions);
while (it.hasNext()) {
Map.Entry<byte[],byte[]> entry = (Map.Entry<byte[],byte[]>) it
.next();
String key = new String(entry.getKey(), CHARSET);
String value = new String(entry.getValue(), CHARSET);
System.out.println("key: " + key + " value: " + value);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
@Test
public void writeOptionsTest() {
DBFactory factory = new Iq80DBFactory();
Options options = new Options();
DB db = null;
try {
db = factory.open(FILE, options);
WriteOptions writeOptions = new WriteOptions().sync(true); // 线程安全
// 没有writeOptions时，会new一个，所以猜测这里添加了这个参数的意义就是可以设置sync和snapshot参数，建议采用这种方式
db.put("key-06".getBytes(CHARSET), "value-06".getBytes(CHARSET), writeOptions);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
@Test
public void deleteTest() {
DBFactory factory = new Iq80DBFactory();
Options options = new Options();
DB db = null;
try {
db = factory.open(FILE, options);
// 存在会删除，之后查询不出，根据说明可能不是真删除，而是添加一个标记，待测试（大量数据之后删除，文件大小是否明显变化）
db.delete("key-02".getBytes(CHARSET));
// 不存在不会报错
db.delete("key02".getBytes(CHARSET));
Snapshot snapshot = db.getSnapshot();
ReadOptions readOptions = new ReadOptions();
readOptions.fillCache(false);
readOptions.snapshot(snapshot);
DBIterator it = db.iterator(readOptions);
while (it.hasNext()) {
Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
.next();
String key = new String(entry.getKey(), CHARSET);
String value = new String(entry.getValue(), CHARSET);
System.out.println("key: " + key + " value: " + value);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
@Test
public void writeBatchTest() {
DBFactory factory = Iq80DBFactory.factory;
Options options = new Options();
DB db = null;
try {
db = factory.open(FILE, options);
// 批量保存，批量修改
WriteBatch writeBatch = db.createWriteBatch();
writeBatch.put("key-07".getBytes(CHARSET), "value-07".getBytes(CHARSET));
writeBatch.put("key-08".getBytes(CHARSET), "value-08".getBytes(CHARSET));
writeBatch.put("key-09".getBytes(CHARSET), "value-09".getBytes(CHARSET));
// 这里也可以添加writeOptions
db.write(writeBatch);
DBIterator it = db.iterator();
while (it.hasNext()) {
Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
.next();
String key = new String(entry.getKey(), CHARSET);
String value = new String(entry.getValue(), CHARSET);
System.out.println("key: " + key + " value: " + value);
}
it.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
@Test
public void writeBatchDeleteTest() {
DBFactory factory = Iq80DBFactory.factory;
Options options = new Options();
DB db = null;
try {
db = factory.open(FILE, options);
WriteBatch writeBatch = db.createWriteBatch();
writeBatch.put("key-10".getBytes(CHARSET), "value-10".getBytes(CHARSET));
writeBatch.put("key-11".getBytes(CHARSET), "value-11".getBytes(CHARSET));
// 会将key-01的value置为""
writeBatch.delete("key-01".getBytes(CHARSET));
db.write(writeBatch);
writeBatch.close();
DBIterator it = db.iterator();
while (it.hasNext()) {
Map.Entry<byte[], byte[]> entry = (Map.Entry<byte[], byte[]>) it
.next();
String key = new String(entry.getKey(), CHARSET);
String value = new String(entry.getValue(), CHARSET);
System.out.println("key: " + key + " value: " + value);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}
性能测试
| 数据量 | 存储 | 随机读一条 | 顺序读一条 | 顺序读所有 | 磁盘占用 | 测试环境 | 速率 |
|---|---|---|---|---|---|---|---|
| 10w | 494ms | 36ms | -- | 174ms | 9.75M | windows | 20.29w/s |
| 40w | 2772ms | 46ms | 1ms | 551ms | 28.6M | windows | 14.43w/s |
| 100w | 5533ms | 44ms | 1ms | 457ms | 71.5M | windows | 18.07w/s |
| 1000w | 117661ms | 11m+ | -- | -- | 715M | windows | -- |
数据类型
1. 对象
需要实现序列化接口
public class DataTest {
private static final String PATH = "/data/leveldb";
private static final File FILE = new File(PATH);
private static final Charset CHARSET = Charset.forName("UTF-8");
@Test
public void writeObject() {
Options options = new Options();
DBFactory factory = Iq80DBFactory.factory;
DB db = null;
try {
db = factory.open(FILE, options);
User user = new User();
user.setName("admin");
user.setSex("男");
WriteOptions writeOptions = new WriteOptions();
writeOptions.snapshot(true);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(user);
db.put(user.getName().getBytes(CHARSET), baos.toByteArray());
} catch (IOException e) {
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
@Test
public void readObject() {
Options options = new Options();
DBFactory factory = Iq80DBFactory.factory;
DB db = null;
try {
db = factory.open(FILE, options);
byte[] valueByte = db.get("admin".getBytes(CHARSET));
ByteArrayInputStream bais = new ByteArrayInputStream(valueByte);
ObjectInputStream ois = new ObjectInputStream(bais);
User user = (User) ois.readObject();
System.out.println(user);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}
2. 对象集合
@Test
public void writeObjectList() {
Options options = new Options();
DBFactory factory = Iq80DBFactory.factory;
DB db = null;
try {
db = factory.open(FILE, options);
List<User> userList = new ArrayList<User>();
User user = new User();
user.setName("admin");
user.setSex("男");
User user2 = new User();
user2.setName("root");
user2.setSex("女");
userList.add(user);
userList.add(user2);
WriteOptions writeOptions = new WriteOptions();
writeOptions.snapshot(true);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(userList);
db.put("userList".getBytes(CHARSET), baos.toByteArray());
} catch (IOException e) {
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
@Test
public void readObjectList() {
Options options = new Options();
DBFactory factory = Iq80DBFactory.factory;
DB db = null;
try {
db = factory.open(FILE, options);
byte[] valueByte = db.get("userList".getBytes(CHARSET));
ByteArrayInputStream bais = new ByteArrayInputStream(valueByte);
ObjectInputStream ois = new ObjectInputStream(bais);
List<User> userList = new ArrayList<User>();
userList = (List<User>) ois.readObject();
System.out.println(userList);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (db != null) {
try {
db.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
参考：
扫一扫
961
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
