YCSB 压测hbase

使用YCSB对HBase进行压测 (一)YCSB介绍: YCSB,全称为“Yahoo!Cloud Serving Benchmark”。是雅虎开发的用来对云服务进行基础测试的工具,其内部涵盖了常见的NoSQL数据库产品,如Cassandra、MongoDB、HBase、Redis等等。在运行YCSB的时候,可以配置不同的workload和DB,也可以指定线程数&并发数等其他参数。 (二)使用姿势: 以最常见的HBas 阅读详情

文章来源 https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload

 

安装包下载

链接:https://pan.baidu.com/s/1GBBtCdj4ArRiQA1DV3o-Rg 密码:yvv2

 

关键:hbase 创建空表 create 'usertable',{NMAE=>'f1'}

使用ycsb的脚本会根据配置的脚本,选择插入的数据量和读写的比例,之后会生成一个报告,关于读写的时间,平均时间,等等指标。

 

第一步生产数据

[root@mini1 ycsb-0.12.0]# ./bin/ycsb load hbase10 -P workloads/workloada -p columnfamily=f1 -s > load.dat

第二步测试

[root@mini1 ycsb-0.12.0]# ./bin/ycsb run hbase10 -P workloads/workloada -p columnfamily=f1 -s > transactions.dat

 

上面参数含义:

  • hbase-10 说明测试的是针对hbase1.x版本
  • -p columnfamily=cf 插入hbase的列簇cf
  • -p recordcount=100000000 总计插入1亿条数据 【可选】
  • -jvm-args '-Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m' 设置JVM参数【可选】
  • -p threadcount=100 客户端线程数为100,并发100【可选】
  • -s load.dat 记录测试结果到 load.dat
  • -P workloads/workloada 采用默认配置workloads/workloada,里面设置了插入占比50%,更新占比50%,默认往cf列簇插入10个列的数据,每个列的数据大小为100bytes,10个列的话总计大约1K大小数据

 

全文如下:

Running a Workload

Step 1. Set up the database system to test

The first step is to set up the database system you wish to test. This can be done on a single machine or a cluster, depending on the configuration you wish to benchmark.

You must also create or set up tables/keyspaces/storage buckets to store records. The details vary according to each database system, and depend on the workload you wish to run. Before the YCSB Client runs, the tables must be created, since the Client itself will not request to create the tables. This is because for some systems, there is a manual (human-operated) step to create tables, and for other systems, the table must be created before the database cluster is started.

The tables that must be created depends on the workload. For CoreWorkload, the YCSB Client will assume that there is a "table" called "usertable" with a flexible schema: columns can be added at runtime as desired. This "usertable" can be mapped into whatever storage container is appropriate. For example, in MySQL you would "CREATE TABLE," in Cassandra you would define a keyspace in the Cassandra configuration, and so on. The database interface layer (described in step 2) will receive requests for reading or writing records in "usertable" and translate them into requests for the actual storage you have allocated. This may mean that you have to provide information for the database interface layer to help it understand what the structure of the underlying storage is. For example, in Cassandra, you must define "column families" in addition to keyspaces. Thus, it is necessary to create a column family and give the family some name (for example, you might use "values.") Then, the database access layer will need to know to refer to the "values" column family, either because the string "values" is passed in as a property, or because it is hardcoded in the database interface layer.

Step 2. Choose the appropriate DB interface layer

The DB interface layer is a java class that execute read, insert, update, delete and scan calls generated by the YCSB Client into calls against your database's API. This class is a subclass of the abstract DB class in the com.yahoo.ycsb package. You will specify the class name of the layer on the command line when you run YCSB Client, and the Client will dynamically load your interface class. Any properties specified on the command line, or in parameter files specified on the command line, will be passed to the DB interface instance, and can be used to configure the layer (for example, to tell it the hostname of the database you are benchmarking).

The YCSB Client is distributed with a simple dummy interface layer, com.yahoo.ycsb.BasicDB. This layer just prints the operations it would have executed to System.out. It can be useful for ensuring that the client is operating properly, and for debugging your workloads.

See Using the Database Libraries for details on how to use the clients while running YCSB. For more details about implementing a DB interface layer, see Adding a Database.

You can run commands directly against the database using the ycsb command. This client uses the DB interface layer to send commands to the database. You can use this client to make sure that the DB layer is working properly, that your database is set up correctly, that the DB layer can connect to the database, and so on. It also provides a common interface for a variety of databases, and can be used to inspect data in the database. To run the command line client:

$ ./bin/ycsb shell basic > help Commands: read key [field1 field2 ...] - Read a record scan key recordcount [field1 field2 ...] - Scan starting at key insert key name1=value1 [name2=value2 ...] - Insert a new record update key name1=value1 [name2=value2 ...] - Update a record delete key - Delete a record table [tablename] - Get or [set] the name of the table quit - Quit

Step 3. Choose the appropriate workload

The workload defines the data that will be loaded into the database during the loading phase, and the operations that will be executed against the data set during the transaction phase.

Typically, a workload is a combination of:

  • Workload java class (subclass of com.yahoo.ycsb.Workload)
  • Parameter file (in the Java Properties format)

Because the properties of the dataset must be known during the loading phase (so that the proper kind of record can be constructed and inserted) and during the transaction phase (so that the correct record ids and fields can be referred to) a single set of properties is shared among both phases. Thus the parameter file is used in both phases. The workload java class uses those properties to either insert records (loading phase) or execute transactions against those records (transaction phase). The choice of which phase to run is based on a parameter you specify when you run the ycsb command.

You specify both the java class and the parameter file on the command line when you run the YCSB Client. The Client will dynamically load your workload class, pass it the properties from the parameters file (and any additional properties specified on the command line) and then execute the workload. This happens both for the loading and transaction phases, as the same properties and workload logic applies to both. For example, if the loading phase creates records with 10 fields, then the transaction phase must know that there are 10 fields it can query and modify.

The CoreWorkload is a package of standard workloads that is distributed with the YCSB and can be used directly. In particular, the CoreWorkload defines a simple mix of read/insert/update/scan operations. The relative frequency of each operation is defined in the parameter file, as are other properties of the workload. Thus, by changing the parameter file, a variety of different concrete workloads can be executed. For more details on the CoreWorkload, see Core Workloads.

If the CoreWorkload does not satisfy your needs, you can define your own workload by subclassing the com.yahoo.ycsb.Workload class. Details for doing this are Implementing New Workloads.

Step 4. Choose the appropriate runtime parameters

Although the workload class and parameters file define a specific workload, there are additional settings that you may want to specify for a particular run of the benchmark. These settings are provided on the command line when you run the YCSB client. These settings are:

  • -threads : the number of client threads. By default, the YCSB Client uses a single worker thread, but additional threads can be specified. This is often done to increase the amount of load offered against the database.
  • -target : the target number of operations per second. By default, the YCSB Client will try to do as many operations as it can. For example, if each operation takes 100 milliseconds on average, the Client will do about 10 operations per second per worker thread. However, you can throttle the target number of operations per second. For example, to generate a latency versus throughput curve, you can try different target throughputs, and measure the resulting latency for each.
  • -s : status. for a long running workload, it may be useful to have the Client report status, just to assure you it has not crashed and to give you some idea of its progress. By specifying "-s" on the command line, the Client will report status every 10 seconds to stderr.

Step 5. Load the data

Workloads have two executable phases: the loading phase (which defines the data to be inserted) and the transactions phase (which defines the operations to be executed against the data set). To load the data, you run the YCSB Client and tell it to execute the loading section.

For example, consider the benchmark workload A (more details about the standard workloads are in Core Workloads). To load the standard dataset:

$ ./bin/ycsb load basic -P workloads/workloada

A few notes about this command:

  • The load parameter tells the Client to execute the loading section of the workload.
  • The basic parameter tells the Client to use the dummy BasicDB layer. You can also specify this as a property in your parameters file using the "db" property (for example, "db=com.yahoo.ycsb.BasicDB").
  • The "-P" parameter is used to load property files. In this case, we used it to load the workloada parameter file.

To load the HBase dataset:

$ ./bin/ycsb load hbase -P workloads/workloada -p columnfamily=family

A few notes about this command:

  • The load parameter tells the Client to execute the loading section of the workload.
  • The hbase parameter tells the Client to use the HBase layer.
  • The -P parameter is used to load property files. In this case, we used it to load the workloada parameter file.
  • The -p parameter is used to set parameter. In this HBase case, we used it to set database column. You should have database usertable with column family before running this command. Then all data will be loaded into database usertable with column family.
  • Make sure you have already started Hadoop and HBase before you run this command.

If you used BasicDB, you would see the insert statements for the database. If you used a real DB interface layer, the records would be loaded into the database.

The standard workload parameter files create very small databases; for example, workloada creates only 1,000 records. This is useful while debugging your setup. However, to run an actual benchmark you'll want to generate a much larger database. For example, imagine you want to load 100 million records. Then, you will need to override the default "recordcount" property in the workloada file. This can be done in one of two ways:

  • Specifying a new property file containing a new value of recordcount. If this file is specified on the command line after the workloada file, it will override any properties in workloada. For example, create a file called "large.dat" with the single line:

recordcount=100000000

Then, run the client as follows:

$ ./bin/ycsb load basic -P workloads/workloada -P large.dat

The client will load both property files, but will use the value of recordcount from the last file it loaded, e.g. large.dat.

  • Specify a new value of the recordcount property on the command line. Any properties specified on the command line override properties specified in property files. In this case, run the client as follows:

$ ./bin/ycsb load basic -P workloads/workloada -p recordcount=100000000

In general, it is good practice to store any important properties in new parameter files, instead of specifying them on the command line. This makes your benchmark results more reproducible; instead of having to reconstruct the exact command line you used, you just reuse the property files. Note, however, that the YCSB Client will print out its command line when it begins executing, so if you store the output of the client in a data file, you can retrieve the command line from that file.

Because a large database load will take a long time, you may wish to 1. require the Client to produce status, and 2. direct any output to a datafile. Thus, you might execute the following to load your database:

$ ./bin/ycsb load basic -P workloads/workloada -P large.dat -s > load.dat

The -s parameter will require the Client to produce status report on stderr. Thus, the output of this command might be:

$ ./bin/ycsb load basic -P workloads/workloada -P large.dat -s > load.dat Loading workload... (might take a few minutes in some cases for large data sets) Starting test. 0 sec: 0 operations 10 sec: 61731 operations; 6170.6317473010795 operations/sec 20 sec: 129054 operations; 6450.76477056883 operations/sec ...

This status output will help you to see how quickly load operations are executing (so you can estimate the completion time of the load) as well as verify that the load is making progress. When the load completes, the Client will report statistics about the performance of the load. These statistics are the same as in the transaction phase, so see below for information on interpreting those statistics.

Step 6: Execute the workload

Once the data is loaded, you can execute the workload. This is done by telling the client to run the transaction section of the workload. To execute the workload, you can use the following command:

$ ./bin/ycsb run basic -P workloads/workloada -P large.dat -s > transactions.dat

The main difference in this invocation is that we used the run parameter to tell the Client to use the transaction section instead of the loading section. If you used BasicDB, and examine the resulting transactions.dat file, you will see a combination of read and update requests, as well as statistics about the execution.

Typically you will want to use the -threads and -target parameters to control the amount of offered load. For example, we might want 10 threads attempting a total of 100 operations per second (e.g. 10 operations/sec per thread.) As long as the average latency of operations is not above 100 ms, each thread will be able to carry out its intended 10 operations per second. In general, you need enough threads so that no thread is attempting more operations per second than is possible, otherwise your achieved throughput will be less than the specified target throughput. For this example, we can execute:

$ ./bin/ycsb run basic -P workloads/workloada -P large.dat -s -threads 10 -target 100 > transactions.dat

Note in this example we have used the -threads 10 command line parameter to specify 10 threads, and -target 100 command line parameter to specify 100 operations per second as the target. Alternatively, both values can be set in your parameters file using the threadcount and target properties respectively. For example:

threadcount=10 target=100

At the end of the run, the Client will report performance statistics on stdout. In the above example, these statistics will be written to the transactions.dat file. The default is to produce average, min, max, 95th and 99th percentile latency for each operation type (read, update, etc.), a count of the return codes for each operation, and a histogram of latencies for each operation. The return codes are defined by your database interface layer, and allow you to see if there were any errors during the workload. For the above example, we might get output like:

[OVERALL],RunTime(ms), 10110 [OVERALL],Throughput(ops/sec), 98.91196834817013 [UPDATE], Operations, 491 [UPDATE], AverageLatency(ms), 0.054989816700611 [UPDATE], MinLatency(ms), 0 [UPDATE], MaxLatency(ms), 1 [UPDATE], 95thPercentileLatency(ms), 1 [UPDATE], 99thPercentileLatency(ms), 1 [UPDATE], Return=0, 491 [UPDATE], 0, 464 [UPDATE], 1, 27 [UPDATE], 2, 0 [UPDATE], 3, 0 [UPDATE], 4, 0 ...

This output indicates:

  • The total execution time was 10.11 seconds
  • The average throughput was 98.9 operations/sec (across all threads)
  • There were 491 update operations, with associated average, min, max, 95th and 99th percentile latencies
  • All 491 update operations had a return code of zero (success in this case)
  • 464 operations completed in less than 1 ms, while 27 completed between 1 and 2 ms.

Similar statistics are available for the read operations.

While a histogram of latencies is often useful, sometimes a timeseries is more useful. To request a time series, specify the "measurementtype=timeseries" property on the Client command line or in a properties file. By default, the Client will report average latency for each interval of 1000 milliseconds. You can specify a different granularity for reporting using the "timeseries.granularity" property. For example:

$ ./bin/ycsb run basic -P workloads/workloada -P large.dat -s -threads 10 -target 100 -p \ measurementtype=timeseries -p timeseries.granularity=2000 > transactions.dat

will report a timeseries, with readings averaged every 2,000 milliseconds (e.g. 2 seconds). The result will be:

[OVERALL],RunTime(ms), 10077 [OVERALL],Throughput(ops/sec), 9923.58836955443 [UPDATE], Operations, 50396 [UPDATE], AverageLatency(ms), 0.04339630129375347 [UPDATE], MinLatency(ms), 0 [UPDATE], MaxLatency(ms), 338 [UPDATE], Return=0, 50396 [UPDATE], 0, 0.10264765784114054 [UPDATE], 2000, 0.026989343690867442 [UPDATE], 4000, 0.0352882703777336 [UPDATE], 6000, 0.004238958990536277 [UPDATE], 8000, 0.052813085033008175 [UPDATE], 10000, 0.0 [READ], Operations, 49604 [READ], AverageLatency(ms), 0.038242883638416256 [READ], MinLatency(ms), 0 [READ], MaxLatency(ms), 230 [READ], Return=0, 49604 [READ], 0, 0.08997245741099663 [READ], 2000, 0.02207505518763797 [READ], 4000, 0.03188493260913297 [READ], 6000, 0.004869141813755326 [READ], 8000, 0.04355329949238579 [READ], 10000, 0.005405405405405406

This output shows separate time series for update and read operations, with data reported every 2000 milliseconds. The data reported for a time point is the average over the previous 2000 milliseconds only. (In this case we used 100,000 operations and a target of 10,000 operations per second for a more interesting output). A note about latency measurements: the Client measures the end to end latency of executing a particular operation against the database. That is, it starts a timer before calling the appropriate method in the DB interface layer class, and stops the timer when the method returns. Thus latencies include: executing inside the interface layer, network latency to the database server, and database execution time. They do not include delays introduced for throttling the target throughput. That is, if you specify a target of 10 operations per second (and a single thread) then the Client will only execute an operation every 100 milliseconds. If the operation takes 12 milliseconds, then the client will wait for an additional 88 milliseconds before trying the next operation. However, the reported latency will not include this wait time; a latency of 12 milliseconds, not 100, will be reported.

YCSB配置和测试HBase 本文主要介绍如何使用YCSB来测试HBase,以及如何解决No FileSystem for scheme: hdfs错误和Error accessing HBase table错误。 YCSB项目地址和简介:https://github.com/brianfrankcooper/YCSB/wiki YCSB的目标是开发一个框架和通用的workloads来评估不同的key-value和 阅读详情

相关推荐

YCSB之HBase压力测试.pdf

本文讲解如何安装和配置YCSB,如何使用YCSB给Hbase加载测试数据,如何使用YCSB命令测试Hbase读写性能。主要解释了各个命令行参数的作用。

HBase使用YCSB进行性能测试

Alpine docker中的HBase使用YCSB进行压力测试

weixin_44171618的博客 1026

海思ISP调优实战:从实验室到真实场景的IPC相机图像质量优化指南

本文深入探讨海思ISP调优在IPC相机图像质量优化中的实战应用,从实验室标定到真实场景部署的全链路调优方法。重点解决光照动态范围、色温变化和噪声特性等核心问题,提供AE权重表配置、CCM标定、3DNR时域滤波等关键技术方案,帮助实现从理论到实践的图像质量提升。

s3t4u的博客 751

教程,使用YCSB测试MYSQL数据库,获取千万条测试后的数据

Yahoo! Cloud Serving Benchmark (YCSB)是一个数据库特别是nosql数据库性能测试的benchmark。在GitHub中有3.9千个star和1.9千个forks。本文讲解如何使用YCSB测试MYSQL数据库的性能。 本人最终的结果是获取测试之后保存在MYSQL中的数据,需要10000000条记录。这个过程包含性能测试的步骤。 系统:Windows 10; MYSQL5.7 安装MYSQL数据库,(测试能够登录进去)。 创建库和table。根据这个链接 https://g

ChainingBlocks 4070

WorkBuddy Skill赋能HYSYS与Comsol:AI工作台下的仿真自动化实践

AI Agent正将工业软件自动化带向新高度。在工程仿真领域,流程模拟与多物理场分析常因重复性操作而效率低下,Aspen HYSYS的批量工况调整、Comsol的参数扫描与网格重构均属此类。WorkBuddy这类AI工作台通过Skill将软件二次开发能力封装为可插拔技能包,利用HYSYS的COM/OLE接口和Comsol的Java API,实现自然语言指令到自动化执行的转变。其技术价值在于降低自动化门槛、沉淀工程经验,并支持日志审计与异常重试。在实际应用中,无论化工流程的灵敏度分析,还是激光焊接、移动网格等

weixin_29055699的博客 307

JavaFx之集成FontAwesomeFX与JFoenix(十五)

本文详细介绍了如何在JavaFx项目中同时集成JFoenix和FontAwesomeFX,以创建现代化、美观的桌面应用界面。通过Maven依赖配置、动态代码创建和FXML静态声明两种方式,演示了如何将JFoenix的Material Design控件与FontAwesomeFX的丰富矢量图标完美结合,并提供了实战案例展示如何构建一个功能完善的现代化工具栏。

pepsi的博客 438

YCSB压测MongoDB

一.前言 现在有个项目,五年时间会产生一亿条数据,所以领导想知道使用mongodb数据库,一亿条数据量中查询一条数据的响应时间。

shuijiaxiaowei的博客 1931

C语言获取硬件信息(CPU序列号,硬盘序列号,网卡IP、MAC地址、是否插入网线)

本文主要介绍Linux和Windows下使用C语言获取各种硬件信息,包括CPU序列号、硬盘序列号、网卡信息(包括网卡名字、IP地址、MAC地址、网卡是否插入网线等)。

叶落花枯的博客 9381

使用 YCSB 和 PE 进行 HBase 性能压力测试

HBase主要性能压力测试有两个,一个是 HBase 自带的 PE,另一个是 YCSB,先简单说一个两者的区别。PE 是 HBase 自带的工具,开箱即用,使用起来非常简单,但是 PE 只能按单个线程统计压测结果,不能汇总整体压测数据,更重要的是,PE 没有 YCSB 的 预设模板(Workload) 功能,测试场景单一,相较而言,YCSB 要强大的多,它的 Workload 功能非常实用,可以模拟更贴近实际使用场景的压力状况。

Laurence的技术博客 3204

在Qt中使用EtherCAT-SOEM的初始化中,状态机卡在PreOp->Save-Op过程中,出现State=0x12, Error=0x001e错误的解决办法

1.在调试EtherCAT软件过程中,一直出现配置从站进不了Save_op状态,通过下函数检查从站状态: //使用的从站编号为1 只有1个从站 ec_slave[1].state = EC_STATE_SAFE_OP; ec_writestate(1); //设置到安全模式 ec_statecheck(1, EC_STATE_SAFE_OP, EC_TIMEOUTSTATE * 4); printf("Slave %d State=0x%2.2x StatusCode=0x%4

zeal_for_rov的博客 6829

零成本接入Gemini Pro:GCP+NewAPI+OpenCode全栈方案

AI编程助手在现代开发流程中扮演着越来越重要的角色,其核心原理是通过大语言模型理解开发者意图并生成代码。以Gemini Pro为代表的顶级模型虽然能力出众,但成本控制和接入复杂度成为主要痛点。通过Google Cloud Platform的免费额度与开源工具链组合,开发者可以构建零成本的AI辅助开发环境。该方案采用Vertex AI作为基础模型服务,通过New API实现协议转换,最终与OpenCode终端工具深度集成。这种架构特别适合需要长期稳定使用AI编程助手的开发者,既能享受Gemini Pro的百万

weixin_29051245的博客 431

yz-ycsb压测hbase记录

环境准备 数据准备:usertable 100 个region 并灌入数据。 hbase shell>n_splits=100 hbase shell> create 'usertable',{NAME=>'cf',DATA_BLOCK_ENCODING=>'DIFF',COMPRESSION=&gt...

liyanling的博客 512

基于Gossip算法的柔性互联自治微网群分布式协调控制策略:复现代码与详细解释

摘要(149字): 本文基于Gossip算法实现了柔性互联微网群(MMG)的分布式协调控制策略仿真。研究构建了包含3个微网、10个分布式电源(DG)的MATLAB/Simulink模型,采用双层控制架构:上层通过柔性交流开关(FAS)实现微网间功率协调,下层利用Gossip算法完成DG间的边际成本一致性控制。重点开发了异步通信的Gossip算法核心函数,实现了分布式电源的自主边际成本估计与功率分配。仿真验证了系统在负荷变化和DG故障等工况下的协调控制能力,结果表明该策略能有效维持微网群的经济运行与功率平衡。

huanghm88的专栏 281

HBase压测 ycsb

【代码】HBase压测 ycsb。

1138

Python电商网站全站爬虫实战:以京东为例的数据采集与分析

随着电子商务的蓬勃发展,电商数据已成为市场研究、竞争分析和消费者行为研究的重要资源。本文详细介绍了如何使用Python最新技术栈构建一个高效的电商网站全站爬虫,以京东为例实现商品数据的自动化采集。文章涵盖了爬虫的法律与伦理考量、技术选型、反爬虫策略应对、分布式架构设计以及数据存储方案,并提供了完整的代码实现。本爬虫系统采用Scrapy-Redis分布式框架、Selenium动态渲染、智能代理轮换和机器学习验证码识别等先进技术,能够稳定高效地抓取京东全站商品数据,为电子商务研究提供数据支持。关键词。

2201_76125261的博客 416

android 蓝牙通讯demo

android 下蓝牙通讯demo,实现了发现蓝牙,使能蓝牙,通过蓝牙发送消息。

雅虎(ycsb)测试hbase(压测)

一、下载ycsb 0.10包 https://github.com/brianfrankcooper/YCSB/releases/download/0.10.0/ycsb-0.10.0.tar.gz 二、上传到linux,解压 tar -zxvfycsb-0.10.0.tar.gz 三、YSCB压力测试hbase流程 1、进入workloads目录下进行文件配置:比如:...

weixin_30580943的博客 689

LinePhone3.2.7(android端最新版本) sip客户端

SIP客户端For android:LinePhone3.2.7(最新版本),官网地址:http://www.linphone.org/

ycsb 压测hbase 把生成数据转成图片

1、先下载项目然后编程出jarhttps://github.com/HdrHistogram/HdrHistogram 2、将文件中HistogramLogProcessor 修改权限 目录中jar地址修改成当前地址 3、执行转化图片命令 4、上传图片 google浏览器我的有问题用的火狐浏览器正产 ps 代码编译 idea 好用一些 ...

u011273101的博客 337

VB摄像头监控

该VB模块可对摄像头捕获的画面进行录像等,可在创建的窗口进行控制

TMS_WebGMaps_1.9.8.0_Full_Source

TMS_WebGMaps_1.9.8.0_Full_Source 全源码 delphi cbuilder

HBase压测工具-YCSB

基本安装教程 使用注意 在8核40G内存虚拟机上使用10线程进行10亿行数据load时会需要大概12天的时间

HarryWg的博客 393

hbase性能压测

环境准备 工具准备:ycsb 数据准备:usertable 100 个region 并灌入数据。 hbase shell>n_splits=100 hbase shell> create 'usertable',{NAME=>'cf',DATA_BLOCK_ENCODING=>'DIFF',COMP...

liyanling的博客 1005

YCSB部署

YCSB简介:英文全称:Yahoo! Cloud Serving Benchmark (YCSB) 。是 Yahoo 公司的一个用来对云服务进行基础测试的工具。目标是促进新一代云数据服务系统的性能比较。为四个广泛使用的系统:Cassandra,、HBase、PNUTS和一个简单的片式MySQL执行,订了套核心基准测试和结果报告 我们以YCSB压测hbase为例: 首先下载安装包:https://

warm_day的博客 1097

CDH版HBase的压测测试 YCSB

前置介绍 YCSB 是一个基准测试系统的框架 https://github.com/brianfrankcooper/YCSB github地址 做压测的时候下载自己需要的单个压缩包就好了 Release YCSB 0.17.0 · brianfrankcooper/YCSB · GitHub 下载地址 压测步骤如下: 设置数据库系统进行测试 选择合适的数据库接口层 选择合适的工作量 选择适当的运行时参数(客户端线程数、目标吞吐量等) 加载数据

BOOLTechnology的博客 680

hbase数据读取优化_HBase实践 | HBase内核优化与吞吐能力建设

前言公司的hbase集群早先是基于社区1.2.4版本进行搭建的,在时延表现方面起初并不十分理想,受GC尖刺的影响非常严重,针对P99响应时延也只能给业务提供不高于100毫秒的SLA承诺,因此在公司层面接入hbase的业务普遍还是面向近线或者离线场景,而针对时延响应要求比较高的在线业务则没有办法提供能力支持。近期随着社区补丁的陆续合入,以及公司自研补丁的不断集成,hbase在吞吐能力表现方...

weixin_33013787的博客 297

HBase探索篇 _ 单节点多RegionServer部署与性能测试

目录导读 文章目录目录导读1. 引言2. 合理的Region数量3. Region数量优化3.1提高RegionServer的堆内存3.2 单节点多Region Server的部署4. 单RS、多RS、单RS大堆集群环境的YCSB压测数据对比5. 总结6. 参考链接 1. 引言 随着集群中总的Region数持续增长,每个节点平均管理的Region数已达550左右,某些大表的写入流量一上来,Region Server就会不堪重负,相继挂掉。 在HBase中,Region的一个列族对应一个MemStore,通常

wolf_333的博客 1594

hbase0.94.11版本和hbase1.4.9版本的benchamark区别

1.起初使用ycsb对hbase进行benchmark,分别在100%写的情况下检测写性能;在100%读的情况下检测读的性能。实验数据如下: 2.新版本的habse写性能竟然不如老版本。!!!。于是我从头梳理了hbase的原理和新老版本的配置,参数等等。认为新版本的读写性能应该比旧版本好,于是不甘心,换了hbase的自带的PE工具进行压测。 3. ...

weixin_30530939的博客 185

ycsb测试hbase

1.安装hbase和zookeeper,zookeeper不能使用hbase内置的 2.

来啊 快活啊 2507
上一篇: cvc-complex-type.2.4.c: 问题解决
下一篇: CAS--单点登录解决方案简单介绍
王树民
博客等级 码龄10年 912粉丝 560原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值