OceanBase TPCC测试常见报错汇总

原创已于 2025-05-15 19:20:38 修改 · 1k 阅读

15 ·

CC 4.0 BY-SA版权

文章标签：

#oceanbase #数据库 #RAG

于 2025-04-27 10:27:49 首次发布

OceanBase进阶篇专栏收录该内容

6 篇文章

订阅专栏

OceanBase TPCC测试常见报错汇总

报错1：加载测试数据时创建tablegroup失败
报错2：加载测试数据时执行超时
报错3：加载测试数据时funcs.sh函数找不到
报错4：加载数据时报错超过租户内存上限
报错5：服务器磁盘空间不足
环境配置参考

报错1：加载测试数据时创建tablegroup失败

报错信息：

CREATE TABLEGROUP tpcc_group binding true partition by hash partitions 9;
Access denied; you need (at least one of) the CREATE privilege(s) for this operation 
tablegroup not exist

修改props.ob使用高权限用户测试：

user=root@tpcc
password=******

报错2：加载测试数据时执行超时

报错信息：

Worker 006: ERROR: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again.

登录租户管理员，调大ob_query_timeout参数：

[admin@oceanbase ~]$ mysql -h127.0.0.1 -P2881 -uroot@tpcc -p****** -A

MySQL [(none)]> show variables like '%timeout%';

MySQL [(none)]> set global ob_query_timeout=36000000000;

退出重新登录生效。

报错3：加载测试数据时funcs.sh函数找不到

报错信息：

/opt/benchmarksql-5.0/run/runLoader.sh: line 8: source: funcs.sh: file not found

修改runLoader.sh替换func.sh为绝对路径：

#source funcs.sh $1  
source /opt/benchmarksql-5.0/run/funcs.sh $1

报错4：加载数据时报错超过租户内存上限

报错信息：

Worker 027: ERROR: Over tenant memory limits
Worker 067: ERROR: Over tenant memory limits
Worker 025: ERROR: No memory or reach tenant memory limit
Worker 054: ERROR: No memory or reach tenant memory limit

办法一：增加租户内存

登录SYS租户，检查unit config的内存规格：

obclient(root@sys)[oceanbase]>
SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu,
  round(b.memory_size/1024/1024/1024,2) memory_size_gb
  FROM
  OCEANBASE.DBA_OB_TENANTS a,
  OCEANBASE.DBA_OB_UNIT_CONFIGS b,
  OCEANBASE.DBA_OB_RESOURCE_POOLS c
  WHERE a.tenant_id=c.tenant_id
  AND b.unit_config_id = c.unit_config_id
  ORDER BY a.tenant_id desc;

+-------------+-----------+-----------------+-----------+---------+---------+----------------+
| tenant_name | tenant_id | unit_config     | pool_name | max_cpu | min_cpu | memory_size_gb |
+-------------+-----------+-----------------+-----------+---------+---------+----------------+
| tpcc        |      1002 | tpcc_unit       | tpcc_pool |      36 |      36 |         187.00 |
| sys         |         1 | sys_unit_config | sys_pool  |       4 |       4 |           2.00 |
+-------------+-----------+-----------------+-----------+---------+---------+----------------+
2 rows in set (0.021 sec)

如果unit内存规格很小，调大tpcc租户内存上限：

alter resource unit tpcc_unit min_cpu = 36,max_cpu = 36,memory_size = '10g';

多个副本可能使用了不同的 resource_pool，需要分别做出调整。

这里我们tpcc租户原本的内存规格就很大，感觉应该不是租户内存不足的问题。

办法二：调高转储线程数

转储前数据会大量占据内存，快速转储结束能释放被占用的内存，compaction_high_thread_score是控制并行转储线程数，可以通过调高该参数值来达到快速释放内存的目的。该参数默认值为0，表示有6个并发线程数，修改该参数无需重启OBServer即刻生效。

检查转储线程数：

obclient(root@sys)[oceanbase]> 
select zone,tenant_id,scope,name,value,edit_level,default_value 
from oceanbase.gv$ob_parameters where name='compaction_high_thread_score';
+-------+-----------+--------+------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope  | name                         | value | edit_level        | default_value |
+-------+-----------+--------+------------------------------+-------+-------------------+---------------+
| zone1 |         1 | TENANT | compaction_high_thread_score | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      1001 | TENANT | compaction_high_thread_score | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      1002 | TENANT | compaction_high_thread_score | 0     | DYNAMIC_EFFECTIVE | 0             |
+-------+-----------+--------+------------------------------+-------+-------------------+---------------+
3 rows in set (0.010 sec)

调大tpcc租户的转储线程数：

ALTER SYSTEM SET compaction_high_thread_score=12 tenant = tpcc;

由于是DYNAMIC_EFFECTIVE，无需重启OBServer节点即可生效。

办法三：调整MemStore内存占比和冻结触发阈值

OceanBase数据库中，一个租户Active MemStore内存的使用量达到freeze_trigger_percentage * memstore_limit（其中，memstore_limit = 租户内存 * memstore_limit_percentage）时，系统就会自动触发冻结（转储的前置动作），然后再调度转储。转储后，系统就会释放占用的MemStore内存。

另一个相关的参数是memstore_limit_percentage，也就是租户内存中可用于MemStore写入的比例，默认为0，表示租户使用MemStore的内存占其总可用内存的百分比由系统进行自适应调整。该参数的取值范围为[0, 100)，修改该参数无需重启OBServer即刻生效。

在应急场景下，可以调大memstore_limit_percentage的同时，调低freeze_trigger_percentage，达到临时扩容和尽快转储释放的目的。

检查memstore比例和冻结阈值参数：

select zone,tenant_id,scope,name,value,edit_level,default_value 
from oceanbase.gv$ob_parameters where name in 
('memory_limit','memstore_limit_percentage','freeze_trigger_percentage');

+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope   | name                                  | value | edit_level        | default_value |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone1 |      NULL | CLUSTER | memstore_limit_percentage             | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | memory_limit                          | 225G  | DYNAMIC_EFFECTIVE | 0M            |
| zone1 |         1 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1001 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1002 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
7 rows in set (0.024 sec)

可以看到，memstore_limit_percentage是集群级别的参数，freeze_trigger_percentage是租户级别的参数。

登录SYS租户，调大集群memstore比例参数并调小tpcc租户的冻结阈值参数：

obclient> ALTER SYSTEM SET memstore_limit_percentage = 80;
obclient> ALTER SYSTEM SET freeze_trigger_percentage = 15 tenant = tpcc;

obclient(root@sys)[oceanbase]> 
select zone,tenant_id,scope,name,value,edit_level,default_value  
from oceanbase.gv$ob_parameters where name in  
('memory_limit','memstore_limit_percentage','freeze_trigger_percentage');

+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope   | name                                  | value | edit_level        | default_value |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone1 |      NULL | CLUSTER | memstore_limit_percentage             | 80    | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | memory_limit                          | 225G  | DYNAMIC_EFFECTIVE | 0M            |
| zone1 |         1 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1001 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1002 | TENANT  | freeze_trigger_percentage             | 15    | DYNAMIC_EFFECTIVE | 20            |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
8 rows in set (0.007 sec)

办法四：调大租户写入限流参数

OB内存写入达到一定阈值OB会主动限制客户端导入速度。writing_throttling_trigger_percentage是租户级参数，该参数是调整写入速度的阈值，当MemStore已使用的内存达到该阈值时，触发写入限速。该参数值为100时，表示关闭写入限速机制。该参数的取值范围为[0, 100]，修改该参数无需重启OBServer即刻生效。

检查写入限流参数：

select zone,tenant_id,scope,name,value,edit_level,default_value 
from oceanbase.gv$ob_parameters where name='writing_throttling_trigger_percentage';

+-------+-----------+--------+---------------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope  | name                                  | value | edit_level        | default_value |
+-------+-----------+--------+---------------------------------------+-------+-------------------+---------------+
| zone1 |         1 | TENANT | writing_throttling_trigger_percentage | 60    | DYNAMIC_EFFECTIVE | 60            |
| zone1 |      1001 | TENANT | writing_throttling_trigger_percentage | 60    | DYNAMIC_EFFECTIVE | 60            |
| zone1 |      1002 | TENANT | writing_throttling_trigger_percentage | 60    | DYNAMIC_EFFECTIVE | 60            |
+-------+-----------+--------+---------------------------------------+-------+-------------------+---------------+
3 rows in set (0.043 sec)

调大tpcc租户的写入限流参数：

obclient> ALTER SYSTEM SET writing_throttling_trigger_percentage = 80 tenant = tpcc;

报错5：服务器磁盘空间不足

报错信息：

MySQL [tpcc]> CREATE INDEX bmsql_customer_idx1
    ->   on  bmsql_customer (c_w_id, c_d_id, c_last, c_first) local;
ERROR 4184 (53100): Server out of disk space

检查磁盘空间剩余和数据文件大小：

[admin@coeanbase ~]$ df -Th | grep data
Filesystem                       Type      Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lvobdata    xfs       1.0T  693G  332G  68% /obdata

[admin@coeanbase ~]$ du -sh /obdata/tpcc/data/sstable/block_file 
11G	/obdata/tpcc/data/sstable/block_file

检查数据文件大小：

select zone,tenant_id,scope,name,value,edit_level,default_value  
from oceanbase.gv$ob_parameters where name in  
('datafile_size','datafile_next','datafile_maxsize','data_disk_usage_limit_percentage');

+-------+-----------+---------+----------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope   | name                             | value | edit_level        | default_value |
+-------+-----------+---------+----------------------------------+-------+-------------------+---------------+
| zone1 |      NULL | CLUSTER | data_disk_usage_limit_percentage | 90    | DYNAMIC_EFFECTIVE | 90            |
| zone1 |      NULL | CLUSTER | datafile_maxsize                 | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | datafile_next                    | 2G    | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | datafile_size                    | 10G   | DYNAMIC_EFFECTIVE | 0M            |
+-------+-----------+---------+----------------------------------+-------+-------------------+---------------+
2 rows in set (0.010 sec)

扩容数据盘LV：

lvextend -L +1000G /dev/VolGroup/lvobdata
xfs_growfs /obdata

修改集群的数据文件大小（修改时不支持缩小，只支持扩大），无需重启OBserver即可生效。

obclient(root@sys)[oceanbase]> alter system set datafile_size='500G';

环境配置参考

# CPU

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             40
On-line CPU(s) list:                0-39
Thread(s) per core:                 2
Core(s) per socket:                 10
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              85
Model name:                         Intel(R) Xeon(R) Gold 5215 CPU @ 2.50GHz

# Memory

              total        used        free      shared  buff/cache   available
Mem:          250Gi       161Gi        54Gi       3.4Gi        33Gi        78Gi
Swap:          15Gi        56Mi        15Gi

# OS
############## Kylin Linux Version #################
Release:
Kylin Linux Advanced Server release V10 (Sword)

Kernel:
4.19.90-25.46.v2101.ky10.x86_64

Build:
Kylin Linux Advanced Server
release V10 (SP2) /(Sword)-x86_64-Build09/20210524
#################################################

# Linux Kernel

Linux 4.19.90-25.46.v2101.ky10.x86_64 #1 SMP Wed Mar 5 09:38:50 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux


# OceanBase配置项

+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone  | tenant_id | scope   | name                                  | value | edit_level        | default_value |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+
| zone1 |      NULL | CLUSTER | data_disk_usage_limit_percentage      | 90    | DYNAMIC_EFFECTIVE | 90            |
| zone1 |      NULL | CLUSTER | memstore_limit_percentage             | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | memory_limit                          | 225G  | DYNAMIC_EFFECTIVE | 0M            |
| zone1 |      NULL | CLUSTER | datafile_maxsize                      | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | datafile_next                         | 2G    | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      NULL | CLUSTER | datafile_size                         | 640G  | DYNAMIC_EFFECTIVE | 0M            |
| zone1 |         1 | TENANT  | compaction_high_thread_score          | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |         1 | TENANT  | writing_throttling_trigger_percentage | 60    | DYNAMIC_EFFECTIVE | 60            |
| zone1 |         1 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1001 | TENANT  | compaction_high_thread_score          | 0     | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      1001 | TENANT  | writing_throttling_trigger_percentage | 60    | DYNAMIC_EFFECTIVE | 60            |
| zone1 |      1001 | TENANT  | freeze_trigger_percentage             | 20    | DYNAMIC_EFFECTIVE | 20            |
| zone1 |      1002 | TENANT  | compaction_high_thread_score          | 12    | DYNAMIC_EFFECTIVE | 0             |
| zone1 |      1002 | TENANT  | writing_throttling_trigger_percentage | 80    | DYNAMIC_EFFECTIVE | 60            |
| zone1 |      1002 | TENANT  | freeze_trigger_percentage             | 15    | DYNAMIC_EFFECTIVE | 20            |
+-------+-----------+---------+---------------------------------------+-------+-------------------+---------------+