120.基本的处理方法(原理)、sp_cursor

博客介绍了SQL语句基本处理方法及sp_cursor相关内容,涵盖基本方法原理和sp_cursor的相关知识,属于信息技术领域数据库操作方面的内容。

1.基本方法

查询第X页,每页Y条记录

最基本的处理方法(原理):

如果表中有主键(记录不重复的字段也可以),可以用类似下面的方法,当然y,(x-1)*y要换成具体的数字,不能用变量:

select top y * from 表 where 主键 not in(select top (x-1)*y 主键 from 表)



如果表中无主键,可以用临时表,加标识字段解决.这里的x,y可以用变量.

select id=identity(int,1,1),*  into #tb from 表
select * from #tb where id between (x-1)*y and x*y-1

2.sp_cursor

sp_cursoropen
Defines the attributes of an API server cursor, such as its scrolling behavior and the statement used to build the result set on which the cursor operates, then populates the cursor. The statement can contain embedded parameters.

Syntax
sp_cursoropen [@cursor =] cursor_handle OUTPUT,
    [@stmt =] 'stmt'
    [, [@scrollopt =] scroll_options OUTPUT]
    [, [@ccopt =] concurrency_options OUTPUT]
    [, [@rowcount =] rowcount OUTPUT]

[
    {, [@paramdef =] N'parameter_name data_type [,...n]' }
    {, [@param1 =] value1 [,...n] }

]

Arguments
[@cursor =] cursor_handle OUTPUT 
Is the name of a declared integer variable to receive the cursor handle. cursor_handle is int, with no default. 
[@stmt =] 'stmt' 
Is a string containing a single SELECT statement or a single stored procedure call. The size of the string is limited only by available database server memory. stmt can contain parameters having the same form as a variable name, for example:

'SELECT * FROM Employees WHERE EmployeeID = @IDParameter'

Each parameter included in stmt must have a corresponding entry in both the @paramdef parameter definition list and the parameter values list.

[@scrollopt =] scroll_options OUTPUT 
Is the cursor scroll type. scroll_options is int with a default of 1 (keyset-driven), and can be a combination of these values (exactly one of the first 5 must be specified).

Value Description 
0x0001 Keyset-driven cursor. 
0x0002 Dynamic cursor. 
0x0004 Forward-only cursor. 
0x0008 Static cursor. 
0x0010 Fast forward-only cursor. 
0x1000 Parameterized query. 
0x2000 Auto fetch. 
0x4000 Auto close. 
0x8000 Check acceptable types. 
0x10000 Keyset-driven acceptable. 
0x20000 Dynamic acceptable. 
0x40000 Forward-only acceptable. 
0x80000 Static acceptable. 
0x100000 Fast forward-only acceptable. 

On return, @scrollopt contains the type of cursor actually created, which may not match what was requested. 
[@ccopt =] concurrency_options OUTPUT 
Is the cursor concurrency. concurrency_options is int, with a default of 4 (optimistic) and can be a combination of these values (exactly one of the first 4 must be specified).

Value Description 
0x0001 Read-only. 
0x0002 Scroll locks. 
0x0004 Optimistic. 
0x0008 Optimistic w/ checksum values (?). 
0x2000 Open on any SQL. 
0x4000 Update keyset in place. 
0x10000 Read-only acceptable. 
0x20000 Locks acceptable. 
0x40000 Optimistic acceptable. 

On return, @ccopt contains the type of cursor actually created, which may not match what was requested. 
[@rowcount =] rowcount OUTPUT 
Is the name of a declared integer variable to receive the number of affected rows. rowcount is int with no default value. 
[@paramdef =] N'parameter_name data_type [,...n]' 
Is one string that contains the definitions of all parameters that have been embedded in stmt. Each parameter definition consists of a parameter name and a data type. n is a placeholder indicating additional parameter definitions. Every parameter specified in stmt must be defined in @paramdef. If the Transact-SQL statement in stmt does not contain parameters, @paramdef is not needed. The default value for this parameter is NULL. 
[@param1 =] value1 
Is a value for the first parameter defined in the parameter string. The value can be a constant or a variable. There must be a parameter value supplied for every parameter included in stmt. The values are not needed if the Transact-SQL statement in stmt has no parameters. 
n 
Is a placeholder for the values of additional parameters. Values can be only constants or variables. Values cannot be more complex expressions such as functions, or expressions built using operators. 
Return Code Values
0 (success) or 1 (failure).

Result Sets
Returns the result set generated by stmt, but containing no rows.

Remarks
sp_cursoropen is a more powerful (and programmatic) way of creating server-side cursors on SQL Server.

Permissions
Execute permissions default to the public role.

Examples
A. Create a cursor for a simple SELECT statement
This simple example creates a dynamic read-only cursor for a SELECT statement with no parameters.

USE pubs

 

-- Create a dynamc read-only cursor

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, N'SELECT * FROM myTable', 2, 8193

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

B. Create a cursor for a parameterized SELECT statement
This example creates a dynamic read-only cursor for a SELECT statement with 2 parameters.

USE pubs

 

-- Create a dynamc read-only cursor

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, N'SELECT * FROM myTable WHERE col1=@P1 AND col2 LIKE @P2', 2, 8193, N'@P1 INT, @P2 VARCHAR(255)', 10, '%x%' 

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

C. Create a cursor for a stored procedure call
This example creates a dynamic read-only cursor for a stored procedure with 2 parameters (the procedure must return only one result set or the cursor creation will fail). Note that output parameters can also be used and return values retrieved via output parameters.

USE pubs

 

-- Create a dynamc read-only cursor

DECLARE @cursor INT

DECLARE @retval INT

EXEC sp_cursoropen @cursor OUTPUT, 'EXEC @P1=myProc @P2', 2, 8193, N'@P1 INT OUTPUT, @P2 INT', @retval, 1

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

sp_cursorfetch
Fetches a row or block of rows from an API server cursor.

Syntax
sp_cursorfetch [@cursor =] cursor_handle
    [, [@fetchtype =] fetchtype]
    [, [@rownum =] rownum OUTPUT]
    [, [@nrows =] nrows OUTPUT]

Arguments
[@cursor =] cursor_handle 
Is the cursor handle. cursor_handle is int, with no default. 
[@fetchtype =] fetchtype 
Is the fetch type. fetchtype is int, with a default of 2 and can have one of these values.

Value Description 
0x0001 First row. 
0x0002 Next row. 
0x0004 Previous row. 
0x0008 Last row. 
0x0010 Absolute row index. 
0x0020 Relative row index. 
0x0040 By value (???). 
0x0080 Refresh. 
0x0100 Result set info. 
0x0200 Previous noadjust (?). 
0x0400 Skip update concurrency (???). 

[@rownum =] rownum OUTPUT 
Is the row number. rownum is int, with a default of NULL. 
[@nrows =] nrows OUTPUT 
Is the number of rows to fetch. nrows is int, with a default of NULL (fetch all rows). 
Return Code Values
0 (success) or 1 (failure).

Result Sets
Returns the requested row or group of rows from the cursor.

Remarks
In addition to fetching rows, the 'result set info' fetch type can be used to retrieve information about the cursor (current row in @rownum and total number of rows in @nrows).

Permissions
Execute permissions default to the public role.

Example
USE pubs

 

-- Create a dynamc read-only cursor 

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, 'SELECT * FROM myTable', 2, 8193

 

-- Fetch the next 3 lines

EXEC sp_cursorfetch @cursor, 2, 0, 3

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

sp_cursorclose
Closes ande deallocates an API server cursor.

Syntax
sp_cursorclose [@cursor =] cursor_handle

Arguments
[@cursor =] cursor_handle 
Is a cursor handle obtained by calling sp_cursorcreate. cursor_handle is int, with no default. 
Return Code Values
0 (success) or 1 (failure).

Result Sets
None.

Permissions
Execute permissions default to the public role.

Example
USE pubs

 

-- Create a dynamic read-only cursor 

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, 'SELECT * FROM myTable', 2, 8193

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

sp_cursoroption
Sets various options for API server cursors.

Syntax
sp_cursorclose [@cursor =] cursor_handle,
    [@code =] code,
    { [@value =] value
    | [@cursorname =] cursorname }

Arguments
[@cursor =] cursor_handle 
Is a cursor handle obtained by calling sp_cursorcreate. cursor_handle is int, with no default. 
[@code =] code 
Is the option code. code is int, with no default and can be one of these values.

Value Description 
1 ??? 
2 Set cursor name. 

[@value =] value 
Is the value of the selected option. value is int, with no default. 
[@cursorname =] cursorname 
Is the name of the cursor. cursorname is sysname, with no default. 
Return Code Values
0 (success) or 1 (failure).

Result Sets
None.

Permissions
Execute permissions default to the public role.

Example
USE pubs

 

-- Create a dynamc cursor 

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, 'SELECT * FROM myTable', 2, 8193

 

-- Name the cursor

EXEC sp_cursoroption @cursor, 2, 'myCursor'

 

-- Use a cursor variable to access the cursor

DECLARE @x CURSOR

EXEC sp_describe_cursor @x out, N'global', 'myCursor'

FETCH NEXT FROM @x

 

-- Use the cursor directly by name

FETCH NEXT FROM myCursor

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

sp_cursor
Can be used to request inserts and positioned updates or deletes on API server cursors.

Syntax
sp_cursor [@cursor =] cursor_handle,
    [@optype =] optype,
    [@rownum =] rownum,
    [@table =] 'table'
    {, [@param1 =] value1 [,...n] } 

Arguments
[@cursor =] cursor_handle 
Is a cursor handle obtained by calling sp_cursorcreate. cursor_handle is int, with no default. 
[@optype =] optype 
Is a the operation to perform. optype is int, with no default and can be one of these values.

Value Description 
1 Update row (?). 
4 Insert row. 
33 Update row. 
34 Delete row. 
40 提取当前页的行

[@rownum =] rownum 
Is the number of the row to update in the fetch cache. rownum is int, with no default. 
[@table =] 'table' 
Is the name of the table to update (an empty character string seems to be ok). table is sysname, with no default. 
Return Code Values
0 (success) or 1 (failure).

Result Sets
None.

Permissions
Execute permissions default to the public role.

Example
USE pubs

 

-- Create a dynamc cursor 

DECLARE @cursor INT

EXEC sp_cursoropen @cursor OUTPUT, 'SELECT * FROM myTable', 2, 8193

 

-- Fetch the next 2 lines; this puts lines 1 and 2 in the fetch buffer

EXEC sp_cursorfetch @cursor, 2, 0, 2

 

-- Update the second line in the fetch buffer

EXEC sp_cursor @cursor, 33, 2, '', @intCol=5, @charCol='x'

 

-- Close the cursor

EXEC sp_cursorclose @cursor

 

 

import sys import pymysql import datetime if __name__ == '__main__': # 10.100.254.42 13306 semi_de_yms_db semi Semi@200 10.100.254.42 19030 semi_de_yms_db semi Semi@200 mysql_ip = sys.argv[1] mysql_port = sys.argv[2] mysql_db = sys.argv[3] mysql_user = sys.argv[4] mysql_pwd = sys.argv[5] doris_ip = sys.argv[6] doris_port = sys.argv[7] doris_db = sys.argv[8] doris_user = sys.argv[9] doris_pwd = sys.argv[10] load_time = str(datetime.datetime.now()).split(".")[0] data_list = [] db_mysql = pymysql.connect(host=mysql_ip, port=int(mysql_port), user=mysql_user, password=mysql_pwd, database=mysql_db, charset='utf8') mysql_cursor = db_mysql.cursor() db_doris = pymysql.connect(host=doris_ip, port=int(doris_port), user=doris_user, password=doris_pwd, database=doris_db, charset='utf8') doris_cursor = db_doris.cursor() wat_test_log_sql="select count(1) from wat_test_log where status=2" ods_wat_main_table_sql="select count(1) from ods_wat_main_table where dell_flag=1" metrology_test_log_sql="select count(1) from metrology_test_log where status<>0" ods_metrology_main_table_sql="select count(1) from ods_metrology_main_table where dell_flag=1" ods_defect_main_table_sql="select count(1) from ods_defect_main_table where dell_flag=1" test_event_sql="select count(1) from test_event where status<>0" cp_sp_test_event_sql="select count(1) from cp_sp_test_event where status<>0" defect_wafer_log_sql="select count(1) from defect_wafer_log where status<>0" wip_lot_history_sql="select count(1) from wip_lot_history where status<>0" wip_sub_step_history_sql="select count(1) from wip_sub_step_history where status<>0" doris_cursor.execute(defect_wafer_log_sql) defect_wafer_log_num = doris_cursor.fetchall()[0][0] data_list.append([load_time,'defect_wafer_log',defect_wafer_log_num]) doris_cursor.execute(wip_lot_history_sql) wip_lot_history_num = doris_cursor.fetchall()[0][0] data_list.append([load_time,'wip_lot_history',wip_lot_history_num]) doris_cursor.execute(wip_sub_step_history_sql) wip_sub_step_history_num = doris_cursor.fetchall()[0][0] data_list.append([load_time,'wip_sub_step_history',wip_sub_step_history_num]) doris_cursor.execute(wat_test_log_sql) wat_test_log_num = doris_cursor.fetchall()[0][0] data_list.append([load_time,'wat_test_log',wat_test_log_num]) doris_cursor.execute(ods_wat_main_table_sql) ods_wat_main_table_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'ods_wat_main_table', ods_wat_main_table_num]) doris_cursor.execute(metrology_test_log_sql) metrology_test_log_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'metrology_test_log', metrology_test_log_num]) doris_cursor.execute(ods_metrology_main_table_sql) ods_metrology_main_table_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'ods_metrology_main_table', ods_metrology_main_table_num]) doris_cursor.execute(test_event_sql) test_event_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'test_event', test_event_num]) doris_cursor.execute(ods_defect_main_table_sql) ods_defect_main_table_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'ods_defect_main_table', ods_defect_main_table_num]) doris_cursor.execute(cp_sp_test_event_sql) cp_sp_test_event_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'cp_sp_test_event', cp_sp_test_event_num]) mysql_cursor.executemany('INSERT ignore INTO semi_de_yms_db.all_data_warn_status(statistic_time, table_name, warn_count) VALUES(%s, %s, %s)',data_list) db_mysql.commit() mysql_cursor.close() db_mysql.close() doris_cursor.close() db_doris.close() 解释分析这段代码
最新发布
10-16
<think>我们正在分析一个Python脚本,该脚本连接两个数据库(MySQL和Doris),从Doris数据库的多个表中查询记录数,然后将这些记录数插入到MySQL的一个表中。 脚本流程: 1. 从命令行参数获取MySQL和Doris数据库的连接信息。 2. 获取当前时间(精确到秒,去掉毫秒部分)作为统计时间。 3. 建立到MySQL和Doris数据库的连接。 4. 定义多个SQL查询语句,用于查询Doris数据库中不同表的记录数(根据某些状态条件)。 5. 依次执行这些SQL语句,将查询结果(记录数)与当前时间、表名一起组成一个元组,添加到数据列表(data_list)中。 6. 使用executemany将data_list中的所有数据批量插入到MySQL的指定表中。 7. 提交事务,关闭所有数据库连接。 注意:插入语句使用INSERT IGNORE,避免重复记录(假设表中存在唯一索引或主键冲突时忽略)。 下面我们详细解释代码: </think>### 代码解释分析 这段代码的核心功能是**从 Doris 数据库查询多个表的异常记录数量,并将结果存储到 MySQL 数据库的监控表中**。以下是详细分析: --- #### 1. 核心逻辑流程 ```mermaid graph TD A[获取命令行参数] --> B[连接MySQL和Doris数据库] B --> C[定义多个表的计数SQL] C --> D[循环执行SQL并获取结果] D --> E[准备数据列表] E --> F[批量插入MySQL监控表] F --> G[关闭所有连接] ``` --- #### 2. 关键代码解析 ##### 2.1 数据库连接处理 ```python # 获取MySQL连接参数 mysql_ip = sys.argv[1] mysql_port = sys.argv[2] mysql_db = sys.argv[3] mysql_user = sys.argv[4] mysql_pwd = sys.argv[5] # 获取Doris连接参数 doris_ip = sys.argv[6] doris_port = sys.argv[7] doris_db = sys.argv[8] doris_user = sys.argv[9] doris_pwd = sys.argv[10] # 建立数据库连接 db_mysql = pymysql.connect(host=mysql_ip, port=int(mysql_port), ...) db_doris = pymysql.connect(host=doris_ip, port=int(doris_port), ...) ``` - **参数顺序**:脚本需要11个命令行参数 - **数据库类型**: - Doris:用于查询生产数据(OLAP) - MySQL:用于存储监控结果(OLTP) ##### 2.2 监控SQL定义 ```python wat_test_log_sql="select count(1) from wat_test_log where status=2" ods_wat_main_table_sql="select count(1) from ods_wat_main_table where dell_flag=1" metrology_test_log_sql="select count(1) from metrology_test_log where status<>0" # ... 其他SQL类似 ``` - **监控逻辑**: - 通过状态字段(status/dell_flag)识别异常记录 - 每种异常类型有独立的判断标准 - 涉及的表:测试日志、缺陷日志、WIP历史等 ##### 2.3 数据收集与存储 ```python data_list = [] load_time = str(datetime.datetime.now()).split(".")[0] # 当前时间 # 执行SQL并收集结果 doris_cursor.execute(wat_test_log_sql) wat_test_log_num = doris_cursor.fetchall()[0][0] data_list.append([load_time, 'wat_test_log', wat_test_log_num]) # 批量插入MySQL insert_sql='''INSERT IGNORE INTO semi_de_yms_db.all_data_warn_status (statistic_time, table_name, warn_count) VALUES(%s, %s, %s)''' mysql_cursor.executemany(insert_sql, data_list) ``` - **数据结构**:(统计时间, 表名, 异常计数) - **批量操作**:使用`executemany`高效插入 --- #### 3. 关键特点与注意事项 1. **监控指标定义**: | 表名 | 异常条件 | 监控内容 | |------|----------|----------| | `wat_test_log` | `status=2` | 特定状态的测试日志 | | `ods_wat_main_table` | `dell_flag=1` | 标记删除的WAT记录 | | `metrology_test_log` | `status<>0` | 非正常状态的计量日志 | | `defect_wafer_log` | `status<>0` | 异常缺陷晶圆日志 | 2. **时间处理**: - 所有监控点使用同一个时间戳 - 格式:`YYYY-MM-DD HH:MM:SS`(去除毫秒) 3. **数据库设计**: - MySQL监控表:`all_data_warn_status` - 字段设计: - `statistic_time`: 监控时间点 - `table_name`: 被监控的表名 - `warn_count`: 异常记录数 - 使用`INSERT IGNORE`避免重复记录 4. **执行方式**: ```bash python script.py \ mysql_ip mysql_port mysql_db mysql_user mysql_pwd \ doris_ip doris_port doris_db doris_user doris_pwd ``` --- #### 4. 潜在问题与改进建议 1. **SQL注入风险**: - 所有SQL都是硬编码字符串,安全但缺乏灵活性 - 建议将SQL存储在配置文件中 2. **错误处理缺失**: ```python try: doris_cursor.execute(sql) except pymysql.Error as e: print(f"查询失败: {sql}\n错误: {e}") # 可以记录为-1表示失败 return -1 ``` 3. **性能优化**: - 并行执行查询(使用线程池): ```python from concurrent.futures import ThreadPoolExecutor def execute_sql(sql): doris_cursor.execute(sql) return doris_cursor.fetchone()[0] with ThreadPoolExecutor() as executor: results = list(executor.map(execute_sql, sql_list)) ``` 4. **时间精度问题**: - 不同查询执行时间不同但共享时间戳 - 建议为每条记录单独记录执行时间 --- ###
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值