direct path read/write等待的分析

本文分析了DirectPath读写等待的问题,深入探讨了其在特定数据库环境下的表现及优化方法。

direct path read/write等待的分析

http://iamwang.blog.sohu.com/93404343.html

[@more@]

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/350519/viewspace-1042372/,如需转载,请注明出处,否则将追究法律责任。

转载于:http://blog.itpub.net/350519/viewspace-1042372/

#!/bin/bash all_options= check_disk_info=0 disk_command=0 ac_cycle=0 dc_cycle=0 reboot_cycle=0 quit_option=0 io_test=0 tmp_log_path_tmp=$(pwd) tmp_log_path="${tmp_log_path_tmp}/$(echo $0 | awk -F '/' '{print $NF}' | awk -F .sh '{print $1}')" server_ip="" server_passwd="" pdu_ip_port="" PROGRAM_NAME=$(basename $0) date_time=`date +%Y.%m.%d.%H.%M.%S` sleep_time=40 ac_cycle_flag=0 dc_cycle_flag=0 reboot_cycle_flag=0 quit_option=0 io_test=0 all_times=0 function red_message() { echo -e "\033[31m${1}\033[0m" } function green_message() { echo -e "\033[32m${1}\033[0m" } function yellow_message() { echo -e "\033[33m${1}\033[0m" } function error() { echo "$PROGRAM_NAME run with error at $date_time : $1" echo "$PROGRAM_NAME run with error at $date_time : $1" >> ${reboot_flag}_all_log exit 1 } function usage_and_exit() { echo -e "*-------------------------------------------------------------------* Filename : $0 Version : 1.0 Description : run cycle test like: nvme reset without IO Generate 10g file and Save MD5 before cycle test check MD5 when cycle test complete $(yellow_message "!!! Note: If your disk has partitions, please format it before testing") $(yellow_message "!!! Note: Please confirm the tool of DERA nvme tool under ${tmp_log_path_tmp}/") *-------------------------------------------------------------------* $PROGRAM_NAME support option is : [-c|-n number|-l number|--help] -c <Check disk info (speed,width,smart,state)> -n [num] <Custom hotplug times,a space is needed after the follow number!> -l [num] <Nvme disk count,excluding the os disk!> --help <Display this content> For Example: If you want to test hotplug 10 cycle,run ./$PROGRAM_NAME -n 100 If you want to check disk info,run ./$PROGRAM_NAME -c -n [num] If you need add nvme disk count,run ./$PROGRAM_NAME -c -n [num] -l [num] Suggestions are as follows: $(yellow_message "When testing a cycle, it is best to add parameters -c,-n [num],l [num]") run as: ./$PROGRAM_NAME -n [cycle_num] -c -l [disk_count] " exit 0 } function deal_command_log_files_when_error() { if [ -e ${tmp_log_path}/genfile_basic ] ;then while read files do if [ $2 = "cp" ];then cp ${tmp_log_path}/$files $1 2> /dev/null else mv ${tmp_log_path}/$files $1 2> /dev/null fi done < ${tmp_log_path}/genfile_basic fi if [ -e ${tmp_log_path}/genfile_after ];then while read files do mv ${tmp_log_path}/$files $1 2> /dev/null done < ${tmp_log_path}/genfile_after fi } function collect_log_delect_startup_before_exit() { name_flag=$1 dir_name=${name_flag}_${date_time} mkdir -p "${tmp_log_path}/$dir_name/before_after_info" deal_command_log_files_when_error "${tmp_log_path}/$dir_name" "mv" cd ${tmp_log_path} mv ${reboot_flag}_all_times ${reboot_flag}_all_log failed_times succeed_times "$dir_name" 2> /dev/null mv *after *basic "$dir_name/before_after_info" 2> /dev/null mv ${tmp_log_path}/$dir_name/*after ${tmp_log_path}/$dir_name/*basic "$dir_name/before_after_info" 2> /dev/null mv nohup.out smart_info/ pci_info/ fio_log/ dmesg_log/ state_info/ md5_check/ nor_log/ telemetry_log/ vendor_smart/ -t "$dir_name" 2> /dev/null cd - > /dev/null pkill -9 fio } function generate_file() { file_result=`cat "${tmp_log_path}/genfile_$1" 2> /dev/null` #preserve dmesg log mkdir -p ${tmp_log_path}/dmesg_log #检测硬盘的信息 if [ $check_disk_info -eq 1 ] ;then fdisk -l 2> /dev/null | sed -n 's/^\(磁盘\|Disk\) \([^::]*\)\(:\|:\)\([^,]*\),.*$/\2 \4/p' | awk '{if (( $2>=32 && ($3=="GB" || $3=="GiB"))||($3=="TB"||$3=="TiB")) print }'| sort > "${tmp_log_path}/disk_$1" #test -z "$file_result" && echo "disk_$1" >> "${tmp_log_path}/genfile_$1" mkdir -p ${tmp_log_path}/smart_info mkdir -p ${tmp_log_path}/pci_info #HDD/SSD get smartctl info for sdy in $(lsblk |grep -vw ${osdisk}|grep disk |awk '{print $1}' |grep sd) do smartctl -a /dev/${sdy} > ${tmp_log_path}/smart_info/${sdy}_smart_${all_times} done ###nvme SSD get info log for nvmey in $(lsblk |grep -vw ${osdisk}|grep disk |awk '{print $1}' |grep nvme) do sn=$(nvme list | awk '/'${nvmey}'/ {print $3}') result_cycle=$((${all_times} % 5)) #[ "$result_cycle" -eq 0 ] && ${tmp_log_path}/nvme dera nor-log /dev/$nvmey -f ${tmp_log_path}/nor_log/${sn}_${all_times}_nor.log nvme smart-log /dev/${nvmey} > ${tmp_log_path}/smart_info/"${sn}"_smart_${all_times} nvme intel smart-log-add /dev/${nvmey} > ${tmp_log_path}/smart_info/"${sn}"_smart_add_${all_times} if `echo "$sn" | grep -q '^D8'` ;then #if_d8_ssd="yes" mkdir -p ${tmp_log_path}/vendor_smart ${tmp_log_path}/nvme drd8 get-vendor-smart /dev/$nvmey > ${tmp_log_path}/vendor_smart/"${sn}"_vendor_smart_${all_times} #[ "$result_cycle" -eq 0 ] && ${tmp_log_path}/nvme telemetry-log /dev/$nvmey --output-file ${tmp_log_path}/telemetry_log/${sn}_${all_times}_telemetry.log -d 4 [ "$result_cycle" -eq 0 ] && mkdir -p ${tmp_log_path}/telemetry_log/${sn}_${all_times}_log && ${tmp_log_path}/nvme drd8 get-tlog /dev/${nvmey} -s ${tmp_log_path}/telemetry_log/${sn}_${all_times}_log -e -c else #if_d8_ssd="no" mkdir -p ${tmp_log_path}/state_info #mkdir -p ${tmp_log_path}/nor_log [ "$result_cycle" -eq 0 ] && ${tmp_log_path}/nvme dera nor-log /dev/$nvmey -f ${tmp_log_path}/nor_log/${sn}_${all_times}_nor.log ${tmp_log_path}/nvme dera state /dev/${nvmey} > ${tmp_log_path}/state_info/"${sn}"_state_${all_times} cat ${tmp_log_path}/state_info/"${sn}"_state_${all_times} | grep -E "device_status" > ${tmp_log_path}/state_${sn}_$1 test -z "$file_result" && echo "state_${sn}_$1" >> "${tmp_log_path}/genfile_$1" fi cat ${tmp_log_path}/smart_info/"${sn}"_smart_${all_times} | grep -E "critical_warning|media_errors|num_err_log_entries" > ${tmp_log_path}/smart_${sn}_$1 cat ${tmp_log_path}/smart_info/"${sn}"_smart_add_${all_times} | grep -E "crc_error_count" > ${tmp_log_path}/smart_add_${sn}_$1 test -z "$file_result" && echo "smart_${sn}_$1" >> "${tmp_log_path}/genfile_$1" test -z "$file_result" && echo "smart_add_${sn}_$1" >> "${tmp_log_path}/genfile_$1" done lspci -vvd 1d78: > ${tmp_log_path}/pci_info/"pci_vvd_${all_times}" lspci -vvd 1d78: | grep -E "Non-V|LnkSta:" > ${tmp_log_path}/pci_vvd_$1 #nvme list|awk 'NR >2' |awk '{print $1," ",$2," ",$3," ",$4," ",$5," ",$9,$10," ",$11," "$12,$13," ",$16}' > ${tmp_log_path}/nvmelist_$1 nvme list|awk 'NR >2' > ${tmp_log_path}/nvmelist_$1 #test -z "$file_result" && echo "nvmelist_$1" >> "${tmp_log_path}/genfile_$1" test -z "$file_result" && echo "pci_vvd_$1" >> "${tmp_log_path}/genfile_$1" fi if [ "${io_test}" -eq 1 ] && [ "$all_times" -lt "$custom_times" ];then mkdir -p ${tmp_log_path}/fio_log #nvme FIO with 4K QD=128 for nvmex in $(lsblk -l | grep -v ${osdisk} |awk '{print $1}' |grep nvme | grep "p2$") do sn=$(nvme list | awk '/'${nvmex//p2/}'/ {print $3}') nohup fio --filename=/dev/${nvmex} --ioengine=libaio --randrepeat=0 --norandommap --thread --direct=1 --group_reporting --name=mytest --runtime=1m --time_based --numjobs=1 --iodepth=128 --rw=randrw --rwmixread=50 --bs=4k >> ${tmp_log_path}/fio_log/${sn}_randrw_50_50_${all_times}.log & done wait && wait fi if [ "$all_times" -eq "$custom_times" ];then echo "wait md5 check ..." check_md5_after ###nvme SSD get info log for nvmey in $(lsblk |grep -vw ${osdisk}|grep disk |awk '{print $1}' |grep nvme) do sn=$(nvme list | awk '/'${nvmey}'/ {print $2}') #result_cycle=$((${all_times} % 20)) ${tmp_log_path}/nvme dera nor-log /dev/$nvmey -f ${tmp_log_path}/nor_log/${sn}_${all_times}_nor.log done fi dmesg -T > ${tmp_log_path}/dmesg_log/"dmesg_$all_times" } function show_basic_file_to_user() { if [ -e ${tmp_log_path}/genfile_basic ] ;then while read basic do echo -e "$basic shows that:\n" cat ${tmp_log_path}/$basic echo -e "\n\n" done < ${tmp_log_path}/genfile_basic echo "is all basic is right (Y/N):" while read input do if [ $input != 'y' -a $input != 'Y' ] && [ $input != 'n' -a $input != 'N' ]; then echo "input error Y/y for right, N/n for error,Input again:" continue else if [ $input = 'y' -o $input = 'Y' ] ;then echo "user confirmed!" break else collect_log_delect_startup_before_exit "invalid" error "user ensure basic is error" fi fi done fi } function compare_file_result() { tmp_result= if [ -e ${tmp_log_path}/genfile_after ];then while read files do filename=`echo $files|sed 's/^\(.*\)_after/\1/'` tmp=`diff ${tmp_log_path}/${filename}_basic ${tmp_log_path}/${filename}_after | wc -l` test $tmp -ne 0 && tmp_result="$tmp_result $filename" done < ${tmp_log_path}/genfile_after fi echo $tmp_result } function md5_file_gen() { [ ! -d ${tmp_log_path}/md5_check ] && mkdir -p ${tmp_log_path}/md5_check if [ "$io_test" -eq 1 ] ;then for nvmey in $(lsblk |grep -i disk |grep -vw $osdisk |awk {'print $1'} |grep nvme) do { sn=$(nvme list | awk '/'${nvmey}'/ {print $3}') nvme format -s 1 /dev/${nvmey} --force [ ! -d /mnt/${sn} ] && mkdir -p /mnt/${sn} echo yes | parted /dev/${nvmey} mklabel gpt echo I | parted /dev/${nvmey} mkpart primary 0 50% parted /dev/${nvmey} mkpart primary 50% 100% ; sleep 10 echo yes | mkfs.ext4 /dev/${nvmey}p1 && sleep 10 mount /dev/${nvmey}p1 /mnt/${sn} && sleep 1 fio --name=write_latency --filename=/mnt/${sn}/${sn}_100Gfile --ioengine=libaio --direct=1 --numjobs=1 --iodepth=128 --rw=write --group_reporting --norandommap --randrepeat=0 --size=100G --bs=1M md5sum /mnt/${sn}/${sn}_100Gfile > ${tmp_log_path}/md5_check/${sn}_md5_basic sync && sync umount /mnt/${sn} } & done else for nvmey in $(lsblk |grep -i disk |grep -vw $osdisk |awk {'print $1'} |grep nvme) do { sn=$(nvme list | awk '/'${nvmey}'/ {print $3}') nvme format -s 1 /dev/${nvmey} --force [ ! -d /mnt/${sn} ] && mkdir -p /mnt/${sn} echo yes| mkfs.ext4 /dev/${nvmey} && sleep 10 mount /dev/${nvmey} /mnt/${sn} && sleep 1 fio --name=write_latency --filename=/mnt/${sn}/${sn}_100Gfile --ioengine=libaio --direct=1 --numjobs=1 --iodepth=128 --rw=write --group_reporting --norandommap --randrepeat=0 --size=100G --bs=1M md5sum /mnt/${sn}/${sn}_100Gfile > ${tmp_log_path}/md5_check/${sn}_md5_basic sync && sync umount /mnt/${sn} } & done fi wait && wait lsblk -f echo "check parted is right (Y/N):" while read input do if [ $input != 'y' -a $input != 'Y' ] && [ $input != 'n' -a $input != 'N' ]; then echo "input Y/y for right, N/n for error,Input again:" continue else if [ $input = 'y' -o $input = 'Y' ] ;then echo "user confirmed!" break else exit fi fi done } function check_md5_after() { [ ! -d ${tmp_log_path}/md5_check ] && mkdir -p ${tmp_log_path}/md5_check if [ "$io_test" -eq 1 ] ;then for nvmey in $(lsblk |grep -i disk |grep -vw $osdisk |awk {'print $1'} |grep nvme) do { sn=$(nvme list | awk '/'${nvmey}'/ {print $3}') [ ! -d /mnt/${sn} ] && mkdir -p /mnt/${sn} mount /dev/${nvmey}p1 /mnt/${sn} && sleep 1 md5sum -c ${tmp_log_path}/md5_check/${sn}_md5_basic | tee ${tmp_log_path}/md5_check/${sn}_md5_after sync && sync umount /mnt/${sn} } & done else for nvmey in $(lsblk |grep -i disk |grep -vw $osdisk |awk {'print $1'} |grep nvme) do { sn=$(nvme list | awk '/'${nvmey}'/ {print $3}') [ ! -d /mnt/${sn} ] && mkdir -p /mnt/${sn} mount /dev/${nvmey} /mnt/${sn} && sleep 1 md5sum -c ${tmp_log_path}/md5_check/${sn}_md5_basic | tee ${tmp_log_path}/md5_check/${sn}_md5_after sync && sync umount /mnt/${sn} } & done fi wait && wait } while getopts :cin:l: opt do case $opt in c) check_disk_info=1; disk_command=1; all_options="$all_options -c" ;; n) custom_times=$OPTARG; all_options="$all_options -n $custom_times" ;; i) io_test=1 ; all_options="$all_options -i" ;; l) disk_list=1 ; disk_count=$OPTARG;all_options="$all_options -l $disk_count" ;; ?) usage_and_exit ;; esac done [ $# -lt 1 ] && usage_and_exit && exit reboot_flag="hotplug" [ ! -d ${tmp_log_path} ] && mkdir -p ${tmp_log_path} if [ -f ${tmp_log_path_tmp}/nvme ] ;then nvme_version=`${tmp_log_path_tmp}/nvme --version | awk '/^nvme version/ {print $3}' | tr -d '.'` [ "${nvme_version}" -lt "398" ] && echo "Please download new nvme tool,version need newer than 3.9.8" && exit chmod +x ${tmp_log_path_tmp}/nvme cp ${tmp_log_path_tmp}/nvme ${tmp_log_path} else yellow_message "Please download DERA nvme tool to ${tmp_log_path_tmp}/" exit fi #osdisk bootdisk=`df -h | awk '{print $1}' | grep -iE "/dev/sd" | sed 's/[0-9]//g' |sort -u` if test -z "$bootdisk" ;then #OS disk = nvmeXn1 bootdisk=`df -h | awk '{print $1}' | grep -iE "/dev/nvme" | sed 's/p[0-9]//g' |sort -u` osdisk=`echo ${bootdisk#/dev/}` else osdisk=`echo ${bootdisk#/dev/}` fi nvme_count_cur=$(nvme list | grep -vw $osdisk | grep -c nvme) if [ "${nvme_count_cur}" != "${disk_count}" ] ;then yellow_message "!!! Note: Check disk count failed,continue?(yes/no)" while read input do if [ $input != 'y' -a $input != 'Y' ] && [ $input != 'n' -a $input != 'N' ]; then echo "input Y/y for continue, N/n for stop,Input again:" continue else if [ $input = 'y' -o $input = 'Y' ] ;then echo "user confirmed, continue!" break else exit fi fi done fi parted_if=`lsblk | grep -v ${osdisk} | grep -E "n1p1|n1p2"` [ ! -z "$parted_if" ] && yellow_message "!!! Note: Please format it before testing" && exit generate_file "basic" show_basic_file_to_user echo "Generate 100g file...." md5_file_gen echo "the 1 times $reboot_flag at $date_time_before with option $all_options to generate basic file" | tee -a ${tmp_log_path}/${reboot_flag}_all_log for ((all_times=1;all_times<="$custom_times";all_times++));do date_time_before=`date +%Y.%m.%d.%H.%M.%S` echo "the $all_times times $reboot_flag at $date_time_before start" | tee -a ${tmp_log_path}/${reboot_flag}_all_log echo "Please output disk,If complete input enter..." read sleep 30 echo "Please input disk,If complete input enter..." read sleep 20 while((1));do ssd_list=`nvme list | grep -v "$osdisk" | awk '/nvme/ {print $1}'` nvme_count=`nvme list | grep -v "$osdisk"| grep -c nvme` if [ "$nvme_count" -ne "$disk_count" ] ;then echo "check nvme count fail,cur:$nvme_count;exp:$disk_count" echo "wait 10s......" sleep 10 else echo "check nvme count ok!" break fi done generate_file "after" result=$(compare_file_result) sync date_time_after=`date +%Y.%m.%d.%H.%M.%S` if [ -z "$result" ];then #没有错误的时候的处理方式,succeed次数加1 touch ${tmp_log_path}/succeed_times succeed=`cat ${tmp_log_path}/succeed_times` succeed=$(($succeed+1)) echo $succeed > ${tmp_log_path}/succeed_times echo "the $all_times times $reboot_flag at $date_time_after with no error" | tee -a ${tmp_log_path}/${reboot_flag}_all_log else #出错的时候的处理方式,failed次数加1 touch ${tmp_log_path}/failed_times failed=`cat ${tmp_log_path}/failed_times` failed=$(($failed+1)) echo $failed > ${tmp_log_path}/failed_times #处理出错的日志 echo "the $all_times times $reboot_flag at $date_time_after , an error occurred:check $result failed " | tee -a ${tmp_log_path}/${reboot_flag}_all_log dir_name=${tmp_log_path}/err_env_$date_time_after mkdir -p "$dir_name" deal_command_log_files_when_error "$dir_name" "cp" cp ${tmp_log_path}/${reboot_flag}_all_log "$dir_name" 2> /dev/null if [ $disk_command -eq 1 ] ;then mv ${tmp_log_path}/fdisk_l_after "$dir_name" 2> /dev/null cp ${tmp_log_path}/fdisk_l_basic "$dir_name" 2> /dev/null fi fi sleep 10 done echo "we has run $custom_times times $reboot_flag automaticly" >> ${tmp_log_path}/${reboot_flag}_all_log collect_log_delect_startup_before_exit "times_end" 在执行测试时,提示错误:root@bogon:/home/meitest/fuc_006# ./NVMe_FUNC_00006.sh -n 1 -l 1 !!! Note: Check disk count failed,continue?(yes/no) yes input Y/y for continue, N/n for stop,Input again: y user confirmed, continue! Generate 100g file.... Success formatting namespace:1 Success formatting namespace:1 Success formatting namespace:1 Success formatting namespace:1 mkdir: cannot create directory ‘/mnt/D87527K8015331’: No space left on device mkdir: cannot create directory ‘/mnt/D87527K8019231’: No space left on device mkdir: cannot create directory ‘/mnt/D87528K8001A31’: No space left on device mkdir: cannot create directory ‘/mnt/D87528K8001531’: No space left on device mke2fs 1.47.2 (1-Jan-2025) mke2fs 1.47.2 (1-Jan-2025) mke2fs 1.47.2 (1-Jan-2025) mke2fs 1.47.2 (1-Jan-2025) Discarding device blocks: Discarding device blocks: Discarding device blocks: Discarding device blocks: Success formatting namespace:1 Success formatting namespace:1 mkdir: cannot create directory ‘/mnt/D87527K801EU31’: No space left on device mkdir: cannot create directory ‘/mnt/D87528K8001P31’: No space left on device
09-18
考虑柔性负荷的综合能源系统低碳经济优化调度【考虑碳交易机制】(Matlab代码实现)内容概要:本文围绕“考虑柔性负荷的综合能源系统低碳经济优化调度”展开,重点研究在碳交易机制下如何实现综合能源系统的低碳化与经济性协同优化。通过构建包含风电、光伏、储能、柔性负荷等多种能源形式的系统模型,结合碳交易成本与能源调度成本,提出优化调度策略,以降低碳排放并提升系统运行经济性。文中采用Matlab进行仿真代码实现,验证了所提模型在平衡能源供需、平抑可再生能源波动、引导柔性负荷参与调度等方面的有效性,为低碳能源系统的设计与运行提供了技术支撑。; 适合人群:具备一定电力系统、能源系统背景,熟悉Matlab编程,从事能源优化、低碳调度、综合能源系统等相关领域研究的研究生、科研人员及工程技术人员。; 使用场景及目标:①研究碳交易机制对综合能源系统调度决策的影响;②实现柔性负荷在削峰填谷、促进可再生能源消纳中的作用;③掌握基于Matlab的能源系统建模与优化求解方法;④为实际综合能源项目提供低碳经济调度方案参考。; 阅读建议:建议读者结合Matlab代码深入理解模型构建与求解过程,重点关注目标函数设计、约束条件设置及碳交易成本的量化方式,可进一步扩展至多能互补、需求响应等场景进行二次开发与仿真验证。
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符  | 博主筛选后可见
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值