MySQL 派生表查询导致 Crash 的根源分析与解决方案

【信息科学与工程学】计算机科学与自动化-——第十五篇云计算 12 公有云里的“多Region + 多AZ“ 02 运营算法01 算法逐步思考推理思考(含数学方程式、对象、资源、任务、进程/协程/线程及对应的代数、约束、目标函数/传递函数/依赖函数、参数及参数的数值范围及边界条件、逐步推理的数学方程式列表、上下文切换、并发/串行/随机/乱序/顺序)算法逐步思考推理思考(含数学方程式、对象、资源、任务、进程/协程/线程及对应的代数、约束、目标函数/传递函数/依赖函数、参数及参数的数值范围及边界条件、逐步推理的数学方程式列表、上下文切换、并发/串行/随机/乱序/顺序)云厂商提高闲置资源利用率。算法复杂度为O(N),N为实例数量。 阅读详情

MySQL 派生表查询导致 Crash 的根源分析与解决方案

一、问题发现

在之前的 MySQL 8.0.32 使用中,发现使用以下带有派生表的 SQL 会导致 MySQL Crash,以下的sequence_table(2)替换为任何非常量表都行:

仅 MySQL 8.0.32 版本有影响。

EXPLAIN FORMAT=TREE 
select  
    trim(ref_15.c_ogj),
    0<>0 as c_lrcm63eani
  from 
    (select 
    0<>0 as c_ogj
  from 
   sequence_table(2) t1
  where 0<>0
  order by c_ogj asc) as ref_15;

Crash 的堆栈如下:

Thread 55 "mysqld" received signal SIGSEGV, Segmentation fault.
Item_view_ref::used_tables (this=0x7fff2418f410)
    at sql/item.h:6670
6670            table_map inner_map = ref_item()->used_tables(); ==> ref_item()为空指针,因此crash了
(gdb) bt
#0  Item_view_ref::used_tables (this=0x7fff2418f410)
    at sql/item.h:6670
#1  0x0000555558e978d1 in Item::const_item (this=0x7fff2418f410)
    at sql/item.h:2342
#2  0x0000555558ecc765 in Item_ref::print (this=0x7fff2418f410, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/item.cc:9993
#3  0x000055555903b839 in Item_func_trim::print (this=0x7fff24120d20, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/item_strfunc.cc:3244
#4  0x0000555558ea7fc5 in Item::print_item_w_name (this=0x7fff24120d20, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/item.cc:727
#5  0x00005555593f18c0 in Query_block::print_item_list (this=0x7fff24120768, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/sql_lex.cc:4041
#6  0x00005555593efb50 in Query_block::print_query_block (this=0x7fff24120768, 
    thd=0x7fff24001050, str=0x7fffc83ee7e0, 
    query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/sql_lex.cc:3614
#7  0x00005555593efa3d in Query_block::print (this=0x7fff24120768, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/sql_lex.cc:3598
#8  0x00005555593ee556 in Query_expression::print (this=0x7fff24120670, thd=0x7fff24001050, 
    str=0x7fffc83ee7e0, query_type=(QT_TO_SYSTEM_CHARSET | QT_SHOW_SELECT_NUMBER))
    at sql/sql_lex.cc:3232
#9  0x0000555559a89c2c in print_query_for_explain (query_thd=0x7fff24001050, 
    unit=0x7fff24120670, str=0x7fffc83ee7e0)
    at sql/opt_explain.cc:2288
#10 0x0000555559a10b11 in PrintQueryPlan[abi:cxx11](THD*, THD const*, Query_expression*) (
    ethd=0x7fff24001050, query_thd=0x7fff24001050, unit=0x7fff24120670)
    at sql/join_optimizer/explain_access_path.cc:1894
#11 0x0000555559a8985a in ExplainIterator (ethd=0x7fff24001050, query_thd=0x7fff24001050, 
    unit=0x7fff24120670) at sql/opt_explain.cc:2205
#12 0x0000555559a89e91 in explain_query (explain_thd=0x7fff24001050, query_thd=0x7fff24001050, 
    unit=0x7fff24120670) at sql/opt_explain.cc:2359
#13 0x000055555955cd46 in Sql_cmd_dml::execute_inner (this=0x7fff24165630, thd=0x7fff24001050)

二、问题调查过程

调查执行 SQL 的 optimize 的过程,分析发现该 SQL 的 SQL 变换情况如下:

以下的 trim(ref_15.c_ogj) 执行完 find_order_in_list 后,Item_func_trim的args[0]->m_ref_item[0] 等于0<>0 as c_lrcm63eani,而不是0<>0 as c_ogj,这是因为c_lrcm63eani和c_ogj的名字都一样,都是0<>0,在find_order_in_list函数里面由于名字一样因此内层字段被外层替代了。而后在Item::clean_up_after_removal执行的时候,Item_func_ne即c_lrcm63eani因为出现了2次,因此执行了2次decrement_ref_count(),然而在Query_block::delete_unused_merged_columns函数却把0<>0 as c_lrcm63eani的Item置为空了,因为这个时候c_lrcm63eani的item->decrement_ref_count()以后ref_count()为0因此继续执行Item::clean_up_after_removal了。

EXPLAIN FORMAT=TREE 
select  
    trim(ref_15.c_ogj),
    0<>0 as c_lrcm63eani
  from 
    (select 
    0<>0 as c_ogj
  from 
   sequence_table(2) t1
  where 0<>0
  order by c_ogj asc) as ref_15;

查看函数调用过程发现 Query_block 在 prepare 的时候执行了 delete_unused_merged_columns,

-- 函数调用过程: Query_block::prepare -> Query_block::apply_local_transforms -> Query_block::delete_unused_merged_columns

bool find_order_in_list() {
  if (select_item != not_found_item) {
      if ((*order->item)->real_item() != (*select_item)->real_item()) {
        Item::Cleanup_after_removal_context ctx(
            thd->lex->current_query_block());

        (*order->item)
            ->walk(&Item::clean_up_after_removal, walk_options,  ==>Item_func_ne执行了2次,也执行了2次decrement_ref_count()
                   pointer_cast<uchar *>(&ctx));
      }
  }
}

bool Query_block::apply_local_transforms(THD *thd, bool prune) {
  DBUG_TRACE;

  assert(first_execution);
  -- 这个函数把((Item_func *)&fields[0][0])->args[0]->m_ref_item[0]给删了
  if (derived_table_count) delete_unused_merged_columns(&m_table_nest);
}

void Query_block::delete_unused_merged_columns(
    mem_root_deque<Table_ref *> *tables) {
  DBUG_TRACE;

  for (Table_ref *tl : *tables) {
    if (tl->nested_join == nullptr) continue;
    if (tl->is_merged()) {
      for (Field_translator *transl = tl->field_translation;
           transl < tl->field_translation_end; transl++) {
        Item *const item = transl->item;
        // Decrement the ref count as its no more used in
        // select list.
        if (item->decrement_ref_count()) continue; -- 因为执行完decrement_ref_count()以后返回的m_ref_count=0因此不会跳出这个循环

        // Cleanup the item since its not referenced from
        // anywhere.
        assert(item->fixed);
        Item::Cleanup_after_removal_context ctx(this);
        item->walk(&Item::clean_up_after_removal, walk_options,
                   pointer_cast<uchar *>(&ctx));
        transl->item = nullptr; -- 这个地方把Item_view_ref引用的Item_func_ne对象置为空了,即把trim函数参数的c_lrcm63eani列删除了
      }
    }
    delete_unused_merged_columns(&tl->nested_join->m_tables);
  }
}

三、解决方案

通过上面的分析,我们可以发现问题在于多执行了一次Item::clean_up_after_removal,随后在 MySQL 最新代码尝试执行以上 SQL 发现该 BUG 已经被修复,找到相关修复代码,可以发现以下修复代码。

相关commit ID号为: 2171a1260e2cdbbd379646be8ff6413a92fd48f4

-- 相关修复代码如下:
@@ -7575,7 +7865,6 @@ bool Item::clean_up_after_removal(uchar *arg) {
 
   if (reference_count() > 1) {
     (void)decrement_ref_count();
+    ctx->stop_at(this);
   }
   return false;
 }

修改完查看一下这个函数的堆栈信息:

#0  Item::clean_up_after_removal (this=0x2, 
    arg=0x41 <error: Cannot access memory at address 0x41>)
    at sql/item.cc:9236
#1  0x0000555558fea5a8 in Item::walk (this=0x7fff2c338db8, processor=&virtual table offset 864, 
    walk=7, arg=0x7fffc83ee4b0 "") at sql/item.h:2543
#2  0x00005555596cc6f2 in find_order_in_list (thd=0x7fff2c001070, ref_item_array=..., 
    tables=0x7fff2c330b90, order=0x7fff2c32eae8, fields=0x7fff2c32fb20, is_group_field=false, 
    is_window_order=false) at sql/sql_resolver.cc:4625
#3  0x00005555596cd0ae in setup_order (thd=0x7fff2c001070, ref_item_array=..., 
    tables=0x7fff2c330b90, fields=0x7fff2c32fb20, order=0x7fff2c32eae8)
    at sql/sql_resolver.cc:4811
#4  0x00005555596bf528 in Query_block::prepare (this=0x7fff2c32fae0, thd=0x7fff2c001070, 
    insert_field_list=0x0) at sql/sql_resolver.cc:400
#5  0x00005555597d035d in Query_expression::prepare (this=0x7fff2c32f9e8, thd=0x7fff2c001070, 
    sel_result=0x7fff2c33b2a8, insert_field_list=0x0, added_options=0, removed_options=0)
    at sql/sql_union.cc:758
#6  0x0000555559590772 in Table_ref::resolve_derived (this=0x7fff2c339790, thd=0x7fff2c001070, 
    apply_semijoin=true) at sql/sql_derived.cc:451
#7  0x00005555596c2a80 in Query_block::resolve_placeholder_tables (this=0x7fff2c333f08, 
    thd=0x7fff2c001070, apply_semijoin=true)
    at sql/sql_resolver.cc:1408
#8  0x00005555596bea62 in Query_block::prepare (this=0x7fff2c333f08, thd=0x7fff2c001070, 
    insert_field_list=0x0) at sql/sql_resolver.cc:265

对于0<>0 as c_lrcm63eani这个Item_func_ne对象,执行到Item::clean_up_after_removal的时候,因为reference_count() > 1因此会执行新添加的ctx->stop_at(this),等到下一次再执行到这个Item_func_ne的clean_up_after_removal()函数的时候,就会因为ctx->is_stopped(this)而直接返回,不再执行一次decrement_ref_count(),从而避免了执行后面的transl->item = nullptr。

bool find_order_in_list() {
  if (select_item != not_found_item) {
      if ((*order->item)->real_item() != (*select_item)->real_item()) {
        Item::Cleanup_after_removal_context ctx(
            thd->lex->current_query_block());

        (*order->item)
            ->walk(&Item::clean_up_after_removal, walk_options,  -- Item_func_ne执行了2次,而只执行了一次decrement_ref_count()
                   pointer_cast<uchar *>(&ctx));
      }
  }
}

void Query_block::delete_unused_merged_columns(
    mem_root_deque<Table_ref *> *tables) {
  DBUG_TRACE;

  for (Table_ref *tl : *tables) {
    if (tl->nested_join == nullptr) continue;
    if (tl->is_merged()) {
      for (Field_translator *transl = tl->field_translation;
           transl < tl->field_translation_end; transl++) {
        Item *const item = transl->item;
        // Decrement the ref count as its no more used in
        // select list.
        if (item->decrement_ref_count()) continue; 因为执行完decrement_ref_count()以后返回的m_ref_count=1因此不会继续执行后面的置空设置

        // Cleanup the item since its not referenced from
        // anywhere.
        assert(item->fixed);
        Item::Cleanup_after_removal_context ctx(this);
        item->walk(&Item::clean_up_after_removal, walk_options,
                   pointer_cast<uchar *>(&ctx));
        transl->item = nullptr; ==>这个地方不会运行到
      }
    }
    delete_unused_merged_columns(&tl->nested_join->m_tables);
  }
}

四、问题总结

通过以上分析我们可以发现,对于复杂的 SQL 会执行复杂的 Item 变换和删除不需要的 Item,但是正是由于这样才更容易导致 Crash 的出现。分析类似这样的 Crash 问题的时候,因为涉及代码量大,代码逻辑复杂往往很难找到相关修复代码,因此需要对代码运行流程比较熟悉,同时要有相关复杂问题解决的经验才能更好的应对这类问题。

MySQL实例crash的案例详细分析 【问题描述】 我们生产环境有一组集群的多台MySQL服务器(MySQL 5.6.21),不定期的会crash,但error log中只记录了重启信息,未记录crash时的堆栈: mysqld_safe Number of processes running now: 0 mysqld_safe mysqld restarted 接下来首先排查系统日志/var/log/message文件,crash时没有其他异常信息,也不是OOM导致的。 【排查思路】 由于日志中未记录有价值的信息。为定位crash的原因,首先开启mysql core dump的功能。 下面是开启core dump的步骤 立即下载

相关推荐

笔记本电脑更换主板后出现2203:System configuration is invalid,以及2201、2202系统错误的解决

主板更换后刷新S/N

晓夏IT笔记 3519

MySQL 数据库崩溃(crash)的常见原因和解决办法

墨墨导读:本文来自墨天轮用户投稿,详述MySQL 数据库崩溃(crash)的常见原因和解决办法,希望对大家有帮助。数据技术嘉年华,十周年盛大开启,点我立即报名!大会以“自研·智能·新基建...

Enmotech的博客 4927

SpringBoot整合海康SDK实战:从设备初始化到PS流录像转MP4全流程(附避坑指南)

本文详细介绍了SpringBoot整合海康SDK的完整实战流程,涵盖设备初始化、通道管理、录像检索与下载。针对海康设备生成的PS流录像文件无法直接播放的痛点,重点解析了如何利用FFmpeg高效、无损地将其转换为通用的MP4格式,并提供了关键的避坑指南和性能优化建议。

chair的专栏 764

mysql crash_MySQL Crash 问题处理思路

MySQL Crash 问题处理思路(一)当发生Crash 时 error log 没有日志信息,这时候需要查看/var/log/messages这种情况一般是发生在MySQL进程被强制杀死,发生在下列的两种情形之一:1. kill -9命令如果是被kill-9强制杀死的,/var/log/messages会有类似如下输出内容:May11 06:10:44 localhost systemd: m...

weixin_39636707的博客 803

【初阶】手把手教你轻松上手Denodo | 连接数据库并创建基本视图 (二)

在本教程的第二部分,我们将涵盖:连接到数据库,与创建基本视图。

Denodo的博客 1106

MR2数据集实战:5步搞定多模态谣言检测模型训练(附完整代码)

本文详细介绍了如何使用MR2多模态数据集构建高精度谣言检测系统,涵盖环境配置、数据准备、多模态特征工程、模型架构设计、训练调优及部署优化等5个关键步骤。通过双线性融合技术处理文本、图像和网页三模态数据,解决特征对齐和模态冲突等工程难题,并提供完整代码实现和性能优化方案。

weixin_42657024的博客 239

MySQL 数据库崩溃(crash)的常见原因和解决办法---发表到 《数据和云》 公众号

文章目录检查mysql数据库的启动时间检查mysql服务状态检查mysql中的uptime状态使用ps检查进程启动时间检查mysql日志MySQL 数据库 crash的常见原因mysql的bug 检查mysql数据库的启动时间 linux系统中的systemd会在mysqld进程crash后自动重新启动mysql的服务,需要注意的是使用kill -9杀死mysqld进程系统会自动重新启动,而只使用kill命令则不会重新启动,因为执行kill命令,系统会发送一个SIGTERM信号给mysqld,mysql数据

姚远OracleACE的博客 1万+

什么是DHCP Snooping?如何配置DHCP Snooping?

功能描述防护类型防御伪造 DHCP 服务器、避免地址冲突、增强网络安全性使用场景企业园区网、校园网、大型局域网配合机制IP Source Guard、DAI、防止 ARP 欺骗部署难度⭐⭐(简单)“DHCP Snooping,是内网 DHCP 安全防护的第一道门神。” 🛡️。

网络技术联盟站 1267

第55期:MySQL 频繁 Crash 怎么办?

针对该问题 MySQL Crush,其实问题原因有很多。ChatDBA 能够罗列出decimal字段可能导致该问题也是因为 ChatDBA 底层的知识库足够丰富,后续我们也将持续补充数据库领域更多的专业知识。回到该问题,是由于两个字段没有获取到有效值,而且他们的数据类型为decimal类型,当进行数值转换的时候触发了 MySQL 的 bug 导致崩溃。最后配合业务方针对该情况提前做数据判断,防止异常值直接入库来避免该问题的出现。

ActionTech的博客 1058

使用linuxdeployqt打包Qt程序问题及解决方法

安装过程中提示缺少依赖multiarch-support: 下载对应依赖包。安装过程中提示缺少libpq.so.5: 下载对应依赖包。下载libmysqlclient18/5.6.25。

weixin_74239923的博客 1059

cdk&java | 绘制含有丰富信息的化学反应(适合论文、网页、PPT)

package org.example; import org.openscience.cdk.DefaultChemObjectBuilder; import org.openscience.cdk.depict.DepictionGenerator; import org.openscience.cdk.exception.CDKException; import org.openscience.cdk.interfaces.IReaction; import org.openscience.cdk.

ASKCOS博客 897

mysql table crashed_快速解决MySQL:Table xxx is marked as crashed and should be repaired五个办法...

查看MySQL错误日志看到Table xxx is marked as crashed and should be repaired解决办法如下第一种:1、首先进入mysql命令台:mysql -u root -p 回车 输入密码2、查询所有的库mysql> show databases;3、进入数据库“sp1”是库名mysql> use sp1;4、check table tab...

weixin_42499444的博客 872

AI爬虫:一文讲通AI爬虫技术和原理,及34个AI爬虫工具推荐

AI 爬虫 是一种结合了传统网络爬虫技术和人工智能(AI)技术的工具,能够更智能、高效地从网页中提取和处理数据。与传统爬虫相比,AI 爬虫能够更好地处理动态内容、复杂网页结构以及非结构化数据,同时具备数据清洗、分类、情感分析等高级功能。

数据知道的博客 1万+

mysql crash 如何定位_MySQL实例crash的案例详细分析

【问题描述】我们生产环境有一组集群的多台MySQL服务器(MySQL 5.6.21),不定期的会crash,但error log中只记录了重启信息,未记录crash时的堆栈:mysqld_safe Number of processes running now: 0mysqld_safe mysqld restarted接下来首先排查系统日志/var/log/message文件,crash时没有其...

weixin_29469195的博客 294

迈瑞BS系列全自动生化分析仪LIS协议接口手册(内含HL7和ASTM)

迈瑞BS系列全自动生化分析仪LIS协议接口手册,内含HL7和ASTM)

VB摄像头监控

该VB模块可对摄像头捕获的画面进行录像等,可在创建的窗口进行控制

mysql案例_MySQL实例crash的案例详细分析

【问题描述】我们生产环境有一组集群的多台MySQL服务器(MySQL 5.6.21),不定期的会crash,但error log中只记录了重启信息,未记录crash时的堆栈:mysqld_safe Number of processes running now: 0mysqld_safe mysqld restarted接下来首先排查系统日志/var/log/message文件,crash时没有其...

weixin_33491377的博客 373

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

mysql数据表Table is marked as crashed and should be repaired 的解决办法

网上查了一下,错误的产生原因,有网友说是频繁查询和更新XXXX表造成的索引错误,还有说法是Mysql数据库因某种原因而受到了损坏。【如:数据库服务器突发性断电,在数据表提供服务时对表的源文件进行某种操作都有可能导致,Mysql数据表被损坏而无法读取数据,总之就是因为某些不可测的问题造成表的损坏。】 回车 输入密码 3、进入数据库,“database_name”是自己的库名 4、检查出现问题的表的状态(table_name—出现错误的表名)

Roc的博客 4599

文档:亲测可用解决vnc灰屏,黑屏,鼠标大黑叉情况

主要解决vnc-viewer灰屏,黑屏,鼠标大黑叉等现象,详细见文档。

全网唯一解决Mysql数据库宕机生产事故的通用方法高级DBA真实案例解答

全网唯一解决MySql数据库宕机生产事故的通用方法高级DBA真实案例解答

nasen512的博客 4694

mysql crash_解决mysql crash问题

mysql提示Changed limits: max_open_files: 2048 max_connections: 1510 table_cache: 64解决办法在windows下安装Mysql系统日志出现max_open_files: 2048 max_connections: 510 table_cache: 64 类似错误是因为 max_connections 最大连接数和max_o...

weixin_32016817的博客 310

MySQL错误修复:Table xx is marked as crashed and last (automatic?) repair failed

问题一 Table xx is marked as crashed and last (automatic?) repair failed 有开发找到我,说数据库坏了,连不上数据库,看了下 MySQL 的错误日志,报错如下: Error: Table './db_name/table_name' is marked as crashed and last (automatic?) repair failed 修复 1、先停止 MySQL:service mysqld stop 2、进入到 MySQL 数

运维相关技术闲聊,专业背锅内容媒体 1513

mysql crash_MySQL不断 crash 是怎么回事?

我在 1G 内存的 linode 上跑了个 Django 站: http://readfree.me/数据库是 MySQL 5.5, 存储引擎是5.5开始默认的 InnoDB. 一般同时在线用户数不超过20.我每天会收到几十次 Django 发来的出错邮件, 内容都是各种数据库查询失败:......OperationalError: (2002, "Can't connect to local M...

weixin_28999945的博客 344

INSERT UPDATE导致MySQL Crash的场景解惑

技术社群的这篇文章《故障分析 | 一则 INSERT UPDATE 触发 MySQL Crash 的案例》给出一个和SQL语句执行相关的场景,问题排查过程包括了如何找到官方bug,很实用,值得学习参考。1故障现象某个业务 MySQL 实例(MySQL 5.7.20 社区版)发生 Crash,现需要对其具体原因进行分析。/mysql/mysql-5.7.20/bin/mysqld(my_print_...

bisal的专栏 200

MySQL5.7 中连续 Crash 引发 GTID 丢失

在 MySQL 5.7 版本下,因为 GTID 持久化机制的原因,当 MySQL 处于崩溃恢复阶段时,如果再次遇到 Crash,就可能会丢失最后 1 个 binlog 中的 GTID。

ActionTech的博客 1193

mysql表出现crash 修复_MySQL 表索引损坏致 Crash?来看看如何修复吧~

原标题:MySQL 表索引损坏致 Crash?来看看如何修复吧~监控到一台 MySQL 实例在早上发生过 Crash,上去看了一下,已经被 mysqld_safe 成功拉起。上去检查一下错误日志,发现错误日志如下(已对表名,库名,路径做脱敏处理):……………………………………(大量相同的报错)…………………………………………2017-08-31T11:11:04.291424Z32394522[E...

weixin_39854951的博客 377

Mysql crash

近期mysql服务经常断掉,求高手帮忙看看是什么原因造成的,附数据库错误日志,十分感谢!02:12:32 UTC - mysqld got exception 0xc0000005 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x25926354d80 Attempting backtrace. You can use the

weixin_42117584的博客 1336

mysql crash_Mysql 无故crash

一入十二月,家里事多.前几天看统计,无意间发现一个站点没有流量了,缸开始还以为是宕机了,吓我一跳,因为这个服务器上还有几个VM 跑着很重要的数据查看了半天数据才发现原来是mysql 挂掉了mysql 在这个VM上稳定了那么长时间,怎么会挂掉??查看了半天的err log,结果上面什么都没有刚开始以为是innodb 的buffer 出现了问题,仔细查了查也没有问题。。。后来无意间才发现是磁盘满了。。...

weixin_28842367的博客 239
上一篇: MySQL派生条件下推优化导致自定义变量结果错误问题分析
下一篇: GreatSQL连接数被打满的3种紧急解决方案
GreatSQL社区
博客等级 码龄5年 1210粉丝 390原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值