GIL 的囚笼与自由:Python 多线程性能之谜完全解析

GIL 的囚笼与自由:Python 多线程性能之谜完全解析

引言:一个让人困惑的实验

2019年,我在优化一个数据处理系统时遇到了职业生涯中最反直觉的现象。我将单线程改为4线程处理,期待性能提升4倍,结果却发现多线程版本比单线程慢了20%。

这不是我的代码问题,而是触碰到了 Python 最核心的设计哲学——全局解释器锁(GIL)。今天,我想用通俗的语言、丰富的实验和深入的思考,带你彻底理解这个让无数开发者又爱又恨的机制。

根据 Stack Overflow 2023 年开发者调查,Python 使用率高达 49.28%,但关于 GIL 的困惑问题占 Python 相关提问的 15%。这篇文章将解答:

  • 为什么多线程可能更慢?
  • GIL 背后的设计权衡是什么?
  • 如果重新设计,有哪些可行方案?
  • 实战中如何绕过 GIL 的限制?

第一章:重现"多线程更慢"的魔幻现象

1.1 实验一:CPU 密集型任务

让我们用一个计算密集型任务来重现这个现象:

【无标题】MACD转变 ‌为了跨股票比较‌:请使用 ‌方案一(对数 MACD)‌。这是唯一能在数学上保证“走势一样(百分比涨幅一样),指标数值就一样”的方法。‌为了观察当前股票的相对强度‌:方案一得出的数值虽然很小,但其‌形态、金叉死叉位置、背离结构‌与传统 MACD 完全一致,不影响技术分析的使用习惯,只是刻度尺变了。‌可视化技巧‌:在画图时,由于对数 MACD 数值较小,建议将 Y 轴刻度自动调整,或者将结果乘以 100 或 1000 以便观察。 阅读详情

相关推荐

StarRocks慢查询排查实战:从Query Plan到Profile的保姆级调优指南

本文提供StarRocks慢查询排查的实战指南,从Query Plan解析到Profile诊断,详细讲解如何识别执行计划中的危险信号和优化关键指标。通过物化视图优化、分布式执行调整和资源管控方案,帮助DBA快速定位并解决查询性能问题,提升数据库效率。

weixin_42568801的博客 377

单细胞多组学数据整合分析实战:从原始数据到跨平台合并

本文详细介绍了单细胞多组学数据整合分析的实战流程,从原始数据处理到跨平台合并分析。通过Seurat和Harmony等工具的应用,帮助研究者有效解决批次效应问题,保留真实的生物学信号。文章包含代码示例和常见问题解决方案,特别适合需要进行单细胞合并分析的研究人员参考。

weixin_30752699的博客 89

用于多种无人机协作的群体智能算法:综合综述

随着无人机技术的快速高水平发展,将多架无人机作为自主控制的编队应用于复杂情况是一个新兴领域的代表之一。在实现自主控制的挑战中,碰撞避免、任务分配、路径规划和编队重构是关键功能。本文对各种经典的无人机群智能算法进行了系统的总结和分析。

erjia_的博客 1万+

从 SM30 到 Fiori:在 SAP BTP ABAP Environment 里创建 Business Configuration 配置维护对象的完整路径

摘要 在SAP BTP的ABAP环境中,传统SAP GUI和SM30配置方式被Business Configuration (BC)替代。BC通过Fiori应用维护配置,支持传输和权限控制。配置表需满足可维护性条件,如允许数据维护、包含技术字段等。通过ADT生成Maintenance Object和Transport Object,实现配置发布与治理。权限管理需创建IAM App和Business Catalog,确保配置可维护。BC在云端延续了可控、可追溯、可发布的治理原则。 (字数:150)

2007 年 ~ 2025 年,深耕 SAP 技术 18 年 237

大华 DSS 数字监控系统 itcBulletin SQL 注入漏洞复现

大华 DSs 数字监控系统是大华开发的一款安防视频监控系统,拥有实时监视、云台操作、录像回放、报警处理、设备管理等功能。大华 DSS存在SQL注入漏洞,攻击者 pota/services/itcBuletin 路由发送特殊构造的数据包,利用报错注入获取数据库敏感信息。攻击者除了可以利用 SQL注入漏词获取数据库中的信息例如,管理员后台密码、站点的用户人人信息)之外,甚至在高权限的情况可向服务器中写入木马,进一步获取服务器系统权限poc验证是否存在漏洞如果你想查询用户名和密码。

holyxp的博客 4001

基于VLFM的中文指令视觉语言导航系统设计与实现

文章摘要 本文介绍了一个基于VLFM(视觉语言边界地图)的中文指令视觉语言导航系统设计与实现。该系统采用模块化架构,包含三维场景渲染、URDF机器人仿真、VLFM导航引擎、中文指令解析和PyQt图形界面五大核心模块。系统支持自定义GLB场景模型和URDF机器人模型,通过transformers和langchain实现中文自然语言指令解析,利用VLFM算法进行语义导航决策。技术实现上详细介绍了GLB模型加载器开发、URDF机器人控制、VLFM导航算法集成以及PyQt界面设计。该系统在Ubuntu环境下运行,依

AI的内部世界的博客 205

Kong API Gateway 配置与深入

Kong是一个开源的、高性能的API网关,提供了一种简单易用的服务间路由、流量管理和安全防护方案,其最初起源于Nginx Inc.,但后迁移至Apache Software Foundation。Kong支持RESTful、GraphQL、TCP/UDP等协议,可与AWS Lambda、Azure Functions、Google Cloud Functions等云函数平台集成。其主要功能包括:服务发现与负载均衡、请求代理、身份认证与授权、缓存、日志记录、监控指标、灰度发布、多种插件等。

AI天才研究院 2281

人形机器人电源系统硬件开发:从原理图设计到PCB布局的完整流程

人形机器人是“移动的智能终端”,其电源系统如同“心脏+血管”:既要为大脑(主控)、肌肉(电机)、感官(传感器)提供稳定能量,又要在有限体积内应对高动态负载(如跳跃时电机瞬间大电流)。本文聚焦“硬件开发”维度,覆盖从需求分析到PCB制板的完整流程,适合硬件工程师、机器人爱好者快速掌握工程实战方法。

AI天才研究院 2173

jre9下载 jdk9

(Java SE Runtime Environment 9.0) 9.0u175 官方最新版 【x86|x64】

老版QQ原生表情gif动态

这是老版的QQ表情,不过全是动态的的,Windows照片查看器是不会动的,可以右击用浏览器打开查看,再说一遍是老版的,只是分享给宁愿用老版也要动态的人。

Statistics and Machine Learning Toolbox

Statistics and Machine Learning Toolbox(带书签),MATLAB统计学习工具

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

上一篇: 重构 CPython 的遐想:三个改变 Python 未来的关键设计
下一篇: Python 内存管理的秘密:从循环引用到垃圾回收的完整揭秘
铭渊老黄
铭渊老黄 新星创作者: Python技术领域 新星创作者: Python技术领域
博客等级 码龄7年 1万+粉丝 2218原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

铭渊老黄

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值