python中在使用asyncio中使用requests

你用requests爬虫效率永远比不过别人用httpx 快了整整一倍 Python 后端项目进行重构,整个后端逻辑基本都变更为采用"异步"协程的方式实现。看着满屏幕经过 async await(协程在 Python 中的实现)修饰的代码, 用上"协程",借此 给大家详细的介绍一下 什么是协程? 简单来说,协程是一种基于线程之上,但又比线程更加轻量级的存在。对于系统内核来说,协程具有不可见的特性,所以这种由程序员自己写程序来管理的轻量级线程又常被称作 "用户空间线程"。 协程比多线程好在哪呢? 1. 线程的控制权在操作系统... 阅读详情

起因

需要写个爬虫去爬一些数据,于是用python写了个,但由于众所周知的GIL锁问题,python的多线程其实效率并不高,于是准备采用协程的方法去实现,在写demo测试的时候就遇到问题了,使用await去等待requests的响应却是无效的

测试代码

    import asyncio
    import requests

    async def hello1(url):
        print('55555555555555555555')
        async with requests.get(url) as resp:
        print(url, resp.status_code)
        print('666666666666')

    async def hello(n,url):
        print("协程" + str(n) +"启动")
        await hello1(url)
        print("协程" + str(n) + "结束")

    if __name__ == "__main__":
        tasks = []
        url = 'http://localhost:8080/TBIMPSWEB/drugPrice/query.tran?REQ_MESSAGE={}'
        for i in range(0, 3):
            tasks.append(hello(i,url))
        loop = asyncio.get_event_loop()
        loop.run_until_complete(asyncio.wait(tasks))
        loop.close()

希望实现的效果是:

当hello1()执行requests.get请求时,服务器还没响应就先去执行其他的协程的代码,也就是输出55555555555部分,我给服务器上对应controlle

别再傻傻用requests了!用Python asyncio + aiohttp 5分钟搞定100个网页并发爬取 本文详细介绍了如何使用Python的asyncio和aiohttp库实现高效异步爬虫,通过协程技术将100个网页请求的耗时从50秒压缩到1秒以内。文章包含核心组件解析、生产级爬虫构建、性能优化技巧及实战案例,帮助开发者大幅提升爬虫效率。 阅读详情

相关推荐

分别用selenium,requests,模拟scrapy,asyncio4种方式采集用户的文章,并比较差异 (附python源码)

爬取全部文章(341篇),所用时间统计: 使用selenium爬取csdn(需要9分钟); 使用requests爬取csdn(需要139秒就能爬取全部文章); 模拟scrapy爬取(需要102秒); 使用asyncio异步爬取csdn(只需要17秒就能爬取全部文章)。 1. 使用selenium爬取csdn(需要9分钟) #-*- coding:utf-8 -*- import time import requests from selenium import webdriver from seleni

数据知道的博客 1万+

Python爬虫实战:使用Requests-HTML与Asyncio高效提取API接口数据

在当今数据驱动的时代,网络爬虫已成为获取互联网信息的重要工具。相比传统的网页爬取,直接从API接口获取数据更加高效、稳定。API(Application Programming Interface)是服务提供商对外暴露的数据接口,通常返回结构化的JSON或XML数据,省去了解析HTML的复杂过程。同步请求库:从urllib、requests发展到更强大的requests-html异步框架:从多线程/多进程到asyncio+aiohttp的高并发方案解析工具。

2201_76125261的博客 1568

Python-为Python的requests添加asyncawait语法支持

为Python的requests添加async/await语法支持

MPU6050的四元数解算姿态方法

使用MPU6050硬件DMP解算姿态是非常简单的,下面介绍由三轴陀螺仪和加速度计的值来使用四元数软件解算姿态的方法。 我们先来看看如何用欧拉角描述一次平面旋转(坐标变换): MPU6050的四元数解算姿态方法 设坐标系绕旋转α角后得到坐标系,在空间中有一个矢量在坐标系中的投影为,在内的投影为由于旋转绕进行,所以Z坐标未变,即有。 转换成矩阵形式表示为: 即 所以从旋转到可以写成上面仅仅是绕一根轴的旋转,如果三维空间中的欧拉角旋转要转三次: MPU6050的四元数解算姿态方法 上面得到了一个表示旋转的方

qq_41938005的博客 8392

SpringBoot+Vue企业级开源聚合支付系统!支持30+款支付平台,只需配置商户证书就能直接使用!

一款开源SAAS支付系统推荐

技术Z先生的博客 2720

AD9026芯片开发实战指南:从硬件配置到射频校准

本文为AD9026芯片开发实战指南,详细解析了从硬件设计、电源时钟配置、SPI初始化到多芯片同步与JESD204高速链路建立的完整流程。重点探讨了射频校准技术,包括初始化校准与跟踪校准,以保障5G、MIMO等现代通信系统所需的射频性能。旨在帮助开发者系统掌握这颗高性能射频收发器的应用,规避常见设计陷阱。

tomato的博客 851

周立功CAN盒二次开发实录:用QT5.1.0解析汽车CAN报文(附数据可视化技巧)

本文详细介绍了基于QT5.1.0和周立功USBCAN-E-U设备进行汽车CAN总线二次开发的实战过程。核心内容包括利用CANdb++数据库解析原始报文并转换为物理值,以及通过QCustomPlot库实现高性能实时数据可视化,为车载诊断与车联网开发提供了一套完整的工程实践方案。

sugar的博客 1057

零基础玩转华为AP 4050DN-HD:FAT模式家庭无线网络配置实战

本文为零基础用户提供华为AP 4050DN-HD从FIT模式转换为FAT模式的完整实战教程。详细介绍了固件升级、模式转换以及配置家庭无线网络(包括路由、DHCP、Wi-Fi和NAT)的每一步操作,帮助用户将企业级瘦AP改造为高性能家庭胖AP,打造稳定可靠的家庭网络环境。

weixin_29196613的博客 628

5分钟搞定PettingZoo多智能体环境搭建:从安装到运行第一个Demo

本文详细介绍了如何在5分钟内完成PettingZoo多智能体环境的搭建,从安装到运行第一个Demo。通过精准的版本控制和常见问题解决方案,帮助开发者快速上手多智能体强化学习(MARL)环境配置,并提供了实战案例和性能优化技巧。

backprop5master的博客 221

车载网络测试实操源码_使用CAPL脚本进行UDS刷写及其自动化测试

本文主要讲解如何使用CAPL脚本进行UDS刷写及其自动化测试,并结合了几个测试实例进行介绍,希望对大家有所帮助。

文章主要涉及嵌入式软件开发、测试、项目管理等,欢迎交流,共同进步。 1551

BVH2SMPL 实战笔记

BVH2SMPL 实战笔记

jacke121的专栏 840

jre9下载 jdk9

(Java SE Runtime Environment 9.0) 9.0u175 官方最新版 【x86|x64】

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

jdk-8u144-linux-x64.tar.gz.

该资源为搭建Linux系统Java环境用,资源的后缀为tar.gz格式的压缩包

上一篇: SpringMvc图片压缩上传
下一篇: dialog出现Uncaught RangeError: Maximum call stack size exceeded错误
Pcject
博客等级 码龄9年 8粉丝 18原创
评论 5
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值