0127(http+request)

爬虫的流程:

   获取网页内容→解析网页内容→储存和分析数据

获取网页内容

HTTP请求(客户端发送求情)

组成部分:请求行,请求头,请求体

请求行中包含:

方法类型,资源路径,协议版本

请求头:

中存放的是服务端能接受的信息

请求体:

请求体中发送给服务器的数据

HTTP响应(客户端请求返回响应)

状态行→响应头→响应体

状态行:

协议版本,状态码,状态消息

状态码

状态码
1xx

信息,服务器收到请求,需要请求者继续执行操作

2xx成功,操作被成功接收并处理
3xx重定向,需要进一步的操作完成请求
4xx客户端错误,请求包含语法错误或无法完成请求
5xx服务器错误,服务器在处理请求的过程中发生了错误

今天记忆几个:

100(Continue)              继续。客户端应继续其请求

101(Switching Protocols)              切换协议(切换到更高的http协议)

200(OK)                      请求成功   一般用于get和post请求

202(created)               已创建。成功并创建了新的资源

300 (multiple choices)多种选择。请求的资源可以包括多个位置,响应可返回一个资源特征与地址的列表用于浏览器的选择。

301 (move permanently)永久移动。请求的资源被移动到新的URL里

400(bad request)          客户端请求的语法错误,服务器无法理解

401(unauthorized)          请求要求用户的身份认证

500(internal server error) 服务器内部错误,无法完成请求

501(not implemented) 服务器不支持请求的功能,无法完成请求。

响应头

包含一些客户端的信息

响应体

服务器传给客户端的内容。

Request库

基础使用

requests.get(’url‘)向括号内的地址发送get请求

一般在进行爬取是都会定义一下heades请求

so我们在进行get请求的时候,需要用到这个headers参数

定义headers ={user-agent‘:'  XXXXXXXXXXXX'}

requests.get(’url‘,headers)

通过参数访问特定的地方

定义一个参数的关键字

例如:kw={‘wd’:‘python’}

通过关键字传参,与上面规定header一致

r=requests.get(‘url’,params=)

r.text(查看相应内容,返回的是Unicode格式的数据)、

可以通过

r.encoding = ’编码’(手动切换编码)

post请求

data中用键值对存放payload中的内容

exception was thrown while activating Castle.Proxies.AirCityAPPAttendanceServiceProxy -> Castle.Proxies.EquipmentAnalysisServiceProxy -> AirCity.Services.Asset.Application.Device.DeviceManagementAppService -> AirCity.Services.Asset.Application.Device.InventoryRecordAppService.] [ operacityproduct] [e0127d8f52964b0b978a8b770a5f208c] Autofac.Core.DependencyResolutionException: An exception was thrown while activating Castle.Proxies.AirCityAPPAttendanceServiceProxy -> Castle.Proxies.EquipmentAnalysisServiceProxy -> AirCity.Services.Asset.Application.Device.DeviceManagementAppService -> AirCity.Services.Asset.Application.Device.InventoryRecordAppService. ---------- Exception Data ---------- ActivatorChain = Castle.Proxies.AirCityAPPAttendanceServiceProxy -> Castle.Proxies.EquipmentAnalysisServiceProxy -> AirCity.Services.Asset.Application.Device.DeviceManagementAppService -> AirCity.Services.Asset.Application.Device.InventoryRecordAppService ---> Autofac.Core.DependencyResolutionException: None of the constructors found with 'Autofac.Core.Activators.Reflection.DefaultConstructorFinder' on type 'AirCity.Services.Asset.Application.Device.InventoryRecordAppService' can be invoked with the available services and parameters: 2025-11-21 11:52:02.680 +08:00Cannot resolve parameter 'Volo.Abp.Domain.Repositories.IRepository`2[Suncere.AirCityProduct.Models.InventoryRecord,System.Int32] inventoryRecordRepository' of constructor 'Void .ctor(Volo.Abp.Domain.Repositories.IRepository`2[Suncere.AirCityProduct.Models.InventoryRecord,System.Int32], Microsoft.Extensions.Logging.ILogger`1[AirCity.Services.Asset.Application.Device.InventoryRecordAppService], Volo.Abp.Domain.Repositories.IRepository`1[Suncere.AirCityProduct.Models.BSDDevice])'. [ at Autofac.Core.Activators.Reflection.ReflectionActivator.GetValidConstructorBindings(ConstructorInfo[] availableConstructors, IComponentContext context, IEnumerable`1 parameters) INF at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters) at Autofac.Core.Resolving.InstanceLookup.CreateInstance(IEnumerable`1 parameters) ] [ --- End of inner exception stack trace --- operacityproduct at Autofac.Core.Resolving.InstanceLookup.CreateInstance(IEnumerable`1 parameters) ] [ at Autofac.Core.Resolving.InstanceLookup.Execute() e0127d8f52964b0b978a8b770a5f208c at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request) at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request) ] at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance) Executing ObjectResult, writing value of type ' at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters) Suncere.Abp.Application.Contracts.Dtos.RemoteServiceResponseDto`1[[System.Object, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] at Microsoft.AspNetCore.Mvc.Controllers.ServiceBasedControllerActivator.Create(ControllerContext actionContext) '. at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location where exception was thrown --- 2025-11-21 11:52:02.686 +08:00 at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) [[INF11:52:02 ] [operacityproductERR] [] e0127d8f52964b0b978a8b770a5f208c---------- Exception Data ---------- ActivatorChain = Castle.Proxies.AirCityAPPAttendanceServiceProxy -> Castle.Proxies.EquipmentAnalysisServiceProxy -> AirCity.Services.Asset.Application.Device.DeviceManagementAppService -> AirCity.Services.Asset.Application.Device.InventoryRecordAppService ] Executed action [AirCity.Services.OperaBase.Application.KaoQin.AirCityAPPAttendanceService.GetAttendanceManagement (AirOpera.Services.OperaBase.Application)11:52:02 in 29.8185INFms] Executing ObjectResult, writing value of type '2025-11-21 11:52:02.689 +08:00Suncere.Abp.Application.Contracts.Dtos.RemoteServiceResponseDto`1[[System.Object, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] [DBG'. ] [[11:52:02operacityproduct ] [INFe0127d8f52964b0b978a8b770a5f208c] ] HandleAfterActionEvent: Missing span contextExecuted action 2025-11-21 11:52:02.690 +08:00AirCity.Services.OperaBase.Application.KaoQin.AirCityAPPAttendanceService.GetAttendanceManagement (AirOpera.Services.OperaBase.Application) [INF in ] [29.8185operacityproductms] [ [11:52:02e0127d8f52964b0b978a8b770a5f208c DBG] ] HandleAfterActionEvent: Missing span contextExecuted endpoint ' AirCity.Services.OperaBase.Application.KaoQin.AirCityAPPAttendanceService.GetAttendanceManagement (AirOpera.Services.OperaBase.Application)' [2025-11-21 11:52:02.692 +08:0011:52:02 [ INFINF] ] [Executed endpoint 'operacityproduct] [AirCity.Services.OperaBase.Application.KaoQin.AirCityAPPAttendanceService.GetAttendanceManagement (AirOpera.Services.OperaBase.Application)'] Request finished in 44.1754ms 500 application/json; charset=utf-8 [11:52:02 INF] Request finished in 44.1754ms 500 application/json; charset=utf-8 2025-11-21 11:52:02.694 +08:00 [DBG] [operacityproduct] [] HandleStartEvent: Ignoring path: /api/v2/spans [11:52:02 DBG] HandleStartEvent: Ignoring path: /api/v2/spans 2025-11-21 11:52:04.335 +08:00 [DBG] [operacityproduct] [] HandleStartEvent: Ignoring path: /api/v2/spans [11:52:04 DBG] HandleStartEvent: Ignoring path: /api/v2/spans 2025-11-21 11:52:05.242 +08:00 [DBG] [operacityproduct] [] HandleExceptionEvent: Missing span context [11:52:052025-11-21 11:52:05.244 +08:00 [ DBGDBG] [] operacityproduct] [HandleExceptionEvent: Missing span context ] HandleStopEvent: Missing span context [11:52:05 DBG] HandleStopEvent: Missing span context 2025-11-21 11:52:05.256 +08:00 [DBG] [operacityproduct] [] HandleExceptionEvent: Missing span context [2025-11-21 11:52:05.257 +08:0011:52:05 [DBG ] [DBGoperacityproduct] [] ] HandleExceptionEvent: Missing span context HandleStopEvent: Missing span context [11:52:05 DBG] HandleStopEvent: Missing span context
11-22
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

...Rain...

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值