python爬虫（二）requests网络爬取实战

最新推荐文章于 2025-06-01 22:50:05 发布

原创最新推荐文章于 2025-06-01 22:50:05 发布 · 998 阅读

4 ·

CC 4.0 BY-SA版权

本文介绍了如何使用Python的requests库来实现对不同类型的网络资源进行爬取，包括京东和亚马逊的商品页面信息、百度搜索结果、网络图片以及IP地址的归属地查询。

1.京东商品页面爬取

import requests
url = "https://item.jd.com/2967929.html"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
except:
    print("爬取失败")

2.亚马逊商品页面爬取

import requests
url = "https://www.amazon.cn/gp/product/B01M8L5Z3Y"
try:
    #伪装一下身份
    kv = {'user-agent':'Mozilla/5.0'}
    r = requests.get(url,headers=kv)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
except:
    print("爬取失败")

3.百度/360搜索关键词提交

import requests
url = "http://www.baidu.com/s"
#360的url：http://www.so.com/s"
try:
    keyword = 'python'
    kv = {'wd':keyword}
    #360的为kv = {'q':keyword}
    r = requests.get(url,params=kv)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
except:
    print("爬取失败")

4.网络图片爬取

import requests
import os
url = "http://img.zcool.cn/community/0117e2571b8b246ac72538120dd8a4.jpg@1280w_1l_2o_100sh.jpg"
root = "D://pics//"
path = root + url.split('/')[-1]
try :
  if not os.path.exists(root) :
      os.mkdir(root)
  if not os.path.exists(path) :
    r = requests.get(url)
    with open (path,'wb') as f:
      f.write(r.content)
      f.close ()
      print('文件保存成功')
  else:
    print('文件已存在')
  
except :
  print ('爬取失败')

5.IP地址归属地自动查询

import requests
url = "http://m.ip138.com/ip.asp?ip="
try:
  r = requests.get(url+'111.111.111.111')
  r.raise_for_status()
  r.encoding = r.apparent_encoding
  print(r.text)
except:
  print('爬取失败')