python3爬取斗图啦

最新推荐文章于 2021-08-13 21:12:04 发布

ccccccfl

最新推荐文章于 2021-08-13 21:12:04 发布

阅读量736

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/ccccccfl/article/details/82713616

声明：此程序是根据网易云课堂知了课堂学习后修改自己编写的，如有侵权或不妥之处请联系我删除。联系方式：微信 956788177

只用于学习！！！

本人自学python一个多月了，一直也没有什么大的进展，前几天跟着网易云课堂学了爬图片，现在想分享一下。

# -*- coding: utf-8 -*-


import random
import requests
from bs4 import BeautifulSoup
import urllib
import os


BASE_URL = 'https://www.doutula.com/photo/list/?page='
URL_LIST = []
for x in range(1, 2):
    REAL_URL = BASE_URL+str(x)
    URL_LIST.append(REAL_URL)


def get_url(url):
    my_headers = [
        "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14",
        "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)",
    ]
    header = {
        "User-Agent": random.choice(my_headers)
    }
    re = requests.get(url, headers=header)
    soup = BeautifulSoup(re.content, "lxml")
    IMG_LIST = soup.find_all('img', 'img-responsive lazy image_dta')
    for img in IMG_LIST:
        imgurl = img['data-original']
        download_img(imgurl)


def download_img(imgurl):
    filename = imgurl.split("/").pop()
    path = os.path.join('JPGGIF', filename)
    # urllib.urlretrieve(imgurl, filename=path)
    print(path)

def main():
    for url in URL_LIST:
        get_url(url)


if __name__ == '__main__':
    main()

原码如上。

斗图啦还算是比较好爬取的，我只做了随机header。

下一步打算爬取妹子图！！大福利哦！！！