抓取福布斯排行榜

最新推荐文章于 2022-11-15 15:48:31 发布

Coby_Wang

最新推荐文章于 2022-11-15 15:48:31 发布

阅读量722

点赞数

分类专栏： python 爬虫文章标签： python

本文链接：https://blog.youkuaiyun.com/u012209424/article/details/80614874

版权

问题：使用request库和BeautifulSoup抓取福布斯排行榜，并将其按照表格形式打印出来
网址：https://www.phb123.com/renwu/fuhao/20802.html
实现：

import requests
from bs4 import BeautifulSoup
import bs4

def getText(url):
    headers = {
  'user-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36'}
    try:
        r = requests.get(url, headers=headers, timeout = 30)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return 'Scrapy Failed!'

def parserText(ulist, html):
    soup =  BeautifulSoup(html, 'html.parser')
    for tr in soup.find('tbody').children:
        if isinstance(tr, bs4.element.Tag):
            tds = tr('td')
            for i in range(6):
                if tds[i].string is None:
                    continue
            if tds[i].string is not None:
                ulist.append(
                    [tds[0].string, tds[1].string, tds[2].string, tds[3].string, tds[4].string, tds[5].string])
    return ulist

def printText(ulist, num=