python爬虫：获取菜鸟网站上url

最新推荐文章于 2024-07-21 18:02:38 发布

原创最新推荐文章于 2024-07-21 18:02:38 发布 · 1.4k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#python #爬虫

python 专栏收录该内容

10 篇文章

订阅专栏

本文介绍了一段Python代码，该代码使用requests和BeautifulSoup库从Runoob网站抓取Python教程的所有链接。通过分析网页结构，代码能够识别并打印出教程页面的链接，包括基础语法、数据类型、条件语句等内容。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

致敬菜鸟网站. 在上面自学了python, html, javascript等

import requests
from bs4 import BeautifulSoup

def main():
    base_url = "http://www.runoob.com/"
    url = "http://www.runoob.com/python3/python3-string.html"
    req_obj = requests.get(url)
    bresp = BeautifulSoup(req_obj.text,'lxml')
    leftcolumn = bresp.find(id='leftcolumn')
    a = leftcolumn.find_all('a')
    for item in a:
        #print(type(item['href']))
        if ("http" in item['href']):
            print(item['href'])
        else:
            print(base_url + item['href'])
    
    return

if __name__ == '__main__':
    main()

运行结果：

http://www.runoob.com//python3/python3-tutorial.html
http://www.runoob.com/python3-install.html
http://www.runoob.com/python3-basic-syntax.html
http://www.runoob.com/python3-data-type.html
http://www.runoob.com//python3/python3-interpreter.html
http://www.runoob.com//python3/python3-comment.html
http://www.runoob.com/python3-basic-operators.html
http://www.runoob.com//python3/python3-number.html
http://www.runoob.com//python3/python3-string.html
http://www.runoob.com//python3/python3-list.html
http://www.runoob.com/python3-tuple.html
http://www.runoob.com/python3-dictionary.html
http://www.runoob.com/python3-set.html
http://www.runoob.com//python3/python3-step1.html
http://www.runoob.com//python3/python3-conditional-statements.html
http://www.runoob.com//python3/python3-loop.html
http://www.runoob.com/python3-iterator-generator.html
http://www.runoob.com//python3/python3-function.html
http://www.runoob.com//python3/python3-data-structure.html
http://www.runoob.com//python3/python3-module.html
http://www.runoob.com//python3/python3-inputoutput.html
http://www.runoob.com/python3-file-methods.html
http://www.runoob.com/python3-os-file-methods.html
http://www.runoob.com//python3/python3-errors-execptions.html
http://www.runoob.com//python3/python3-class.html
http://www.runoob.com//python3/python3-stdlib.html
http://www.runoob.com//python3/python3-examples.html
http://www.runoob.com//quiz/python-quiz.html
http://www.runoob.com//python3/python3-reg-expressions.html
http://www.runoob.com//python3/python3-cgi-programming.html
http://www.runoob.com/python-mysql-connector.html
http://www.runoob.com//python3/python3-mysql.html
http://www.runoob.com//python3/python3-socket.html
http://www.runoob.com//python3/python3-smtp.html
http://www.runoob.com//python3/python3-multithreading.html
http://www.runoob.com//python3/python3-xml-processing.html
http://www.runoob.com//python3/python3-json.html
http://www.runoob.com//python3/python3-date-time.html
http://www.runoob.com//python3/python3-built-in-functions.html
http://www.runoob.com//python3/python-mongodb.html
http://www.runoob.com//python3/python-uwsgi.html