用python+selenium+firefox爬取动态网页

最新推荐文章于 2024-01-02 21:42:01 发布

原创

最新推荐文章于 2024-01-02 21:42:01 发布 · 2.2k 阅读

4 ·

CC 4.0 BY-SA版权

本文介绍了如何使用Python的Selenium库和Firefox浏览器来爬取动态生成的网页。针对JS渲染的内容，需模拟浏览器行为。文中提到安装Firefox的Xpath Checker和Firebug插件辅助定位信息，通过定位页面元素并执行滚动和翻页操作来抓取全部数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

browser = webdriver.Firefox()  # Get local session of Firefox
browser.get("www.baidu.com")  # Load page

我们需要爬取的信息在一般的静态网页中，是直接写在源代码里面的。我们可以方便使用正则表达式抓取，比如：

rr.firstInit({"data":[{"author":"袁理,翟堃","change":"首次","companyCode":"80116848","datetime":"2016-01- 28T08:13:29","infoCode":"APPH2FEzZ2tFASearchReport","insCode":"80000031","insName":"东吴证券","insStar":"3","jlrs": ["206000000","259000000","352000000","",""],"rate":"增持","secuFullCode":"002322.SZ","secuName":"理工监测","sratingName":"增持","sy":"","syls":["24.4","19.37","14.19","",""],"sys":["0.5","0.63","0.86","",""],"title":"业绩有望筑底，收购整合加速","profitYear":"2014","type":"1","newPrice":"16.17"},

但是对于js生成的动态页面，就需要我们模拟浏览器的行为加载页面，再爬取：