urllib2 爬虫打印页面内容，部分无法显示

最新推荐文章于 2025-06-09 21:21:05 发布

原创

最新推荐文章于 2025-06-09 21:21:05 发布 · 1.2k 阅读

0 ·

CC 4.0 BY-SA版权

在尝试爬取起点小说网的内容时，使用urllib2进行网页抓取，发现打印的页面内容存在部分丢失的情况。代码执行后，一些数据未能正常显示。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

爬了一下起点小说网的内容，结果发现打印页面内容时丢失了部分内容

这个是我的代码

import re
import urllib2
user_agent ='Mozilla/5.0 (Windows NT 10.0; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0' 
headers = { 'User-Agent':user_agent}
url="https://www.qidian.com/finish"
reque=urllib2.Request(url,headers=headers)
respo=urllib2.urlopen(reque)
page=respo.read()
print page

爬下来无法显示的内容：

        </div>
    </div>
</div>
        <div class="main-content-wrap fl" data-l1="5">
            
<div class="tool-bar cf">
    <div class="view-mode" id="view-mode">
        <a id="img-text" class="iconfont act" href="//www.qidian.com/finish?action=hidden&orderId=&page=1&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B54"></a><em>|</em><a id="only-text" class="iconfont  " href="//www.qidian.com/finish?action=hidden&orderId=&page=1&style=2&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B55"></a>
    </div>
    <div class="select-wrap">
        
        <a data-id="" class=act href="//www.qidian.com/finish?action=hidden&page=1&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B62">人气排序<cite class="iconfont"></cite></a>
        <a data-id="6"  href="//www.qidian.com/finish?action=hidden&orderId=6&page=1&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B49">完本时间</a>
        <a data-id="11"  href="//www.qidian.com/finish?action=hidden&orderId=11&page=1&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B80">总收藏</a>
        <a data-id="3"  href="//www.qidian.com/finish?action=hidden&orderId=3&page=1&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=2" data-eid="qd_B52">总字数</a>
        <div id="vipClicks" class="lbf-combobox vipClicks">
            
            <a href="javascript:" class="lbf-button lbf-combobox-label " hidefocus="true">
                <span class="lbf-combobox-caption">会员点击</span>
                <span class="lbf-combobox-icon"> </span>
            </a>
        </div>
        
        <div id="recomm" class="lbf-combobox">
            <a href="javascript:" class="lbf-button lbf-combobox-label " hidefocus="true">
                <span class="lbf-combobox-caption">推荐票</span>
                <span class="lbf-combobox-icon"> </span>
            </a>
        </div>
        <em>|</em>
                                <span class="lbf-checkbox get-new-book " data-eid="qd_B53">
                                    <input id="only" type="checkbox" />
                                </span>
        <label for="only" data-eid="qd_B53">只看3个月内新书</label>
        <div class="sort-count count-info" id="sort-count">
            <div class="count-text">共<span>44433</span>本相关作品
            </div>
            <div class="yiwen" id="yiwen">
                <div class="iconfont"></div>
            </div>
            <i>|</i>
        </div>
    </div>
</div>

            
<div class="all-book-list">
    
        <div class="book-img-text">
            <ul class="all-img-list cf">
                
                <li data-rid="1">
                <div class="book-img-box">
                    <a href="//book.qidian.com/info/1003438608" data-bid="1003438608" data-eid="qd_B57" target="_blank"><img src="//qidian.qpic.cn/qdbimg/349573/1003438608/150"></a>
                </div>
                <div class="book-mid-info">
                    <h4><a href="//book.qidian.com/info/1003438608" target="_blank" data-eid="qd_B58" data-bid="1003438608">废土崛起</a></h4>
                    <p class="author">
                        <img src="//qidian.gtimg.com/qd/images/ico/user.f22d3.png"><a class="name" href="//my.qidian.com/author/2962092" data-eid="qd_B59" target="_blank">通吃道人.QD</a><em>|</em><a href="//www.qidian.com/kehuan" target="_blank" data-eid="qd_B60">科幻</a><i>·</i><a class="go-sub-type" data-typeid="9" data-subtypeid="251" href="javascript:" data-eid="qd_B61">时空穿梭</a><em>|</em><span >完本</span>
                        
                    </p>
                    <p class="intro">
                        开局一条狗，装备全靠捡。带着新手福利杀进末世将临的世界。发起狠来，核弹都朝自己头上砸！这是废土之王的崛起之路。验证群：669035131。
                    </p>
                    <p class="update"><span >441.12万字</span>
                        
                    </p>
                </div>
                </li>
                
                <li data-rid="2">
                <div class="book-img-box">
                    <a href="//book.qidian.com/info/1003543215" data-bid="1003543215" data-eid="qd_B57" target="_blank"><img src="//qidian.qpic.cn/qdbimg/349573/1003543215/150"></a>
                </div>
                <div class="book-mid-info">
                    <h4><a href="//book.qidian.com/info/1003543215" target="_blank" data-eid="qd_B58" data-bid="1003543215">恐怖广播</a></h4>
                    <p class="author">
                        <img src="//qidian.gtimg.com/qd/images/ico/user.f22d3.png"><a class="name" href="//my.qidian.com/author/3780268" data-eid="qd_B59" target="_blank">纯洁滴小龙</a><em>|</em><a href="//www.qidian.com/qihuan" target="_blank" data-eid="qd_B60">奇幻</a><i>·</i><a class="go-sub-type" data-typeid="1" data-subtypeid="202" href="javascript:" data-eid="qd_B61">黑暗幻想</a><em>|</em><span >完本</span>
                        
                    </p>
                    <p class="intro">
                        “听众朋友们，欢迎收听【恐怖广播】，希望您能够好好享受我们节目陪伴你的时光，好了，下面，请您转过头，看一看您的身后，是不是有一张人脸正在微笑着看着你？”——————起点
                    </p>
                    <p class="update"><span >339.19万字</span>
                        
                    </p>
                </div>
                </li>
                
                <li data-rid="3">
                <div class="book-img-box">
                    <a href="//book.qidian.com/info/2597043" data-bid="2597043" data-eid="qd_B57" target="_blank"><img src="//qidian.qpic.cn/qdbimg/349573/2597043/150"></a>
                </div>