| <html> | |
| <head> | |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> | |
| <title>详解抓取网站，模拟登陆，抓取动态网页的原理和实现（Python，C#等）</title> | |
| <link rel="stylesheet" type="text/css"href="http://www.crifan.com/files/res/docbook/css/common_html.css"> | |
| <meta name="generator" content="DocBook XSL Stylesheets V1.77.1"> | |
| <meta name="description" content="本文主要介绍了抓取网站，模拟登陆，抓取动态网页相关的逻辑，原理和如何实现。 主要包括： 抓取网页，模拟登陆等背后的通用的逻辑和原理以提取songtaste网页中标题为例，详解如何抓取网站并提取网页内容以模拟登陆百度为例，详解如何模拟登陆网站以抓取网易博客帖子中的最近读者信息为例，详解如何抓取动态网页中的内容详解了在模拟登陆和抓取动态网页过程中，如何用对应的网页分析工具，如IE9的F12，Chrome的Ctrl+Shift+J，Firefox的Firebug，去分析出对应的逻辑针对抓取网站，模拟登陆，抓取动态网页，全部给出了完整的可用的，多种语言的示例代码：Python，C#等 本文提供多种格式供：在线阅读HTMLHTMLsPDFCHMTXTRTFWEBHELP下载（7zip压缩包）HTMLHTMLsPDFCHMTXTRTFWEBHELPHTML版本的在线地址为：有任何意见，建议，提交bug等，都欢迎去讨论组发帖讨论："> | |
| <meta name="keywords" content="抓取网站, 模拟登陆, 抓取动态网页, Python, C#"> | |
| </head> | |
| <body bgcolor="white" text="black"link="#0000FF" vlink="#840084"alink="#0000FF"> | |
| <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-28297199-1']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })();</script> | |
| <div lang="zh_CN" class="book"> | |
| <div class="titlepage"> | |
| <div> | |
| <div><h1 class="title"> | |
| <a name="idm3505352"></a>详解抓取网站，模拟登陆，抓取动态网页的原理和实现（Python，C#等）</h1></div> | |
| <div><p class="releaseinfo"> | |
| 版本：<span class="emphasis"><em>v1.0</em></span> | |
| </p></div> | |
| <div><h2 class="subtitle"></h2></div> | |
| <div><div class="author"><h3class="author"> | |
| <span class="firstname">Crifan</span><span class="surname">Li</span> | |
| </h3></div></div> | |
| <div><div class="abstract"> | |
| <p class="title"><b>摘要</b></p> | |
| <p>本文主要介绍了抓取网站，模拟登陆，抓取动态网页相关的逻辑，原理和如何实现。</p> | |
| <p>主要包括： | |
| </p> | |
| <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc;"> | |
| <li class="listitem">抓取网页，模拟登陆等背后的通用的逻辑和原理</li> | |
| <li class="listitem">以提取songtaste网页中标题为例，详解如何抓取网站并提取网页内容</li> | |
| <li class="listitem">以模拟登陆百度为例，详解如何模拟登陆网站</li> | |
| <li class="listitem">以抓取网易博客帖子中的最近读者信息为例，详解如何抓取动态网页中的内容</li> | |
| <li class="listitem">详解了在模拟登陆和抓取动态网页过程中，如何用对应的网页分析工具，如IE9的F12，Chrome的Ctrl+Shift+J，Firefox的Firebug，去分析出对应的逻辑</li> | |
| <li class="listitem">针对抓取网站，模拟登陆，抓取动态网页，全部给出了完整的可用的，多种语言的示例代码：Python，C#等</li> | |
| </ul></div> | |
| <p> | |
| </p> | |
| <div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><tableborder="0" summary="Tip: 本文提供多种格式供："> | |
| <tr> | |
| <td rowspan="2" align="center"valign="top" width="25"><imgalt="[提示]" src="http://www.crifan.com/files/res/docbook/images/tip.png"></td> | |
| <th align="left"> | |
| <a name="common_multi_format"></a>本文提供多种格式供：</th> | |
| </tr> | |
| <tr><td align="left" valign="top"> | |
| <div class="informaltable"><tablestyle="border-collapse: collapse;border-top: 1.5pt solid black; border-bottom: 1.5pt solid black;"> | |
| <colgroup> | |
| <col class="col1"> | |
| <col class="col2"> | |
| <col class="col3"> | |
| <col class="col4"> | |
| <col class="col5"> | |
| <col class="col6"> | |
| <col class="col7"> | |
| <col class="col8"> | |
| </colgroup> | |
| <tbody> | |
| <tr> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center">在线阅读</td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/html/web_scrape_emulate_login.html"target="_top">HTML</a></td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/htmls/index.html"target="_top">HTMLs</a></td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/pdf/web_scrape_emulate_login.pdf"target="_top">PDF</a></td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/chm/web_scrape_emulate_login.chm"target="_top">CHM</a></td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/txt/web_scrape_emulate_login.txt"target="_top">TXT</a></td> | |
| <td style="border-right: 0.5pt solid green; border-bottom: 0.5pt solid green;" align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/rtf/web_scrape_emulate_login.rtf"target="_top">RTF</a></td> | |
| <td style="border-bottom: 0.5pt solid green; " align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/webhelp/index.html"target="_top">WEBHELP</a></td> | |
| </tr> | |
| <tr> | |
| <td style="border-right: 0.5pt solid green; "align="center">下载（7zip压缩包）</td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/html/web_scrape_emulate_login.html.7z"target="_top">HTML</a></td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/htmls/index.html.7z"target="_top">HTMLs</a></td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/pdf/web_scrape_emulate_login.pdf.7z"target="_top">PDF</a></td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/chm/web_scrape_emulate_login.chm.7z"target="_top">CHM</a></td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/txt/web_scrape_emulate_login.txt.7z"target="_top">TXT</a></td> | |
| <td style="border-right: 0.5pt solid green; "align="center"><a class="link"href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/rtf/web_scrape_emulate_login.rtf.7z"target="_top">RTF</a></td> | |
| <td style="" align="center"><aclass="link" href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/webhelp/web_scrape_emulate_login.webhelp.7z"target="_top">WEBHELP</a></td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <p>HTML版本的在线地址为：</p> | |
| <p><a class="link" href="http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/html/web_scrape_emulate_login.html"target="_top">http://www.crifan.com/files/doc/docbook/web_scrape_emulate_login/release/html/web_scrape_emulate_login.html</a></p> | |
| <p>有任何意见，建议，提交bug等，都欢迎去讨论组发帖讨论：</p> | |
| <p><a class="link" href="http://www.crifan.com/bbs/categories/web_scrape_emulate_login/"target="_top">http://www.crifan.com/bbs/categories/web_scrape_emulate_login/</a></p> | |
| </td></tr> | |
| </table></div> | |
| </div></div> | |
| <div><p class="pubdate"> | |
| 2013-02-06 | |
| </p></div> | |
| <div><div class="revhistory"><tableborder="1px solid black" width="100%"summary="修订历史"> | |
| <tr><th align="left" valign="top"colspan="3"><b>修订历史</b></th></tr> | |
| <tr> | |
| <td align="left">修订 1.0</td> | |
| <td align="left">2013-02-06</td> | |
| <td align="left">crl</td> | |
| </tr> | |
| <tr><td align="left" colspan="3"> | |
| <div class="orderedlist"><ol class="orderedlist" type="1"><liclass="listitem">把之前教程的地址整理过来</li></ol></div> | |
| </td></tr> | |
| </table></div></div> | |
| <div><p class="copyright">版权 © 2013 Crifan,<a class="link" href="http://crifan.com"target="_top">http://crifan.com</a></p></div> | |
| <div><div class="legalnotice"> | |
| <a name="common_legalnotice"></a><p>本文章遵从：<aclass="link" href="http://www.crifan.com/files/doc/docbook/soft_dev_basic/release/html/soft_dev_basic.html#cc_by_nc"target="_top">署名-非商业性使用 2.5 中国大陆(CC BY-NC 2.5)</a></p> | |
| </div></div> | |
| </div> | |
| <hr> | |
| </div> | |
| <div class="toc"> | |
| <p><b>目录</b></p> | |
| <dl> | |
| <dt><span class="preface"><ahref="#preface">前言</a></span></dt> | |
| <dd><dl><dt><span class="sect1"><ahref="#purpose">1. 本文目的</a></span></dt></dl></dd> | |
| <dt><span class="chapter"><ahref="#scrape_common_logic">1. 网站抓取，模拟登陆，抓取动态网页的通用逻辑</a></span></dt> | |
| <dt><span class="chapter"><ahref="#scrape_static_website">2. 如何抓取静态网页并提取特定内容</a></span></dt> | |
| <dt><span class="chapter"><ahref="#emulate_login">3. 如何模拟登陆网站</a></span></dt> | |
| <dt><span class="chapter"><ahref="#scrape_dynamic_website">4. 如何抓取动态网页并提取特定内容</a></span></dt> | |
| <dt><span class="bibliography"><ahref="#reference">参考书目</a></span></dt> | |
| </dl> | |
| </div> | |
| <div class="preface"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="preface"></a>前言</h1></div></div></div> | |
| <div class="toc"> | |
| <p><b>目录</b></p> | |
| <dl><dt><span class="sect1"><ahref="#purpose">1. 本文目的</a></span></dt></dl> | |
| </div> | |
| <div class="sect1"> | |
| <div class="titlepage"><div><div><h2class="title" style="clear: both"> | |
| <a name="purpose"></a>1. 本文目的</h2></div></div></div> | |
| <p>本文目的在于，如何从无到有的，了解抓取网站，模拟登陆，抓取动态网页方面的逻辑和具体实现。</p> | |
| </div> | |
| </div> | |
| <div class="chapter"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="scrape_common_logic"></a>第 1 章 网站抓取，模拟登陆，抓取动态网页的通用逻辑</h1></div></div></div> | |
| <div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><tableborder="0" summary="Tip: 相关旧帖"> | |
| <tr> | |
| <td rowspan="2" align="center"valign="top" width="25"><imgalt="[提示]" src="http://www.crifan.com/files/res/docbook/images/tip.png"></td> | |
| <th align="left"> | |
| <a name="tip.scrape_common_logic"></a>相关旧帖</th> | |
| </tr> | |
| <tr><td align="left" valign="top"> | |
| <p><a class="link" href="http://www.crifan.com/how_to_use_some_language_python_csharp_to_implement_crawl_website_extract_dynamic_webpage_content_emulate_login_website"target="_top">如何用Python，C#等语言去实现抓取静态网页+抓取动态网页+模拟登陆网站</a></p> | |
| <p><a class="link" href="http://www.crifan.com/summary_webbrowser_developer_tool_ie9_f12_chrome_ctrl_shift_j_firefox_firebug"target="_top">【整理】各种浏览器中的开发人员工具Developer Tools：IE9的F12，Chrome的Ctrl+Shift+J，Firefox的Firebug</a></p> | |
| <p><a class="link" href="http://www.crifan.com/browser_developer_tool_chrome_vs_ie9"target="_top">【总结】浏览器中的开发人员工具（IE9的F12和Chrome的Ctrl+Shift+I）-网页分析的利器</a></p> | |
| <p><a class="link" href="http://www.crifan.com/summary_about_flow_process_of_fetch_webpage_simulate_login_website_and_some_notice"target="_top">【整理】关于抓取网页，分析网页内容，模拟登陆网站的逻辑/流程和注意事项</a></p> | |
| <p><a class="link" href="http://www.crifan.com/use_ie9_f12_to_analysis_the_internal_logical_process_of_login_baidu_main_page_website"target="_top">【教程】手把手教你如何利用工具(IE9的F12)去分析模拟登陆网站(百度首页)的内部逻辑过程</a></p> | |
| <p><a class="link" href="http://www.crifan.com/use_ie9_f12_to_analysis_the_root_source_of_values_of_parameter_cookie"target="_top">【教程】如何利用IE9的F12去分析网站登陆过程中的复杂的（参数，cookie等）值（的来源）</a></p> | |
| <p><a class="link" href="http://www.crifan.com/summary_url_encode_and_decode_during_http_get_post_request"target="_top">【整理】关于http(GET或POST)请求中的url地址的编码(encode)和解码(decode)</a></p> | |
| <p><a class="link" href="http://www.crifan.com/summary_explain_what_is_html_charset_and_common_value_of_gb2312_gbk_utf_8_iso8859_1"target="_top">【整理】关于HTML网页源码的字符编码（charset）格式（GB2312，GBK，UTF-8，ISO8859-1等）的解释</a></p> | |
| <p><a class="link" href="http://www.crifan.com/website_crawl_process_related_headers_cookies_post_data_handle_logic"target="_top">【整理】网页抓取，模拟登陆，抓取动态网页内容等过程中，所涉及的Headers信息，Cookie信息，POST数据的处理逻辑</a></p> | |
| <p><a class="link" href="http://www.crifan.com/for_process_html_with_many_tag_recommend_use_third_lib_while_simple_html_use_regular_expression"target="_top">【整理】关于用正则表达式处理html代码方面的建议</a></p> | |
| </td></tr> | |
| </table></div> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| </div> | |
| <div class="chapter"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="scrape_static_website"></a>第 2 章 如何抓取静态网页并提取特定内容</h1></div></div></div> | |
| <div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><tableborder="0" summary="Tip: 相关旧帖"> | |
| <tr> | |
| <td rowspan="2" align="center"valign="top" width="25"><imgalt="[提示]" src="http://www.crifan.com/files/res/docbook/images/tip.png"></td> | |
| <th align="left"> | |
| <a name="tip.scrape_static_website"></a>相关旧帖</th> | |
| </tr> | |
| <tr><td align="left" valign="top"> | |
| <p><a class="link" href="http://www.crifan.com/crawl_website_html_and_extract_info_using_python/"target="_top">【教程】抓取网并提取网页中所需要的信息 之 Python版</a></p> | |
| <p><a class="link" href="http://www.crifan.com/crawl_website_html_and_extract_info_using_csharp"target="_top">【教程】抓取网并提取网页中所需要的信息 之 C#版</a></p> | |
| </td></tr> | |
| </table></div> | |
| <p></p> | |
| </div> | |
| <div class="chapter"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="emulate_login"></a>第 3 章 如何模拟登陆网站</h1></div></div></div> | |
| <p></p> | |
| <div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><tableborder="0" summary="Tip: 相关旧帖"> | |
| <tr> | |
| <td rowspan="2" align="center"valign="top" width="25"><imgalt="[提示]" src="http://www.crifan.com/files/res/docbook/images/tip.png"></td> | |
| <th align="left"> | |
| <a name="tip.emulate_login"></a>相关旧帖</th> | |
| </tr> | |
| <tr><td align="left" valign="top"> | |
| <p><a class="link" href="http://www.crifan.com/emulate_login_website_using_python/"target="_top">【教程】模拟登陆网站 之 Python版（内含两种版本的完整的可运行的代码）</a></p> | |
| <p><a class="link" href="http://www.crifan.com/emulate_login_website_using_csharp/"target="_top">【教程】模拟登陆网站 之 C#版（内含两种版本的完整的可运行的代码）</a></p> | |
| </td></tr> | |
| </table></div> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| </div> | |
| <div class="chapter"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="scrape_dynamic_website"></a>第 4 章 如何抓取动态网页并提取特定内容</h1></div></div></div> | |
| <p></p> | |
| <div class="tip" style="margin-left: 0.5in; margin-right: 0.5in;"><tableborder="0" summary="Tip: 相关旧帖"> | |
| <tr> | |
| <td rowspan="2" align="center"valign="top" width="25"><imgalt="[提示]" src="http://www.crifan.com/files/res/docbook/images/tip.png"></td> | |
| <th align="left"> | |
| <a name="tip.scrape_dynamic_website"></a>相关旧帖</th> | |
| </tr> | |
| <tr><td align="left" valign="top"> | |
| <p><a class="link" href="http://www.crifan.com/how_to_crawl_dynamic_webpage_content"target="_top">【教程】如何抓取动态网页内容</a></p> | |
| <p><a class="link" href="http://www.crifan.com/example_to_crawl_dynamic_webpage_content_of_recent_reader_info_for_netease_blog_post"target="_top">【教程】以抓取网易博客帖子中的最近读者信息为例，手把手教你如何抓取动态网页中的内容</a></p> | |
| </td></tr> | |
| </table></div> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| <p></p> | |
| </div> | |
| <div class="bibliography"> | |
| <div class="titlepage"><div><div><h1class="title"> | |
| <a name="reference"></a>参考书目</h1></div></div></div> | |
| <div class="bibliomixed"> | |
| <a name="ref.scrape_website_emulate_login"></a><pclass="bibliomixed">[1] | |
| <span class="bibliosource"><a class="link" href="http://www.crifan.com/how_to_use_some_language_python_csharp_to_implement_crawl_website_extract_dynamic_webpage_content_emulate_login_website"target="_top">如何用Python，C#等语言去实现抓取静态网页+抓取动态网页+模拟登陆网站</a></span> | |
| <span class="author"></span> | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </body> | |
| </html> | |
扫一扫
03-20
4921
4921
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
