A C# Web Crawler

最新推荐文章于 2019-04-08 20:59:02 发布

转载最新推荐文章于 2019-04-08 20:59:02 发布 · 231 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/open-coder/archive/2013/01/21/2870468.html

文章标签：

#c#

本文介绍了一款使用C#编写的网络爬虫，能够下载网页及其附件，适用于在线商店等特定场景。爬虫遵循特定逻辑，从单一URL开始抓取页面并解析，递归寻找所有链接直至遍历整个网站。但遇到某些网站时，如Firefox可以正常访问，爬虫却无法获取内容，返回404错误。作者提出考虑使用外部工具如wget来替代C# Web API进行网页内容下载。

Here is a C# Web Crawler that could help you download web pages and attachments from one web-site. Such kind of crawler used vastly among some online stores during their beginning stage. They called them network spider, robot, crawler or radar. They were trained well to retrieve only some kind of information that not like Google, Baidu does. They service their own small and special group of customizers.

The logic behind the web crawler is very easy: given by one url, the crawler will download this page, and parse and find all Urls. It repeat the whole process again for all those new found urls until all Urls were visited. The full source code could be download from here.

Something need to fix

This crawler could not work well when come across some web sites. I could access those web sites from the FireFox or other web browser. But it seems this crawler could not access those web server, error ‘<404> Not Found 'always report. If any one have some well work solutions, please let me know, thanks! Maybe I could use external tools to download the web content instead of using native C# web API, may be I could use ‘wget’or something similar.