TSE源码分析-url.h

本文介绍了一个用于解析和处理URL的C++类CUrl。该类能够从原始URL中提取协议类型、主机名、端口号及路径等信息,并提供了通过主机名获取IP地址的功能。此外,还实现了一系列用于验证主机名、IP地址以及URL有效性的辅助函数。

//根据一个给定的URL,组成消息体,发送给该URL指向的服务器。

//为此,定义Url类
//url.h
#ifndef _URL_H_030728_
#define _URL_H_030728_

#include<string>

//定义URL和HOST长度

const unsigned int URL_LEN =256;
const unsigned int HOST_LEN = 256;

using namespace std;

//URL类型为HTTP、FTP、或无效
enum url_scheme {
 SCHEME_HTTP,
 SCHEME_FTP,
 SCHEME_INVALID
};

//定义HTTP和FTP的默认端口

const int DEFAULT_HTTP_PORT = 80;
const int DEFAULT_FTP_PORT  = 21;

class CUrl
{
public:
 stringm_sUrl;   //Original URL
 enum url_scheme m_eScheme; //URL scheme

 string m_sHost;  //Extracted hostname
 int m_nPort;  //Port number
 string m_sPath;  // 请求 

public:
 CUrl();
 ~CUrl();

 //bool ParseUrl(string strUrl);

 //解析URL的scheme, host, port and request

 // break  an URL into scheme,host, port and request.
 // result as member variants
 bool ParseUrlEx(string strUrl);

//解析URL的scheme, host, port and request

 // break an URL into scheme, host, port andrequest.
 // result url as argvs

//解析URL的scheme, host, port and request
 void ParseUrlEx(const char *url, char *protocol,int lprotocol,
   char *host,int lhost,
   char*request, int lrequest, int *port);

 // get the ip address by hostname通过主机名得到IP
 char *GetIpByHost(const char *host);

//基本判断函数

 bool IsValidHost(const char *ip);
 bool IsForeignHost(string host);
 bool IsImageUrl(string url);
 bool IsValidIp(const char *ip);
 bool IsVisitedUrl(const char *url);
 bool IsUnReachedUrl(const char *url);
 bool IsValidHostChar(char ch);

private:
 void ParseScheme (const char *url);

};

extern pthread_mutex_t mutexMemory;

#endif

TSE(Tiny Search Engine) ======================= (Temporary) Web home: http://162.105.80.44/~yhf/Realcourse/ TSE is free utility for non-interactive download of files from the Web. It supports HTTP. According to query word or url, it retrieve results from crawled pages. It can follow links in HTML pages and create output files in Tianwang (http://e.pku.edu.cn/) format or ISAM format files. Additionally, it provies link structures which can be used to rebuild the web frame. --------------------------- Main functions in the TSE: 1) normal crawling, named SE, e.g: crawling all pages in PKU scope. and retrieve results from crawled pages according to query word or url, 2) crawling images and corresponding pages, named ImgSE. --------------------------- INSTALL: 1) execute "tar xvfz tse.XXX.gz" --------------------------- Before running the program, note Note: The program is default for normal crawling (SE). For ImgSE, you should: 1. change codes with the following requirements, 1) In "Page.cpp" file, find two same functions "CPage::IsFilterLink(string plink)" One is for ImgSE whose urls must include "tupian", "photo", "ttjstk", etc. the other is for normal crawling. For ImgSE, remember to comment the paragraph and choose right "CPage::IsFilterLink(string plink)". For SE, remember to open the paragraph and choose righ "CPage::IsFilterLink(string plink)". 2) In Http.cpp file i. find "if( iPage.m_sContentType.find("image") != string::npos )" Comment the right paragraph. 3) In Crawl.cpp file, i. "if( iPage.m_sContentType != "text/html" Comment the right paragraph. ii. find "if(file_length < 40)" Choose right one line. iii. find "iMD5.GenerateMD5( (unsigned char*)iPage.m_sContent.c_str(), iPage.m_sContent.length() )" Comment the right paragraph. iv. find "if (iUrl.IsImageUrl(strUrl))" Comment the right paragraph. 2.sh Clean; (Note not remove link4History.url, you should commnet "rm -f link4History.url" line first) secondly use "link4History.url" as a seed file. "link4History" is produced while normal crawling (SE). --------------------------- EXECUTION: execute "make clean; sh Clean;make". 1) for normal crawling and retrieving ./Tse -c tse_seed.img According to query word or url, retrieve results from crawled pages ./Tse -s 2) for ImgSE ./Tse -c tse_seed.img After moving Tianwang.raw.* data to secure place, execute ./Tse -c link4History.url --------------------------- Detail functions: 1) suporting multithreads crawling pages 2) persistent HTTP connection 3) DNS cache 4) IP block 5) filter unreachable hosts 6) parsing hyperlinks from crawled pages 7) recursively crawling pages h) Outputing Tianwang format or ISAM format files --------------------------- Files in the package Tse --- Tse execute file tse_unreachHost.list --- unreachable hosts according to PKU IP block tse_seed.pku --- PKU seeds tse_ipblock --- PKU IP block ... Directories in the package hlink,include,lib,stack,uri directories --- Parse links from a page --------------------------- Please report bugs in TSE to MAINTAINERS: YAN Hongfei * Created: YAN Hongfei, Network lab of Peking University. * Created: July 15 2003. version 0.1.1 * # Can crawl web pages with a process * Updated: Aug 20 2003. version 1.0.0 !!!! * # Can crawl web pages with multithreads * Updated: Nov 08 2003. version 1.0.1 * # more classes in the codes * Updated: Nov 16 2003. version 1.1.0 * # integrate a new version linkparser provided by XIE Han * # according to all MD5 values of pages content, * for all the pages not seen before, store a new page * Updated: Nov 21 2003. version 1.1.1 * # record all duplicate urls in terms of content MD5
<div class="product-item" data-category="novel"> <img src="https://ts1.tc.mm.bing.net/th/id/R-C.115539104325e27be2f0ee05ca4c080b?rik=xSg%2bwFfcC1%2b2QQ&riu=http%3a%2f%2fimages.bookuu.com%2fbook%2fC%2f01638%2f2634742-fm.jpg&ehk=H9B6G0WCUFgsQjHJE1iAtNWwpFnhmdAg30ByUapdJfY%3d&risl=&pid=ImgRaw&r=0" alt="产品2"> <h3>产品2:西游记</h3> <p>价格:$75</p> <button onclick="addToCart('产品2:西游记', 75, 'https://ts1.tc.mm.bing.net/th/id/R-C.115539104325e27be2f0ee05ca4c080b?rik=xSg%2bwFfcC1%2b2QQ&riu=http%3a%2f%2fimages.bookuu.com%2fbook%2fC%2f01638%2f2634742-fm.jpg&ehk=H9B6G0WCUFgsQjHJE1iAtNWwpFnhmdAg30ByUapdJfY%3d&risl=&pid=ImgRaw&r=0')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <img src="https://tse3-mm.cn.bing.net/th/id/OIP-C.3JrruqZy-PznYJxAHsoUdAHaKe?rs=1&pid=ImgDetMain" alt="产品3"> <h3>产品3:水浒传</h3> <p>价格:$100</p> <button onclick="addToCart('产品3:水浒传', 100, 'https://tse3-mm.cn.bing.net/th/id/OIP-C.3JrruqZy-PznYJxAHsoUdAHaKe?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <img src="https://ts1.tc.mm.bing.net/th/id/R-C.741a4e06ace8ed3a2a47d2a7fc53efa6?rik=OTex%2bIz3NhB3vw&riu=http%3a%2f%2fsdwypress.com%2fuploads%2fimage%2f2106%2f24%2f210624143125esxhc.jpg&ehk=bJB%2f6p6aW7wkHHbCiUTcnRf%2bNLwOktGxYScy28rgjwg%3d&risl=&pid=ImgRaw&r=0" alt="产品4"> <h3>产品4:三国演义</h3> <p>价格:$55</p> <button onclick="addToCart('产品4:三国演义', 55, 'https://ts1.tc.mm.bing.net/th/id/R-C.741a4e06ace8ed3a2a47d2a7fc53efa6?rik=OTex%2bIz3NhB3vw&riu=http%3a%2f%2fsdwypress.com%2fuploads%2fimage%2f2106%2f24%2f210624143125esxhc.jpg&ehk=bJB%2f6p6aW7wkHHbCiUTcnRf%2bNLwOktGxYScy28rgjwg%3d&risl=&pid=ImgRaw&r=0')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <img src="https://tse1-mm.cn.bing.net/th/id/OIP-C.4tBovpeIT27lEOj0FuFT5QHaKu?rs=1&pid=ImgDetMain" alt="产品5"> <h3>产品5:红岩</h3> <p>价格:$65</p> <button onclick="addToCart('产品5:红岩', 65, 'https://tse1-mm.cn.bing.net/th/id/OIP-C.4tBovpeIT27lEOj0FuFT5QHaKu?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <img src="https://tse4-mm.cn.bing.net/th/id/OIP-C.PgSOBNz_XC3Q2b_O9Nkh-wHaLS?rs=1&pid=ImgDetMain" alt="产品6"> <h3>产品6:开放中的变迁</h3> <p>价格:$50</p> <button onclick="addToCart('产品6:开放中的变迁', 50, 'https://tse4-mm.cn.bing.net/th/id/OIP-C.PgSOBNz_XC3Q2b_O9Nkh-wHaLS?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <img src="https://tse2-mm.cn.bing.net/th/id/OIP-C.1O2NDFn3NcQ5UtfKVUX9qwHaLD?rs=1&pid=ImgDetMain" alt="产品7"> <h3>产品7:中国近代史</h3> <p>价格:$55</p> <button onclick="addToCart('产品7:中国近代史', 55, 'https://tse2-mm.cn.bing.net/th/id/OIP-C.1O2NDFn3NcQ5UtfKVUX9qwHaLD?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <img src="https://img3.doubanio.com/lpic/s27258223.jpg" alt="产品8"> <h3>产品8:国史大纲</h3> <p>价格:$65</p> <button onclick="addToCart('产品8:国史大纲', 65, 'https://img3.doubanio.com/lpic/s27258223.jpg')">添加到购物车</button> </div> <div class="product-item" data-category="science"> <img src="https://so1.360tres.com/t0151ae7fb7ec6d8870.jpg" alt="产品9"> <h3>产品9:时间简史</h3> <p>价格:$65</p> <button onclick="addToCart('产品9:时间简史', 65, 'https://so1.360tres.com/t0151ae7fb7ec6d8870.jpg')">添加到购物车</button> </div> <div class="product-item" data-category="science"> <img src="https://img.alicdn.com/bao/uploaded/i1/3715517208/O1CN01yttyGV237JTlFwGhn_!!0-item_pic.jpg" alt="产品10"> <h3>产品10:从一到无穷大:科学中的事实和臆测</h3> <p>价格:$65</p> <button onclick="addToCart('产品10:从一到无穷大:科学中的事实和臆测', 65, 'https://img.alicdn.com/bao/uploaded/i1/3715517208/O1CN01yttyGV237JTlFwGhn_!!0-item_pic.jpg')">添加到购物车</button> </div>把这些代码改成<div class="product-item" data-category="novel"> <a href="333.html?id=1"> <img src="https://tse4-mm.cn.bing.net/th/id/OIP-C.E1l9GHWnkbNLjcnJpqTpPAHaKC?rs=1&pid=ImgDetMain" alt="产品1"> <h3>产品1:红楼梦</h3> <p>价格:$50</p> </a> <button onclick="addToCart('产品1:红楼梦', 50, 'https://tse4-mm.cn.bing.net/th/id/OIP-C.E1l9GHWnkbNLjcnJpqTpPAHaKC?rs=1&pid=ImgDetMain')">添加到购物车</button> </div>
05-23
<div class="product-item" data-category="novel"> <a href="555.html?id=1"> <img src="https://tse4-mm.cn.bing.net/th/id/OIP-C.E1l9GHWnkbNLjcnJpqTpPAHaKC?rs=1&pid=ImgDetMain" alt="产品1"> <h3>产品1:红楼梦</h3> <p>价格:$50</p> </a> <button onclick="addToCart('产品1:红楼梦', 50, 'https://tse4-mm.cn.bing.net/th/id/OIP-C.E1l9GHWnkbNLjcnJpqTpPAHaKC?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <a href="555.html?id=2"> <img src="https://ts1.tc.mm.bing.net/th/id/R-C.115539104325e27be2f0ee05ca4c080b?rik=xSg%2bwFfcC1%2b2QQ&riu=http%3a%2f%2fimages.bookuu.com%2fbook%2fC%2f01638%2f2634742-fm.jpg&ehk=H9B6G0WCUFgsQjHJE1iAtNWwpFnhmdAg30ByUapdJfY%3d&risl=&pid=ImgRaw&r=0" alt="产品2"> <h3>产品2:西游记</h3> <p>价格:$75</p> </a> <button onclick="addToCart('产品2:西游记', 75, 'https://ts1.tc.mm.bing.net/th/id/R-C.115539104325e27be2f0ee05ca4c080b?rik=xSg%2bwFfcC1%2b2QQ&riu=http%3a%2f%2fimages.bookuu.com%2fbook%2fC%2f01638%2f2634742-fm.jpg&ehk=H9B6G0WCUFgsQjHJE1iAtNWwpFnhmdAg30ByUapdJfY%3d&risl=&pid=ImgRaw&r=0')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <a href="555.html?id=3"> <img src="https://tse3-mm.cn.bing.net/th/id/OIP-C.3JrruqZy-PznYJxAHsoUdAHaKe?rs=1&pid=ImgDetMain" alt="产品3"> <h3>产品3:水浒传</h3> <p>价格:$100</p> </a> <button onclick="addToCart('产品3:水浒传', 100, 'https://tse3-mm.cn.bing.net/th/id/OIP-C.3JrruqZy-PznYJxAHsoUdAHaKe?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <a href="555.html?id=4"> <img src="https://ts1.tc.mm.bing.net/th/id/R-C.741a4e06ace8ed3a2a47d2a7fc53efa6?rik=OTex%2bIz3NhB3vw&riu=http%3a%2f%2fsdwypress.com%2fuploads%2fimage%2f2106%2f24%2f210624143125esxhc.jpg&ehk=bJB%2f6p6aW7wkHHbCiUTcnRf%2bNLwOktGxYScy28rgjwg%3d&risl=&pid=ImgRaw&r=0" alt="产品4"> <h3>产品4:三国演义</h3> <p>价格:$55</p> </a> <button onclick="addToCart('产品4:三国演义', 55, 'https://ts1.tc.mm.bing.net/th/id/R-C.741a4e06ace8ed3a2a47d2a7fc53efa6?rik=OTex%2bIz3NhB3vw&riu=http%3a%2f%2fsdwypress.com%2fuploads%2fimage%2f2106%2f24%2f210624143125esxhc.jpg&ehk=bJB%2f6p6aW7wkHHbCiUTcnRf%2bNLwOktGxYScy28rgjwg%3d&risl=&pid=ImgRaw&r=0')">添加到购物车</button> </div> <div class="product-item" data-category="novel"> <a href="555.html?id=5"> <img src="https://tse1-mm.cn.bing.net/th/id/OIP-C.4tBovpeIT27lEOj0FuFT5QHaKu?rs=1&pid=ImgDetMain" alt="产品5"> <h3>产品5:红岩</h3> <p>价格:$65</p> </a> <button onclick="addToCart('产品5:红岩', 65, 'https://tse1-mm.cn.bing.net/th/id/OIP-C.4tBovpeIT27lEOj0FuFT5QHaKu?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <a href="555.html?id=6"> <img src="https://tse4-mm.cn.bing.net/th/id/OIP-C.PgSOBNz_XC3Q2b_O9Nkh-wHaLS?rs=1&pid=ImgDetMain" alt="产品6"> <h3>产品6:开放中的变迁</h3> <p>价格:$50</p> </a> <button onclick="addToCart('产品6:开放中的变迁', 50, 'https://tse4-mm.cn.bing.net/th/id/OIP-C.PgSOBNz_XC3Q2b_O9Nkh-wHaLS?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <a href="555.html?id=7"> <img src="https://tse2-mm.cn.bing.net/th/id/OIP-C.1O2NDFn3NcQ5UtfKVUX9qwHaLD?rs=1&pid=ImgDetMain" alt="产品7"> <h3>产品7:中国近代史</h3> <p>价格:$55</p> </a> <button onclick="addToCart('产品7:中国近代史', 55, 'https://tse2-mm.cn.bing.net/th/id/OIP-C.1O2NDFn3NcQ5UtfKVUX9qwHaLD?rs=1&pid=ImgDetMain')">添加到购物车</button> </div> <div class="product-item" data-category="history"> <a href="555.html?id=8"> <img src="https://img3.doubanio.com/lpic/s27258223.jpg" alt="产品8"> <h3>产品8:国史大纲</h3> <p>价格:$65</p> </a> <button onclick="addToCart('产品8:国史大纲', 65, 'https://img3.doubanio.com/lpic/s27258223.jpg')">添加到购物车</button> </div> <div class="product-item" data-category="science"> <a href="555.html?id=9"> <img src="https://so1.360tres.com/t0151ae7fb7ec6d8870.jpg" alt="产品9"> <h3>产品9:时间简史</h3> <p>价格:$65</p> </a> <button onclick="addToCart('产品9:时间简史', 65, 'https://so1.360tres.com/t0151ae7fb7ec6d8870.jpg')">添加到购物车</button> </div> <div class="product-item" data-category="science"> <a href="555.html?id=10"> <img src="https://img.alicdn.com/bao/uploaded/i1/3715517208/O1CN01yttyGV237JTlFwGhn_!!0-item_pic.jpg" alt="产品10"> <h3>产品10:从一到无穷大:科学中的事实和臆测</h3> <p>价格:$65</p> </a> <button onclick="addToCart('产品10:从一到无穷大:科学中的事实和臆测', 65, 'https://img.alicdn.com/bao/uploaded/i1/3715517208/O1CN01yttyGV237JTlFwGhn_!!0-item_pic.jpg')">添加到购物车</button> </div>把这些代码改成这种格式const products = [ { id: '1', name: '产品1:红楼梦', price: 50, image: 'https://tse4-mm.cn.bing.net/th/id/OIP-C.E1l9GHWnkbNLjcnJpqTpPAHaKC?rs=1&pid=ImgDetMain', description: '这是红楼梦的详细描述...' }, { id: '2', name: '产品2:西游记', price: 75, image: 'https://ts1.tc.mm.bing.net/th/id/R-C.115539104325e27be2f0ee05ca4c080b?rik=xSg%2bwFfcC1%2b2QQ&riu=http%3a%2f%2fimages.bookuu.com%2fbook%2fC%2f01638%2f2634742-fm.jpg&ehk=H9B6G0WCUFgsQjHJE1iAtNWwpFnhmdAg30ByUapdJfY%3d&risl=&pid=ImgRaw&r=0', description: '这是西游记的详细描述...' }, // 其他商品... ];
最新发布
05-23
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值