pre-emptive multithreading web spider铪铪铪

抢占式调度器(Preemptive Scheduler)-有锁与无锁实现 原文转自:http://www.tanjp.com/archives/137(即时修正和更新) 抢占式调度器(Preemptive Scheduler) N个业务系统生产作业加入到一个队列里面,队列中的作业被 M个线程抢先消费。也就是说,N的业务系统抢着把生产出来的作业插入到队列,同时 M个线程抢着消费该队列的作业,对队列的抢占非常激烈。可简单竞争抽象为: N*M。 ... 阅读详情
 

pre-emptive multithreading web spider


this article was contributed by sim ayers.


the win32 api supports applications that are pre-emptively multithreaded. this is a very useful and powerful feature of win32 in writing mfc internet spiders. the spider project is an example of how to use preemptive multithreading to gather information on the web using a spider/robot with the mfc wininet classes.

this project produces a spidering software program that checks web sites for broken url links. link verification is done only on href links. it displays a continously updated list of urls in a clistview that reports the status of the href link. the project could be used as a template for gathering and indexing information to be stored in a database file for queries.

search engines gather information on the web using programs called robots. robots (also called web crawlers, spiders, worms, web wanderers, and scooters) automatically gather and index information from around the web, and then put that information into databases. (note that a robot will index a page, and then follow the links on that page as a source for new urls to index.) users can than construct queries to search these databases to find the information they want.

by using preemptive multithreading, you can index a web page of url links, start a new thread to follow each new url link for a new source of urls to index.

the project uses the mdi cdocument used with a custom mdi child frame to display a ceditview when downloading web pages and a clistview when checking url links. the project also uses the cobarray, cinternetsession, chttpconnection, chttpfile, and cwinthread mfc classes. the cwinthread class is used to produce multiple threads instead of using the asynchronous mode in cinternetsession, which is realy left over from the winsock 16 bit windows platform.

the spider project uses simple worker threads to check url links or download a web page. the cspiderthread class is derived from the cwinthread class so each cspiderthread object can use the cwinthread message_map() function. by declaring a "declare_message_map()" in the cspiderthread class the user interface is still responsive to user input. this means you can check the url links on one web server and at the same time download and open a web page from another web server. the only time the user interface will become unresponsive to user input is when the thread count exceedes maximum_wait_objects which is defined as 64.

in the constructor for each new cspiderthread object we supply the threadproc function and the thread paramters to be passed to the threadproc function.

	cspiderthread* pthread;	pthread = null;	pthread = new cspiderthread(cspiderthread::threadfunc,pthreadparams); // create a new cspiderthread object

in the cspiderthread constructor we set the cwinthread* m_pthread pointer in the thread paramters structure so we can point to the correct instance of this thread;

pthreadparams->m_pthread = this;

the cspiderthread threadproc function

// simple worker thread proc functionuint cspiderthread::threadfunc(lpvoid pparam){	threadparams * lpthreadparams = (threadparams*) pparam;	cspiderthread* lpthread = (cspiderthread*) lpthreadparams->m_pthread;		lpthread->threadrun(lpthreadparams);	// use  sendmessage instead of postmessage here to keep the current thread count	// synchronizied. if the number of threads is greater than maximum_wait_objects (64)	// the program will be come	 unresponsive to user input	::sendmessage(lpthreadparams->m_hwndnotifyprogress,		wm_user_thread_done, 0, (lparam)lpthreadparams);  // deletes lpthreadparams and decrements the thread count	return 0;}

the structure passed to the cspiderthread threadproc function

typedef struct tagthreadparams{	hwnd m_hwndnotifyprogress;	hwnd m_hwndnotifyview;	cwinthread* m_pthread;	cstring m_pszurl;	cstring m_contents;	cstring m_strservername;	cstring m_strobject;	cstring m_checkurlname;	cstring m_string;	dword m_dwservicetype;	dword  m_threadid;	dword m_status;	urlstatus m_pstatus;	internet_port  m_nport;	int m_type;	bool m_rootlinks;}threadparams; 

after the cspiderthread object has been created we use the creatthread function to start the execution of the new thread object.

	if (!pthread->createthread())   //  starts execution of a cwinthread object	{		afxmessagebox("cannot start new thread");		delete pthread;		pthread = null;		delete pthreadparams;		return false;	}    

once the new thread is running we use the ::sendmessage function to send messages to the cdocument's-> clistview with the status structure of the url link.

	if(pthreadparams->m_hwndnotifyview != null)		::sendmessage(pthreadparams->m_hwndnotifyview,wm_user_check_done, 0, (lparam) &pthreadparams->m_pstatus);

sturcture used for url status.

typedef struct tagurlstatus{	cstring m_url;	cstring m_urlpage;	cstring m_statusstring;	cstring m_lastmodified;	cstring m_contenttype;	cstring m_contentlength;	dword	m_status;}urlstatus, * purlstatus;

each new thread creats a new cmyinternetsession (derived from cinternetsession) object with enablestatuscallback set to true, so we can check the status on all internetsession callbacks. the dwcontext id for callbacks is set to the thread id.

bool cinetthread::initserver(){		try	{		m_psession = new cmyinternetsession(agentname,m_nthreadid);		int ntimeout = 30;  // very important, can cause a server time-out if set to low							// or hang the thread if set to high.		/*		the time-out value in milliseconds to use for internet connection requests. 		if a connection request takes longer than this timeout, the request is canceled.		the default timeout is infinite. */		m_psession->setoption(internet_option_connect_timeout,1000* ntimeout);				/* the delay value in milliseconds to wait between connection retries.*/		m_psession->setoption(internet_option_connect_backoff,1000);				/* the retry count to use for internet connection requests. if a connection 		attempt still fails after the specified number of tries, the request is canceled.		the default is five. */		m_psession->setoption(internet_option_connect_retries,1);	        m_psession->enablestatuscallback(true);	}	catch (cinternetexception* pex)	{		// catch errors from wininet		//pex->reporterror();		m_psession = null;		pex->delete();		return false ;	}	return true;}

the key to using the mfc wininet classes in a single or multithread program is to use a try and catch block statement surrounding all mfc wininet class functions. the internet is very unstable at times or the web page you are requesting no longer exist, which is guaranteed to throw a cinternetexception error.

	try	{		// some mfc wininet class function	}	catch (cinternetexception* pex)	{		// catch errors from wininet		//pex->reporterror();		pex->delete();		return false ;	} 

the maximum count of threads is initially set to 64, but you can configure it to any number between 1 and 100. a number that is too high will result in failed connections, which means you will have to recheck the url links.

a rapid fire succession of http requests in a /cgi-bin/ directory could bring a server to it's knees. the spider program sends out about 4 http request a second. 4 * 60 = 240 a minute. this can also bring a server to it's knees. be carefull about what server you are checking. each server has a server log with the requesting agent's ip address that requested the web file. you might get some nasty email from a angry web server administrator.

you can prevent any directory from being indexed by creating a robots.txt file for that directory. this mechanism is usually used to protect /cgi-bin/ directories. cgi scripts take more server resources to retrieve.

when the spider program checks url links it's goal is to not request too many documents too quickly. the spider program adheres somewhat to the standard for robot exclusion. this standard is a joint agreement between robot developers, that allows www sites to limit what url's the robot requests. by using the standard to limit access, the robot will not retrieve any documents that web server's wish to disallow.

before checking the root url, the program checks to see if there is a robots.txt file in the main directory. if the spider program finds a robots.txt file the program will abort the search. the program also checks for the meta tag in all web pages. if it finds a meta name="robots" content ="noindex,nofollow" tag it will not index the urls on that page.

build:
windows 95
mfc/vc++ 5.0
wininet.h dated 9/25/97
wininet.lib dated 9/16/97
wininet.dll dated 9/18/97

problems:
can't seem to keep the thread count below 64 at all times.
limit of 32,767 url links in the clistview
wouldn't parse all urls correctly, will crash program occasionally using cstring functions with complex urls.

resources:
internet tools - fred forester
multithreading applications in win32
win32 multithreaded programming

download source code and example (65 kb)

last updated: 21 june 1998




【高速互连技术】合规性测试规范:物理层信号完整性与链路均衡验证系统设计 内容概要:本文档介绍了分析工具进行实践对照,并关注后续规范版本更新以确保符合最终发布标准。 立即下载

相关推荐

frontend-design skills

frontend-design skills

LAB4_Preemptive Multitasking_PartC Preemptive Multitasking and IPC

1. LAB4_Preemptive Multitasking_PartC: Preemptive Multitasking and Inter-Process communication (IPC) 在PartC 部分,我们要实现抢占非协作式环境,并且实现进程间通信。 1.1. Clock Interrupts and Preemption 如果一个进程获得CPU后一直死循环而不主动让出CPU的...

Small_Pond的博客 655

政府科技管理者在推动区域科技成果转化过程中,如何解决成果供需匹配效率低的问题?.docx

政府科技管理者在推动区域科技成果转化过程中,如何解决成果供需匹配效率低的问题?

pre-emptive multithreading web spider

 pre-emptive multithreading web spider this article was contributed by sim ayers. the win32 api supports applications that are pre-emptively multithreaded. this is a very useful and powerf

蝈蝈俊.net 2236

用于计算系统状态的卡尔曼最优增益和最小均方误差(MMSE)估计研究(Matlab代码实现)

内容概要:本文围绕用于计算系统状态的卡尔曼最优增益和最小均方误差(MMSE)估计展开研究,深入探讨了卡尔曼滤波在动态系统状态估计中的核心作用。文章系统阐述了卡尔曼滤波的数学基础,包括状态空间模型的建立、先验与后验状态估计、协方差矩阵的递推更新,以及最优增益矩阵的推导过程,确保在高斯白噪声环境下实现对系统状态的最小均方误差估计。结合Matlab代码实现,完整展示了算法从理论建模到仿真实验的全流程,涵盖系统建模、噪声设置、滤波迭代、状态更新与误差分析等环节,并通过仿真实验验证了算法在噪声干扰下对系统真实状态的高精度跟踪能力。该研究为控制工程、导航系统、信号处理及自动化领域中的状态估计问题提供了坚实的理论依据与高效的数值实现方案。; 适合人群:具备一定控制理论、概率统计基础和Matlab编程能力,从事自动化、电子信息、航空航天、机器人、导航定位等相关领域的科研人员、工程技术人员及高校研究生。; 使用场景及目标:①掌握卡尔曼滤波的基本原理及其在状态估计中的数学推导过程;②学习如何利用Matlab实现卡尔曼滤波算法并进行系统仿真与性能评估;③应用于目标跟踪、传感器融合、惯性导航、故障诊断、时间序列预测等实际工程中的动态状态估计问题。; 阅读建议:建议读者结合文中提供的Matlab代码进行动手实践,通过调整系统模型参数、过程噪声与观测噪声强度,观察滤波效果的变化,深入理解卡尔曼增益的自适应调节机制与MMSE估计的收敛特性。同时可进一步拓展学习扩展卡尔曼滤波(EKF)、无迹卡尔曼滤波(UKF)等非线性滤波方法,以应对更复杂的非线性系统状态估计挑战。

高校技术转移办公室人员如何提升成果转化服务效率?.docx

科易网基于行业知识库+行业大数据+科技资源要素+AI模型,以AI,赋能技术转移、成果转化、技术经理人培训、知识产权管理、区域产业分析与布局、地方科技招商、创新创业大赛等场景,推出科技创新领域的AI工具、场景化服务和数智化解决方案,推动科技创新与产业创新智能化发展。

Non-Preemptive Kernel

我受了刺激,我脑子坏掉,所以我翻开一本讲述嵌入式系统的书看了一节调度。巩固一些东西,也纠正一直以来的一些错误认识,同时又增加一些糊涂。随便说说,说错就改,我的借口是前两天喝了太多的酒。screen.width/2)this.width=screen.width/2;" align="right" border="0" src="http://www.ec66.com/article/pic/3a.

另一个空间 3101

国央企创新负责人如何利用平台实现技术需求的精准挖掘与匹配?.docx

科易网基于行业知识库+行业大数据+科技资源要素+AI模型,以AI,赋能技术转移、成果转化、技术经理人培训、知识产权管理、区域产业分析与布局、地方科技招商、创新创业大赛等场景,推出科技创新领域的AI工具、场景化服务和数智化解决方案,推动科技创新与产业创新智能化发展。

【图像与信号处理】基于MFCC与DTW融合的声纹识别算法在移动支付中的优化研究

本研究针对移动支付中声纹识别面临的噪声干扰、设备差异与安全挑战,优化MFCC与DTW融合算法:引入小波降噪与自适应Mel滤波器提升特征鲁棒性,设计轻量化模型适配嵌入式平台;提出加权DTW路径约束、模板自适应更新及两步匹配策略,结合动态门限判决增强防攻击能力。实验表明,算法识别准确率显著提升,响应时间<3秒,具备高噪声鲁棒性、跨设备兼容性与实时性,为移动支付提供了高效安全的声纹认证方案。

汉斯出版社旗下编辑,定期更新计算机前沿论文 185

自适应多机器人编队规划,以包围和跟踪具有运动和可见性约束的目标(Matlab代码实现)

内容概要:本文提出了一种自适应多机器人编队规划方法,旨在实现对具有运动能力和可见性约束目标的高效包围与持续跟踪,采用Matlab进行算法实现与仿真验证。该方法综合考虑了多机器人系统在动态环境中的协同控制、编队形态生成与保持、障碍物规避以及目标可见性维护等关键问题,通过设计分布式控制律使编队能够根据目标行为和环境变化自适应调整队形与运动轨迹,确保稳定包围并最大化目标观测质量。文中详细阐述了系统的数学建模过程、控制策略设计原理及约束处理机制,并提供了多场景下的仿真实验结果,验证了所提方法在复杂条件下的鲁棒性、灵活性与有效性。; 适合人群:具备自动化、机器人学、控制理论或人工智能等相关领域基础知识的科研人员及研究生,熟悉Matlab编程环境并致力于多智能体系统、协同控制、路径规划或智能无人系统研究的专业人士。; 使用场景及目标:①应用于多机器人系统在监视、围捕、搜救等任务中对移动目标的协同包围与跟踪控制;②为研究分布式编队控制、可见性优化、动态避障与多智能体协同决策等核心技术提供可复现的算法实例与理论支撑; 阅读建议:建议读者结合提供的Matlab代码逐模块分析算法实现细节,重点关注控制律设计、约束建模与仿真结果对比部分,宜在仿真环境中调整参数配置以深入理解系统性能边界与优化潜力。

Unity 升级引擎版本后构建报错:script class layout is incompatible between the editor and the player

Unity升级引擎版本后,项目报错:script class layout is incompatible between the editor and the player。

学习与分享 597

量化数据管道实战:从接口调用到生产级落库(第 15 篇):下半场导览与增量水位线设计

daily表建起来了,幂等写入的钥匙(复合主键)也交到了你手上。但真正的生产环境会立刻追问四个问题——数据怎么一直保持最新(增量调度)?不同来源的时间轴怎么对得上(多源对齐)?这套脚本怎么长期跑下去不出事(生产化)?落库还能再快再稳吗(落库进阶)?这就是下半场(第 15–28 篇)要解决的问题。本篇是下半场的主推篇,既是导览也是一篇技术文章给你一张覆盖第 15–28 篇的地图;增量水位线(watermark)设计与幂等写入原则——它决定了"只拉新数、重跑不重"能不能成为整条管道的默认行为;

2601_96840033的博客 3312

沧小海深入剖析xilinx的GTP/GTX核,掌握高速串行收发机制——第六章 接收端结构及功能说明

总目录在这哦~ https://blog.youkuaiyun.com/z123canghai/article/details/107700878 第六章 接收端结构及功能说明 子目录 6.1 RX的模拟前端 6.2 RX的OOB信号 6.3 RX的均衡器 6.4 RX的CDR 时钟数据恢复 6.5 RX时钟输出结构 6.6 RX 的极性控制 6.7 RX 的PRBS 6.8 Rx字节、字的对齐功能 6.9 Rx接收弹性缓冲区 6.10 Rx的时钟纠正 6.11 Rx接收通道绑定 6...

滄海的博客 1万+

05 · 动态分析 — x64dbg 与调试技术

本章系统讲解Windows平台下基于x64dbg的动态分析技术,涵盖界面操作、四类断点(软件、硬件、内存、条件)、内存分析、调用栈与参数观察、高级技巧(API断点、消息断点、追踪、补丁)及插件生态。结合静态分析(IDA)与动态调试联动工作流,通过三个实践项目强化技能落地,助力掌握程序运行时行为分析核心能力。

2503_91800161的博客 613

SPMI、PMBus、AVSBus:三大电源管理总线对比与选型指南

在现代电子系统中,电源管理是确保高性能与低功耗平衡的核心技术。其原理在于通过数字通信协议,实现电源管理单元(PMIC)与用电芯片间的精准“对话”,从而动态调整供电。这项技术的核心价值在于提升系统能效、增强可靠性并简化设计。其应用场景广泛,从智能手机、服务器到汽车电子无处不在。具体实现上,衍生出了三大主流协议:为移动嵌入式高度集成的SPMI总线、专注于工业标准监控的PMBus总线,以及追求极致实时能效的AVSBus总线。理解SPMI在复杂时序控制与PMBus在标准化配置方面的异同,是进行电源架构选型与调试的关

ithome 463

Unity插件UniWebView实战:UGUI动态调整内嵌网页尺寸与交互优化

本文详细解析了如何在Unity游戏中使用UniWebView插件实现内嵌网页的动态尺寸适配与深度交互。核心在于利用Reference Rect Transform属性,将网页窗口精准绑定至UGUI控件,并随其动态调整。文章进一步探讨了通过消息系统与JavaScript执行实现Unity与网页的双向通信,并提供了加载优化、过渡动画及事件拦截等实用技巧,助力开发者打造无缝、流畅的游戏内网页体验。

earth的博客 278

MFC 多线程技术笔记:CWinThread、CEvent 与 CCriticalSection 实战

本文针对MFC多线程项目中相机并发采集导致程序闪退、轴卡报警等问题,通过重写一个模拟多线程生成TXT文件的Demo,排查多线程处理逻辑。结合实际代码,梳理了MFC多线程创建(AfxBeginThread)、线程同步(事件CEvent、临界区CCriticalSection)、消息传递(SendMessage)及线程安全设计等关键技术点,验证了线程管理与资源协调的合理性,为后续调试提供可靠依据。

... 431

2022年8月27日全国青少年信息素养大赛Python编程小学组决赛真题+答案解析

2022年8月27日全国青少年信息素养大赛Python编程小学组决赛真题+答案解析

伶俐角少儿编程 456
上一篇: 公历到农历的转换法
下一篇: 利用java编写网络通信程序
techweb
博客等级 码龄19年 226粉丝 3842原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值