实现一个简单的爬虫代码时，可以将代码分成以下几个模块：
1. URL管理模块：负责管理需要爬取的URL队列、已经爬取过的URL集合，以及判断一个URL是否可以爬取等功能；
2. HTML下载模块：负责下载指定URL对应的HTML页面，并将其返回给爬虫；
3. HTML解析模块：负责解析HTML页面，提取需要的数据和新的URL，并将它们返回给爬虫；
4. 数据存储模块：负责将爬取到的数据保存到本地文件或数据库中。
下面是一个简单的爬虫代码的
模块化实现：
```python
# url_manager.py
class UrlManager:
def __init__(self):
self.new_
urls = set()
self.old_
urls = set()
def has_new_url(self):
return len(self.new_
urls) != 0
def add_new_url(self, url):
if url not in self.new_
urls and url not in self.old_
urls:
self.new_
urls.add(url)
def get_new_url(self):
url = self.new_
urls.pop()
self.old_
urls.add(url)
return url
# html_downloader.py
import urllib.request
class HtmlDownloader:
def download(self, url):
if url is None:
return None
response = urllib.request.urlopen(url)
if response.getcode() != 200:
return None
return response.read()
# html_parser.py
from bs4 import BeautifulSoup
import re
import urllib.parse
class HtmlParser:
def parse(self, url, html_content):
if url is None or html_content is None:
return None
soup = BeautifulSoup(html_content, 'html.parser')
new_
urls = self._get_new_
urls(url, soup)
new_data = self._get_new_data(url, soup)
return new_
urls, new_data
def _get_new_
urls(self, url, soup):
new_
urls = set()
links = soup.find_all('a', href=re.compile(r'^/item/'))
for link in links:
new_url = urllib.parse.urljoin(url, link['href'])
new_
urls.add(new_url)
return new_
urls
def _get_new_data(self, url, soup):
data = {}
data['url'] = url
title = soup.find('dd', class_='lemmaWgt-lemmaTitle-title').find('h1').get_text()
data['title'] = title
summary = soup.find('div', class_='lemma-summary').get_text()
data['summary'] = summary
return data
# html_outputer.py
class HtmlOutputer:
def __init__(self):
self.datas = []
def collect_data(self, data):
if data is None:
return
self.datas.append(data)
def output_html(self):
fout = open('output.html', 'w', encoding='utf-8')
fout.write('<html>')
fout.write('<head><meta charset="utf-8"></head>')
fout.write('<body>')
fout.write('<table>')
for data in self.datas:
fout.write('<tr>')
fout.write('<td>%s</td>' % data['url'])
fout.write('<td>%s</td>' % data['title'])
fout.write('<td>%s</td>' % data['summary'])
fout.write('</tr>')
fout.write('</table>')
fout.write('</body>')
fout.write('</html>')
fout.close()
# spider_main.py
class SpiderMain:
def __init__(self):
self.
urls = UrlManager()
self.downloader = HtmlDownloader()
self.parser = HtmlParser()
self.outputer = HtmlOutputer()
def crawl(self, root_url):
self.
urls.add_new_url(root_url)
while self.
urls.has_new_url():
try:
new_url = self.
urls.get_new_url()
html_content = self.downloader.download(new_url)
new_
urls, new_data = self.parser.parse(new_url, html_content)
self.
urls.add_new_
urls(new_
urls)
self.outputer.collect_data(new_data)
except Exception as e:
print('crawl failed:', e)
self.outputer.output_html()
```
这个爬虫程序中，每个模块都有自己的功能，且它们之间相互独立。这种
模块化的实现方式可以让代码更加清晰，易于维护和扩展。
当项目变大，urls增多，将其放在一个文件中不利于管理，可采用模块化方式，在app中新建文件存储相关子urls。同时需注意使用include函数包含子urls，匹配规则放于特定变量，拼接时避免多加斜杠。
扫一扫
3332
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
