简单的新闻爬虫

Python爬虫 | 爬取环境新闻实战 (2)第一页的url为“http://hbj.wuhan.gov.cn/hjxw/index.shtml”,,第二页的url“http://hbj.wuhan.gov.cn/hjxw/index_1.shtml”为那么之后第N页的url为在后面加上_ {i},即为“http://hbj.wuhan.gov.cn/hjxw/index_{i}.shtml”Python所有方向的技术点做的整理,形成各个领域的知识点汇总,它的用处就在于,你可以按照上面的知识点去找对应的学习资源,保证自己学得较为全面。 阅读详情

今天我要更新我自己写的一个爬虫程序,哇。厉害了,我一个菜鸟居然有勇气些爬虫程序,简直胆大包天,但是,不管怎样,我也要上传,没办法,我就是脸皮厚,我是不怕别人笑。

# -*- coding: utf-8 -*-

'''
此爬虫程序实现的是每天23:55自动的去抓取凤凰新闻网站当天更新的所有新闻,然后存入数据库.抓取的目标为:图		片,标题,出处,时间,新闻内容之所以要在当天的23:55开始抓取,是因为凤凰新闻网站的新闻为不定时更新的,基本每天会在23之后还会更新一条新闻,为了不漏掉,所以定在23:55开始抓取,第二天就会有前一天所有的新闻数据了.
'''

import csv
from selenium import webdriver
import pymysql
import time

class new(object):
	#初始化数据
	def __init__(self):
    #定义访问路径
    self.url='https://news.ifeng.com/'
    #下面创建数据库对象
    self.db = pymysql.connect('localhost','root','123456','newsdb',charset='utf8')
    #创建游标对象
    self.cursor = self.db.cursor()

	#获取页面节点
	def getweb(self):
    	driver = webdriver.PhantomJS()  #这里也可以声明普通浏览器
   		#打开凤凰新闻网
    	driver.get(self.url)
   		driver.implicitly_wait(10) #等待10秒
    	driver.maximize_window() # 最大化窗口
    	driver.implicitly_wait(5) #等待5秒
    	#先循环点击更多信息按钮,加载足够的新闻,一般点击三次肯定就能看得到前一天的新闻
    	i = 1
    	while i<4:
       		#获取更多信息按钮节点
        	button = driver.find_element_by_class_name('n
一个简单的爬虫——新闻爬虫 公众号的第一篇文章,就先来介绍一下我做的最多的也是最简单的新闻爬虫吧。这个爬虫本身是用java写的,搭载在我之前项目的服务器上,今天用python实现一下。这个爬虫我也给别人讲过很多次,在双创之星的舞台上讲过,在新生导航课上讲过(两次),在课堂上讲过。其实现在回头看一下这个爬虫真的很low很简单,但好歹也是我花了很久学习的,今天就系统的用python来实现一下吧。 欢迎公众号关注:老白和他的爬... 阅读详情

相关推荐

Python 网络爬虫基础知识及体验

电源电路是一个电子产品的重要组成部分,电源电路设计的好坏,直接牵连产品性能的好坏。我们电子产品的电源电路主要有线性电源和高频开关电源。从理论上讲,线性电源是用户需要多少电流,输入端就要提供多少电流;开关电源是用户需要多少功率,输入端就要提供多少功率。 线性电源 线性电源功率器件工作在线性状态,如我们常用的稳压芯片 LM7805、LM317、SPX1117等。下图是 LM7805 稳压电源电路原理图...

程序员「阿基米东」的编程之旅 1160

安装m_map以及用matlab绘制高精度海岸线地图

#如何用matlab绘制高精度海岸线地图

qq_38512775的博客 1万+

戴尔PowerEdge服务器R450 RAID配置步骤

戴尔PowerEdge服务器R450 RAID配置步骤

且听风呤tmj的博客 6917

5分钟搞定阿里云千问大模型API申请:手把手教你获取API Key

本文提供了一份高效的阿里云通义千问大模型API Key获取实战指南。通过清晰的步骤,帮助开发者在五分钟内完成从账号准备、精准导航控制台到创建并验证API Key的全过程,并强调了密钥安全管理和基础调用测试的重要性,助力快速集成AI能力。

ol789012345的博客 1365

深入解析CycloneDDS跨域通信中的XML配置文件优化策略

本文深入解析了CycloneDDS跨域通信中XML配置文件的优化策略,重点介绍了网络接口配置、多播设置、性能调优参数及复杂场景下的配置案例。通过实战技巧和调试方法,帮助开发者在SoC环境中实现高效稳定的跨域通信,提升系统性能。

soda5的博客 690

比利时风电光伏负荷一体化数据集解析与应用

本文深度解析了由比利时输电系统运营商Elia集团提供的风电光伏负荷一体化数据集。该数据集整合了负荷、风电与光伏出力数据,并包含多时间尺度的官方预测值,时间跨度长达四年,是研究能源转型、构建电力预测模型及分析电网运行的宝贵资源。文章详细介绍了数据价值、字段含义、实战应用场景(如预测模型基准测试、风光互补性分析)以及数据处理指南,为相关领域的研究与开发提供了高质量数据基础和实践参考。

weixin_29200741的博客 429

微机原理与接口技术第六版周荷琴课后习题答案及解析(期末重点)

第二章 1.8086/8088CPU可直接寻址多少个内存(字节)单元?多少个I/O端口?它们的外部数据总线各有多少根? 8086/8088CPU使用的是20位地址总线,因此可以直接寻址2的20次方个内存单元(即1,048,576),每个单元大小为1字节,因此可以直接寻址2的20次方个字节的内存。 8086/8088CPU有64K个I/O端口,每个端口占用1字节的地址空间,因此可以直接寻址6...

Wmicheng的博客 1万+

【GameFramework框架】三、快速启动

这篇文章就带领大家如何快速启动GameFramework框架。

优快云博客专家,Unity3D高级软件工程师,《Unity 3D从入门到实战》作者。 4435

9.6华为OD机试真题 新系统【仓储货物有序出库】

华为OD机试真题 新系统【仓储货物有序出库】,提供C语言、Java、C++、Python、Go、JavaScript源码实现,并提供完整思路讲解,以及在线OJ模拟做题。华为OD机试真题 【仓储货物有序出库】多语言题解。

qq_45776114的博客 902

NTscan扫描工具

ntscan是一个扫描器,可以扫描没有密码和弱密码的共享,先到配置文件中填写猜测的用户名和密码,扫描过程中右侧出现的是扫描日志。

android 蓝牙通讯demo

android 下蓝牙通讯demo,实现了发现蓝牙,使能蓝牙,通过蓝牙发送消息。

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

假人有限元模型

假人模型,车辆碰撞假人有限元模型,适合做乘员安全的研究者利用

上一篇: day03
下一篇: day04
study_python3
博客等级 码龄8年 4粉丝 12原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值