python爬虫获取网页数据,python爬取本地网页数据

最新推荐文章于 2025-05-28 19:46:40 发布

2401_85305177

最新推荐文章于 2025-05-28 19:46:40 发布

阅读量425

点赞数 3

文章标签：命令模式

本文链接：https://blog.youkuaiyun.com/2401_85305177/article/details/139655786

版权

大家好，本文将围绕利用python爬取简单网页数据步骤展开说明，python怎么爬取网站上的数据是一个很多人都想弄明白的事情，想搞清楚python怎么爬取网站所有网页需要先了解以下几个事情。

大家好，小编来为大家解答以下问题，python小程序抢购脚本怎么写，python如何做抢东西软件，现在让我们一起来看看吧！

大家好，小编为大家解答python小程序抢购脚本怎么写的问题。很多人还不知道python如何做抢东西软件，现在让我们一起来看看吧！

本篇文章给大家谈谈python小程序抢购脚本，以及python简单小程序代码，希望对各位有所帮助，不要忘了收藏本站喔python画四瓣花。

安装flask

创建一个flask虚拟环境
[root@shaoyu ~]# mkvirtualenv flask
进入flask虚拟环境并安装falsk
(flask) [root@shaoyu ~]# pip install flask  #pip源如果是国外的，可能安装过程会很漫长，更换到国内pip源即可，另外一个因素受限于个人网络环境
#测试倒入flask是否成功
(flask) [root@shaoyu ~]# python
Python 3.6.10 (default, Jul 22 2020, 11:39:06) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask 
>>> exit()   #导入没问题
(flask) [root@shaoyu ~]#

测试flask

(flask) [root@shaoyu ~]# mkdir flask
(flask) [root@shaoyu ~]# vim  
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
    return "Hello World!"
if __name__ == '__main__':
    ()
运行
(flask) [root@shaoyu flask]# python  
 * Serving Flask app "hello" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
在另外一个终端中访问测试
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World!
证明flask没问题

flask框架之路由规划

(flask) [root@shaoyu flask]# vim  
#-*- coding:utf-8 -*-

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return "Hello World!"

@app.route("/api")
def index()
    return "Index page"

@app.route("/api/hello")
def hello()
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0' ) #监听所有端口，用于宿主机访问测试
运行
(flask) [root@shaoyu flask]# python 
在另一个终端中查看结果
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World!
[root@shaoyu ~]# elinks 127.0.0.1:5000/api --dump
   Index page
[root@shaoyu ~]# elinks 127.0.0.1:5000/api/hello --dump
   Hello World

flask.register._blueprint方法

(flask) [root@shaoyu flask]# pwd
/root/flask
(flask) [root@shaoyu flask]# ls
  
(flask) [root@shaoyu flask]# vim 
#-*- coding:utf-8 -*-
from flask import Blueprint

route_imooc = Blueprint( "imooc_page", __name__ )

@route_imooc.route('/')
def index():
    return "imooc index page"

@route_imooc.route('/hello')
def hello():
    return "imooc hello world"

(flask) [root@shaoyu flask]# cp  
(flask) [root@shaoyu flask]# vim  
(flask) [root@shaoyu flask]# cat  
#-*- coding:utf-8 -*-

from flask import Flask
from imooc import route_imooc

app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )

@app.route('/')
def hello_world():
    return "Hello World!"

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0' ) #监听所有端口，用于宿主机访问测试
(flask) [root@shaoyu flask]# ls

运行

(flask) [root@shaoyu flask]# python  
 * Serving Flask app "02-hello-imooc" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
在另外一个终端中测试
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World!
[root@shaoyu ~]# elinks 127.0.0.1:5000/imooc/hello --dump
   imooc hello world
[root@shaoyu ~]# elinks 127.0.0.1:5000/imooc/ --dump
   imooc index page

flask之链接管理器（url_for）和版本管理器

链接管理器url_for
(flask) [root@shaoyu flask]# cp  
(flask) [root@shaoyu flask]# vim  
#-*- coding:utf-8 -*-

from flask import Flask,url_for
from imooc import route_imooc

app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )

@app.route('/')
def hello_world():
    url = url_for( 'index' ) 
    return "Hello World," + url

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0' ) #监听所有端口，用于宿主机访问测试

(flask) [root@shaoyu flask]# python  
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World,/api

(flask) [root@shaoyu flask]# mkdir common
(flask) [root@shaoyu flask]# touch 
(flask) [root@shaoyu flask]# mkdir common/libs
(flask) [root@shaoyu flask]# touch 
(flask) [root@shaoyu flask]# touch 
(flask) [root@shaoyu flask]# vim 
#-*- coding:utf-8 -*- 

class UrlManager(object):
    @staticmethod
    def buildUrl( path ):
        return path

    @staticmethod
    def buildStaticUrl( path ):
        return path
(flask) [root@shaoyu flask]# vim  
#-*- coding:utf-8 -*-

from flask import Flask,url_for
from imooc import route_imooc
from .UrlManager import UrlManager
app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )

@app.route('/')
def hello_world():
    url = url_for( 'index' )
	url_1 = UrlManager.buildUrl( '/api' )
    return "Hello World, url:%s, url_1:%s"%( url, url_1)

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0' ) #监听所有端口，用于宿主机访问测试
(flask) [root@shaoyu flask]# python 
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World, url:/api, url_1:/api
(flask) [root@shaoyu flask]# vim 
#-*- coding:utf-8 -*-
'''
每次发版都会有对应的版本号,如(呵呵0,202007212210),这里为了举例，先随机定义一个版本号
'''

from flask import Flask,url_for
from imooc import route_imooc
from .UrlManager import UrlManager
app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )

@app.route('/')
def hello_world():
    url = url_for( 'index' )
    url_1 = UrlManager.buildUrl( '/api' )
    url_2 = UrlManager.buildStaticUrl('')
    return "Hello World, url:%s, url_1:%s, url_2:%s"%( url, url_1, url_2)

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0' ) #监听所有端口，用于宿主机访问测试
(flask) [root@shaoyu flask]# cat  
#-*- coding:utf-8 -*- 

class UrlManager(object):
    @staticmethod
    def buildUrl( path ):
        return path

    @staticmethod
    def buildStaticUrl( path ):
        path = path + '?ver=' + '202007212210' #这里先把这个版本号写死，为了验举例
        return UrlManager.buildUrl( path ) 
(flask) [root@shaoyu flask]# python  
[root@shaoyu ~]# elinks 127.0.0.1:5000 --dump
   Hello World, url:/api, url_1:/api,
   ?ver=202007212210

详细的用法后续会再做详细介绍

flask日志系统

(flask) [root@shaoyu flask]# cat  
#-*- coding:utf-8 -*-
'''
每次发版都会有对应的版本号,如(呵呵0,202007212210)
'''

from flask import Flask,url_for
from imooc import route_imooc
from .UrlManager import UrlManager
app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )

@app.route('/')
def hello_world():
    url = url_for( 'index' )
    url_1 = UrlManager.buildUrl( '/api' )
    url_2 = UrlManager.buildStaticUrl('')
    msg = "Hello World, url:%s, url_1:%s, url_2:%s"%( url, url_1, url_2)

    app.logger.error( msg )
    ( msg )
    app.logger.debug( msg )
    
    return msg

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0', debug = True ) #监听所有端口，用于宿主机访问测试;debug = true表示开启开发者调试模式，在运行程序时，可以直接更改文件而不需要停掉程序重新运行
运行观察输出结果，debug开启时，运行中的程序会实时神经网络文件，并输出更新内容 
* Detected change in '', reloading
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 125-377-728
[2020-07-22 17:40:05,367] ERROR in 06-flask_log: Hello World, url:/api, url_1:/api, ?ver=202007212210
[2020-07-22 17:40:05,367] INFO in 06-flask_log: Hello World, url:/api, url_1:/api, ?ver=202007212210
[2020-07-22 17:40:05,367] DEBUG in 06-flask_log: Hello World, url:/api, url_1:/api, ?ver=202007212210

flask错误处理

(flask) [root@shaoyu flask]# cp  
(flask) [root@shaoyu flask]# vim  
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
    return "Hello World!"

@app.errorhandler(404)
def page_not_found(error):
	app.logger.error( error )
	return "This is page are not exist", 404 
if __name__ == '__main__':
    ( host='0.0.0.0', debug = True ) #监听所有端口，用于宿主机访问测试
运行程序
(flask) [root@shaoyu flask]# python  
在另一个终端中测试一个不存在的地址
[root@shaoyu flask]# elinks 127.0.0.1:5000/haha/ --dump
   This is page are not exist
回到刚才运行程序窗口观察实时输出
 * Serving Flask app "07-flask_error" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 125-377-728
[2020-07-22 17:48:48,504] ERROR in 07-flask_error: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
127.0.0.1 - - [22/Jul/2020 17:48:48] "GET /haha/ HTTP/1.1" 404 -
[2020-07-22 17:49:08,015] ERROR in 07-flask_error: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

flask框架之数据库ORM

mysql数据库的配置这里就不再赘述了，在安装mysql的时候已经重置过mysql密码了python创意turtle作品。这里直接进行flask-sqlalchemy的安装
(flask) [root@shaoyu flask]# pip install flask-sqlalchemy
安装mysqlclient
解决依赖
(flask) [root@shaoyu flask]# yum install -y mysql-devel gcc gcc-devel python-devel
(flask) [root@shaoyu flask]# pip install mysqlclient

创建进行测试

(flask) [root@shaoyu flask]# vim 
#-*- coding:utf-8 -*-
'''
每次发版都会有对应的版本号,如(呵呵0,202007212210)
'''

from flask import Flask,url_for
from imooc import route_imooc
from .UrlManager import UrlManager
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.register_blueprint( route_imooc, url_prefix = '/imooc' )
app.config['SQLALCHEMY_DATABASE_URI'] =  'mysql://root:Mysql-123@127.0.0.1/mysql'
db = SQLAlchemy(app)

@app.route('/')
def hello_world():
    url = url_for( 'index' )
    url_1 = UrlManager.buildUrl( '/api' )
    url_2 = UrlManager.buildStaticUrl('')
    msg = "Hello World, url:%s, url_1:%s, url_2:%s"%( url, url_1, url_2)

    app.logger.error( msg )
    ( msg )
    app.logger.debug( msg )
    
    return msg

@app.route("/api")
def index():
    return "Index page"

@app.route("/api/hello")
def hello():
    from sqlalchemy import text
    sql = text('select * from `user`')
    result  = db.engine.execute( sql )
    for  row in  result:
        ( row )
    return "Hello World"

if __name__ == '__main__':
    ()
    ( host='0.0.0.0', debug = True ) #监听所有端口，用于宿主机访问测试;debug = true表示开启开发者调试模式，在运行程序时，可以直接更改文件而不需要停掉程序重新运行

访问测试

(flask) [root@shaoyu flask]# python 
在另一个终端中访问
(flask) [root@shaoyu flask]# elinks 127.0.0.1:5000/api/hello --dump
切回运行程序终端查看
* Debugger is active!
 * Debugger PIN: 125-377-728
[2020-07-23 08:03:07,031] INFO in 08-flask_mysql: ('localhost', 'root', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', '', b'', b'', b'', 0, 0, 0, 0, 'mysql_native_password', '*FC9B6504FDD72E185BAC4C3F5BC3AFDD51069E55', 'N', datetime.datetime(2020, 7, 22, 9, 58, 12), None, 'N')
[2020-07-23 08:03:07,032] INFO in 08-flask_mysql: ('localhost', 'mysql.session', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'Y', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', '', b'', b'', b'', 0, 0, 0, 0, 'mysql_native_password', '*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE', 'N', datetime.datetime(2020, 7, 22, 9, 49, 35), None, 'Y')
[2020-07-23 08:03:07,032] INFO in 08-flask_mysql: ('localhost', '', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', '', b'', b'', b'', 0, 0, 0, 0, 'mysql_native_password', '*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE', 'N', datetime.datetime(2020, 7, 22, 9, 49, 35), None, 'Y')
127.0.0.1 - - [23/Jul/2020 08:03:07] "GET /api/hello HTTP/1.1" 200 -

构建高可用的Flask MVC框架(类似下面这种文件结构)

查看目录结构

xueji/
├── 
├── common
│   ├── 
│   ├── libs
│   │   └── 
│   └── models
│       └── 
├── config
│   ├── 
│   ├── 
│   ├── 
│   └── 
├── docs
│   └── 
├── jobs
│   ├── 
│   └── tasks
│       └── 
├── 
├── __pycache__
│   ├── 
│   └── 
├── 
├── 
├── web
│   ├── controllers
│   │   ├── 
│   │   ├── 
│   │   └── __pycache__
│   ├── 
│   └── __pycache__
│       ├── 
│       └── 
└── 

12 directories, 22 files

在/root/flask/flask_high_available/xueji/目录下：

(flask) [root@shaoyu xueji]# tree -L 2 config/
config/
├── 
├── 
├── 
└──

查看

(flask) [root@shaoyu xueji]# cat  
# -*- coding:utf-8 -*-
SERVER_PORT = 5600 
DEBUG = False
SQLALCHEMY_ECHO = False

查看

(flask) [root@shaoyu xueji]# cat  
# -*- coding:utf-8 -*-
DEBUG = True
SQLALCHEMY_ECHO = True
SQLALCHEMY_DATABASE_URI = 'mysql://root:Mysql-123@127.0.0.1/mysql?charset=utf8mb4'
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_ENCODING = 'utf8mb4'

文件可以为空，表示生产环境下的配置，这里可以不写

在/root/flask/flask_high_available/xueji 目录下

查看

(flask) [root@shaoyu xueji]# cat  
# -*- coding:utf-8 -*-
import os
from flask import Flask
from flask_ import Manager
from flask_sqlalchemy import SQLAlchemy

class Application( Flask):
	def  __init__(self, import_name):
		super( Application, self).__init__( import_name )
		self.config.from_pyfile( '' )
		if "ops_config" in os.environ:
			self.config.from_pyfile( ''%os.environ['ops_config'] )
		db.init_app( self )

db = SQLAlchemy()
app = Application( __name__ )
manager = Manager( app )

查看

(flask) [root@shaoyu xueji]# cat  
# -*- coding:utf-8 -*-  

import www
from flask_ import Server
from application import app,manager

#web server
#manager.add_command( 'runserver', Server( host = '0.0.0.0', port = app.config['SERVER_PORT'], use_debugger = True, use_reloader = True) )
manager.add_command( "runserver", Server( host='0.0.0.0',port=app.config['SERVER_PORT'],use_debugger=True,use_reloader = True) )

def main():
	( host='0.0.0.0', debug=True )
	()

if __name__ == '__main__':
	try:
		import sys
		( main() )
	except Exception as e:
		import traceback
		traceback.print_exc()

查看

(flask) [root@shaoyu xueji]# cat  
#-*-  coding:utf-8 -*- 
from application import app
from web.controllers.index import route_index
app.register_blueprint( route_index,url_prefix = '/' )

查看

(flask) [root@shaoyu xueji]# cat  
flask
flask-sqlalchemy
mysqlclient
flask_

查看

(flask) [root@shaoyu xueji]# cat  
====================
Python Flask订餐系统
====================
##启动
*export ops_config=local|production && python  runserver

## flask-sqlalcodegen
	flask-sqlacodegen 'mysql://root:Mysql-123@127.0.0.1/food_db' --outfile
	"" --flask
	flask-sqlacodegen 'mysql://root:Mysql-123@127.0.0.1/food_db' --tables user
	user --outfile ''  --flask

在/root/flask/flask_high_available/xueji/web/controllers目录下

查看

(flask) [root@shaoyu controllers]# cat  
#-*- coding:utf-8 -*-
from flask import Blueprint

route_index = Blueprint( 'index_page', __name__ )

@route_index.route('/')
def index():
	return "Hello controllers"

到此，环境和测试程序一准备完毕，上面没有提到的暂时不用写，只需要这些就可以跑起来，测试下效果，但是文件一定要通上面的文件结构一样，且没有提到的文件可以为空，但是文件必须存在小狗ai仿写软件。

打开两个终端测试

(flask) [root@shaoyu xueji]# export ops_config=local && python  runserver
 * Serving Flask app "application" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5600/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 557-520-524

在另一终端中测试

(flask) [root@shaoyu order]# elinks 127.0.0.1:5600 --dump
   Hello controllers
更改下config下的中的端口，再次运行，观察结果
(flask) [root@shaoyu xueji]# grep '6600'  
SERVER_PORT = 6600

再次运行查看

(flask) [root@shaoyu xueji]# export ops_config=local && python  runsver
 * Serving Flask app "application" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:6600/ (Press CTRL+C to quit) #注意看端口已经更改
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 557-520-524