python scrapy对抓取数据后的存储 sqilte3的安装_python scrapy sqliter3-优快云博客

本文链接：https://blog.youkuaiyun.com/csupengu/article/details/8885610

对于抓取数据量，链接数不是很大的情况下，可以选sqlite作为存储数据库。sqlite轻巧，操作方便，快速。不像mysql那样庞大。

centos下python2.7 scrapy0.16 ，默认情况下，没有sqlite模块，我们要自己安装。

用easy_install 装，安装完后，import，出错了！，如下

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/python2.7.4/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/usr/local/python2.7.4/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3

网上搜到解决方案是：重装python，重新编译，好吧，照做就是了，可编译python过程又出现问题了。。，如下：

Python build finished, but the necessary bits to build these modules were not found:
_tkinter bsddb185 sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_sqlite3

再到网上找解决方案：

在./Modules/_sqlite/connection.c 文件中加入红色部分

#include "cache.h"
#include "module.h"
#include "connection.h"
#include "statement.h"
#include "cursor.h"
#include "prepare_protocol.h"
#include "util.h"
#include "sqlitecompat.h"

#include "pythread.h"

#define ACTION_FINALIZE 1
#define ACTION_RESET 2

#if SQLITE_VERSION_NUMBER >= 3003008
#ifndef SQLITE_OMIT_LOAD_EXTENSION
#define HAVE_LOAD_EXTENSION
#endif
#endif

#ifdef SQLITE_INT64_TYPE
typedef SQLITE_INT64_TYPE sqlite_int64;
typedef unsigned SQLITE_INT64_TYPE sqlite_uint64;
#elif defined(_MSC_VER) || defined(__BORLANDC__)
typedef __int64 sqlite_int64;
typedef unsigned __int64 sqlite_uint64;
#else
typedef long long int sqlite_int64;
typedef unsigned long long int sqlite_uint64;
#endif
typedef sqlite_int64 sqlite3_int64;
typedef sqlite_uint64 sqlite3_uint64;