使用litePal来做为数据库的框架:
0:首先导入litepal 的依赖和fastjson的依赖
dependencies {
compile fileTree(include: ['*.jar'], dir: 'libs')
testCompile 'junit:junit:4.12'
compile 'com.android.support:appcompat-v7:23.4.0'
compile 'org.litepal.android:core:1.3.2'
compile 'com.alibaba:fastjson:1.2.13'
}
1:创建一个MyApplication,
package tech.androidstudio.dbtest;
import org.litepal.LitePalApplication;
/**
* Created by Kodulf on 2016/6/30.
*/
public class MyApplication extends LitePalApplication {
@Override
public void onCreate() {
super.onCreate();
}
}
然后修改清单文件:android:name=".MyApplication"
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="tech.androidstudio.dbtest">
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"></uses-permission>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"></uses-permission>
<application
android:allowBackup="true"
android:icon="@mipmap/ic_launcher"
android:label="@string/app_name"
android:name=".MyApplication"
android:supportsRtl="true"
android:theme="@style/AppTheme">
<activity android:name=".MainActivity">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>2:在main的文件夹下面创建一个assets的文件夹,然后创建一个litepal.xml的文件,
里面的list 是我们下面的创建的类Account,就是会映射为一个表
<?xml version="1.0" encoding="utf-8"?>
<litepal>
<dbname value="demo" ></dbname>
<version value="1" ></version>
<list>
<mapping class="tech.androidstudio.dbtest.Account"></mapping>
</list>
</litepal>3:创建Account的类:记住一定要包含一个空的参数的构造方法,因为下面会用到FastJson转换的时候用到,不然会报错的。
package tech.androidstudio.dbtest;
import org.litepal.crud.DataSupport;
import java.util.Date;
/**
* Created by Kodulf on 2016/6/30.
*/
public class Account extends DataSupport {
//(uid, acc_no, realname, mobile, attribute, status, cdate, edate)
// VALUES ('20009', null, null, '18329160075', 1, 1, '2012-08-30 15:12:35', '2014-02-28 02:54:25');
private int uid;
private String acc_no;
private String realname;
private String mobile;
private int attribute;
private int status;
private Date cdate;
private Date edate;
public Account() {
}
public Account(int uid, String acc_no, String realname, String mobile, int attribute, int status, Date cdate, Date edate) {
this.uid = uid;
this.acc_no = acc_no;
this.realname = realname;
this.mobile = mobile;
this.attribute = attribute;
this.status = status;
this.cdate = cdate;
this.edate = edate;
}
public int getUid() {
return uid;
}
public void setUid(int uid) {
this.uid = uid;
}
public String getAcc_no() {
return acc_no;
}
public void setAcc_no(String acc_no) {
this.acc_no = acc_no;
}
public String getRealname() {
return realname;
}
public void setRealname(String realname) {
this.realname = realname;
}
public String getMobile() {
return mobile;
}
public void setMobile(String mobile) {
this.mobile = mobile;
}
public int getAttribute() {
return attribute;
}
public void setAttribute(int attribute) {
this.attribute = attribute;
}
public int getStatus() {
return status;
}
public void setStatus(int status) {
this.status = status;
}
public Date getCdate() {
return cdate;
}
public void setCdate(Date cdate) {
this.cdate = cdate;
}
public Date getEdate() {
return edate;
}
public void setEdate(Date edate) {
this.edate = edate;
}
}
4:修改activity_main.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:paddingBottom="@dimen/activity_vertical_margin"
android:paddingLeft="@dimen/activity_horizontal_margin"
android:paddingRight="@dimen/activity_horizontal_margin"
android:paddingTop="@dimen/activity_vertical_margin"
android:orientation="vertical"
tools:context="tech.androidstudio.dbtest.MainActivity">
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="开始加载"
android:onClick="doImport"/>
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="生成json的文件"
android:onClick="createJsonFile"/>
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="开始加载更新后的json文件"
android:onClick="startToImportJsonFiles"/>
<ListView
android:layout_width="match_parent"
android:layout_height="300dp"
android:id="@+id/data"></ListView>
</LinearLayout>
5:MainActivity的修改:
首先要说的是事务,通过事务,我们可以一次提交很多的数据,例如16万条的数据,只需要28秒钟,
然后如果要在保存之前进行查找是否有该数据,那么一定要建立索引,如果不建立索引,那么需要54分钟,建立了索引只需要2分钟14秒。
litepal的查找的语句是:
List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);
类似的:
List<News> newsList = DataSupport.select("title", "content")
.where("commentcount > ?", "0")
.order("publishdate desc").limit(10).offset(10)
.find(News.class); 创建索引的方法:
database.execSQL("create index account_index on account(uid)");
16万数据,从数据库中读取list,然后每一行转换为JSON写入文件中,
这两个操作的时间为读取list 需要35秒,将对象转换为JSON然后写入文件中,需要12秒
写入JSON文件的操作,如果使用一次写入然后替换的话,会比一个一个的写入慢5秒钟,
//如果使用替换的话,大概需要17秒钟,比一行一行的输出慢了5分钟,开始写入jsonFri Jul 01 11:54:48 GMT+08:00 2016结束吸入JSON:Fri Jul 01 11:55:05 GMT+08:00 2016 String s = JSON.toJSONString(all); String replace = s.replace("},{", "\n"); String replace1 = replace.replace("[{", "{"); String replaceDone = replace1.replace("]}", "}"); fileWriter.write(replaceDone);
package tech.androidstudio.dbtest;
import android.database.sqlite.SQLiteDatabase;
import android.os.Environment;
import android.os.Handler;
import android.os.Message;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.text.TextUtils;
import android.util.Log;
import android.view.View;
import com.alibaba.fastjson.JSON;
import org.litepal.crud.DataSupport;
import org.litepal.tablemanager.Connector;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
public class MainActivity extends AppCompatActivity {
private SQLiteDatabase database;
private InputStream mInputStream;
private Handler mhandler;
private FileWriter fileWriter;
private File newDBfile;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
database = Connector.getDatabase();
mInputStream = getResources().openRawResource(R.raw.ac_account);
mhandler = new Handler(){
@Override
public synchronized void handleMessage(Message msg) {
super.handleMessage(msg);
}
};
}
public void doImportInhandler(){
DataSupport.deleteAll(Account.class);
int i=0;
String sql;
InputStream inStream=null;
try {
Log.d("sql",new Date().toString()+"");
database.beginTransaction();
inStream = mInputStream;
BufferedReader reader=new BufferedReader(new InputStreamReader(inStream));
while (true){
sql=reader.readLine();
if(TextUtils.isEmpty(sql)){
break;
}
//自己创建文件,命名为ac_account,输入如下
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157314', null, null, null, 0, 1, '2016-05-21 01:58:46', '2016-05-21 01:58:34');
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157315', null, null, null, 0, 1, '2016-05-21 01:58:47', '2016-05-21 01:58:35');
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157316', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157317', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157318', null, null, null, 0, 1, '2016-05-21 01:58:49', '2016-05-21 01:58:37');
// INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157319', null, null, null, 0, 1, '2016-05-21 01:58:50', '2016-05-21 01:58:38');
sql=sql.replace("ac_account","account");
database.execSQL(sql);
i++;
// Log.d("sql",i+"");
}
database.setTransactionSuccessful();
database.endTransaction();
Log.d("sql",new Date().toString()+"");
}catch (Exception e){
e.printStackTrace();
try {
if(inStream!=null) {
inStream.close();
}
} catch (IOException e1) {
e1.printStackTrace();
}
}finally {
try {
if(inStream!=null) {
inStream.close();
}
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
public void doImport(View view) {
mhandler.post(new Runnable() {
@Override
public void run() {
doImportInhandler();
}
});
}
public void createJsonFile(View view) {
mhandler.post(new Runnable() {
@Override
public void run() {
try {
createJson();
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
private void createJson() throws IOException {
String startTime = new Date().toString();
Log.d("kodulf","开始:"+ startTime);
String externalCacheDirPath = getExternalCacheDir().getPath();
String externalStorageDirectoryPath = Environment.getExternalStorageDirectory().getPath();
// File newDBfile = new File(externalCacheDirPath+"/hello.db");
newDBfile = new File(externalStorageDirectoryPath+"/hello.db");
fileWriter = new FileWriter(newDBfile);
List<Account> all = new ArrayList<Account>();
database.beginTransaction();
try {
all = DataSupport.findAll(Account.class);
database.setTransactionSuccessful();
Log.d("kodulf","执行的时候成功了");
}catch (Exception e){
Log.d("kodulf","执行的时候出错了");
}
database.endTransaction();
Log.d("kodulf","开始:"+new Date().toString());
for (int i = 0; i < all.size(); i++) {
String s = JSON.toJSONString(all.get(i));
Log.d("kodulf", s);
fileWriter.write(s);
fileWriter.write("\n");
}
String endTime = new Date().toString();
Log.d("kodulf","结束:"+ endTime);
Log.d("sql","创建JSON开始时间="+startTime+" 结束时间"+endTime);
fileWriter.close();
}
public void startToImportJsonFiles(View view) {
try {
mInputStream = new FileInputStream(newDBfile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
Log.d("kodulf","开始执行新的从JSON格式导入的");
mhandler.post(new Runnable() {
@Override
public void run() {
doImportJSONInhandler();
}
});
}
private void doImportJSONInhandler() {
Date startDate = new Date();
String startTime = startDate.toString();
database.execSQL("create index account_index on account(uid)");
DataSupport.deleteAll(Account.class);
int i=0;
String sql;
InputStream inStream=null;
try {
Log.d("sql", startTime +"");
database.beginTransaction();
inStream = mInputStream;
BufferedReader reader=new BufferedReader(new InputStreamReader(inStream));
while (true){
sql=reader.readLine();
if(TextUtils.isEmpty(sql)){
break;
}
Account account = JSON.parseObject(sql, Account.class);
//————————————————方案一,直接使用save插入,如果数据为16万条的话,大概需要26秒钟——————————————————
//account.save();
//+++++++++++++++方案二,在插入之前首先进行查找,如果数据为16万条的话,需要的时间会很长很长+++++++++++++++++
// 大概需要54分钟,导入开始时间=Thu Jun 30 18:51:41 GMT+08:00 2016 结束时间Thu Jun 30 19:45:50 GMT+08:00 2016
//int uid = account.getUid();
//List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);
//if(accounts.size()==0)
// account.save();
//else
// Log.d("json",i+" 已经有了 account"+account.toString());
//+++++++++++++++++++++++++++++++++
//****************方案三,使用索引的方式,然后在查找和
//在方法最上面执行索引的操作:database.execSQL("create index account_index on account(uid)");
//大概需要2分18秒,比第二种方案块了很多倍,导入开始时间=Thu Jun 30 20:01:45 GMT+08:00 2016 结束时间Thu Jun 30 20:04:03 GMT+08:00 2016
int uid = account.getUid();
List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);
if(accounts.size()==0)
account.save();
else
Log.d("json",i+" 已经有了 account"+account.toString());
i++;
}
database.setTransactionSuccessful();
database.endTransaction();
Date endDate = new Date();
String endTime = endDate.toString();
Log.d("sql","导入开始时间="+startTime+" 结束时间"+endTime);
}catch (Exception e){
e.printStackTrace();
try {
if(inStream!=null) {
inStream.close();
}
} catch (IOException e1) {
e1.printStackTrace();
}
}finally {
try {
if(inStream!=null) {
inStream.close();
}
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
6:自己创建数据内容:
在res下面创建一个文件夹为raw。
自己创建文件,命名为ac_account,输入如下
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157314', null, null, null, 0, 1, '2016-05-21 01:58:46', '2016-05-21 01:58:34');
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157315', null, null, null, 0, 1, '2016-05-21 01:58:47', '2016-05-21 01:58:35');
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157316', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157317', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157318', null, null, null, 0, 1, '2016-05-21 01:58:49', '2016-05-21 01:58:37');
INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157319', null, null, null, 0, 1, '2016-05-21 01:58:50', '2016-05-21 01:58:38');
本文记录了使用LitePal框架在Android中批量导入大量数据的过程。通过在MyApplication中配置,利用事务处理16万条数据只需28秒。建立索引可以显著提升查找效率,未建索引需54分钟,而建索引后只需2分钟14秒。数据写入时,一次性写入JSON文件比逐条写入快5秒。数据源存放在res/raw下的ac_account文件中。
962

被折叠的 条评论
为什么被折叠?



