Android 从文件中批量导入数据,个人记录而已

本文记录了使用LitePal框架在Android中批量导入大量数据的过程。通过在MyApplication中配置,利用事务处理16万条数据只需28秒。建立索引可以显著提升查找效率,未建索引需54分钟,而建索引后只需2分钟14秒。数据写入时,一次性写入JSON文件比逐条写入快5秒。数据源存放在res/raw下的ac_account文件中。

使用litePal来做为数据库的框架:

0:首先导入litepal 的依赖和fastjson的依赖

dependencies {
    compile fileTree(include: ['*.jar'], dir: 'libs')
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:23.4.0'
    compile 'org.litepal.android:core:1.3.2'
    compile 'com.alibaba:fastjson:1.2.13'
}

1:创建一个MyApplication,

package tech.androidstudio.dbtest;

import org.litepal.LitePalApplication;

/**
 * Created by Kodulf on 2016/6/30.
 */
public class MyApplication extends LitePalApplication {
    @Override
    public void onCreate() {
        super.onCreate();
    }
}


然后修改清单文件:android:name=".MyApplication"

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="tech.androidstudio.dbtest">
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"></uses-permission>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"></uses-permission>
    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:name=".MyApplication"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

</manifest>

2:在main的文件夹下面创建一个assets的文件夹,然后创建一个litepal.xml的文件,


里面的list 是我们下面的创建的类Account,就是会映射为一个表

<?xml version="1.0" encoding="utf-8"?>
<litepal>
    <dbname value="demo" ></dbname>

    <version value="1" ></version>

    <list>
        <mapping class="tech.androidstudio.dbtest.Account"></mapping>
    </list>
</litepal>

3:创建Account的类:记住一定要包含一个空的参数的构造方法,因为下面会用到FastJson转换的时候用到,不然会报错的。

package tech.androidstudio.dbtest;

import org.litepal.crud.DataSupport;

import java.util.Date;

/**
 * Created by Kodulf on 2016/6/30.
 */
public class Account extends DataSupport {
//(uid, acc_no, realname, mobile, attribute, status, cdate, edate)
// VALUES ('20009', null, null, '18329160075', 1, 1, '2012-08-30 15:12:35', '2014-02-28 02:54:25');
    private int uid;
    private String acc_no;
    private String  realname;
    private String mobile;
    private int attribute;
    private int status;
    private Date cdate;
    private Date edate;

    public Account() {
    }

    public Account(int uid, String acc_no, String realname, String mobile, int attribute, int status, Date cdate, Date edate) {
        this.uid = uid;
        this.acc_no = acc_no;
        this.realname = realname;
        this.mobile = mobile;
        this.attribute = attribute;
        this.status = status;
        this.cdate = cdate;
        this.edate = edate;
    }

    public int getUid() {
        return uid;
    }

    public void setUid(int uid) {
        this.uid = uid;
    }

    public String getAcc_no() {
        return acc_no;
    }

    public void setAcc_no(String acc_no) {
        this.acc_no = acc_no;
    }

    public String getRealname() {
        return realname;
    }

    public void setRealname(String realname) {
        this.realname = realname;
    }

    public String getMobile() {
        return mobile;
    }

    public void setMobile(String mobile) {
        this.mobile = mobile;
    }

    public int getAttribute() {
        return attribute;
    }

    public void setAttribute(int attribute) {
        this.attribute = attribute;
    }

    public int getStatus() {
        return status;
    }

    public void setStatus(int status) {
        this.status = status;
    }

    public Date getCdate() {
        return cdate;
    }

    public void setCdate(Date cdate) {
        this.cdate = cdate;
    }

    public Date getEdate() {
        return edate;
    }

    public void setEdate(Date edate) {
        this.edate = edate;
    }
}

4:修改activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    android:orientation="vertical"
    tools:context="tech.androidstudio.dbtest.MainActivity">

    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="开始加载"
        android:onClick="doImport"/>
    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="生成json的文件"
        android:onClick="createJsonFile"/>
    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="开始加载更新后的json文件"
        android:onClick="startToImportJsonFiles"/>
    <ListView
        android:layout_width="match_parent"
        android:layout_height="300dp"
        android:id="@+id/data"></ListView>

</LinearLayout>


5:MainActivity的修改:

首先要说的是事务,通过事务,我们可以一次提交很多的数据,例如16万条的数据,只需要28秒钟,

然后如果要在保存之前进行查找是否有该数据,那么一定要建立索引,如果不建立索引,那么需要54分钟,建立了索引只需要2分钟14秒。


litepal的查找的语句是:

List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);

类似的:

List<News> newsList = DataSupport.select("title", "content")  
        .where("commentcount > ?", "0")  
        .order("publishdate desc").limit(10).offset(10)  
        .find(News.class);  

创建索引的方法:

database.execSQL("create index account_index on account(uid)");

16万数据,从数据库中读取list,然后每一行转换为JSON写入文件中,
这两个操作的时间为读取list 需要35秒,将对象转换为JSON然后写入文件中,需要12秒

写入JSON文件的操作,如果使用一次写入然后替换的话,会比一个一个的写入慢5秒钟,

//如果使用替换的话,大概需要17秒钟,比一行一行的输出慢了5分钟,开始写入jsonFri Jul 01 11:54:48 GMT+08:00 2016结束吸入JSON:Fri Jul 01 11:55:05 GMT+08:00 2016
String s = JSON.toJSONString(all);
String replace = s.replace("},{", "\n");
String replace1 = replace.replace("[{", "{");
String replaceDone = replace1.replace("]}", "}");
fileWriter.write(replaceDone);






package tech.androidstudio.dbtest;

import android.database.sqlite.SQLiteDatabase;
import android.os.Environment;
import android.os.Handler;
import android.os.Message;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.text.TextUtils;
import android.util.Log;
import android.view.View;

import com.alibaba.fastjson.JSON;

import org.litepal.crud.DataSupport;
import org.litepal.tablemanager.Connector;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

public class MainActivity extends AppCompatActivity {

    private SQLiteDatabase database;
    private InputStream mInputStream;
    private Handler mhandler;
    private FileWriter fileWriter;
    private File newDBfile;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        database = Connector.getDatabase();
        mInputStream = getResources().openRawResource(R.raw.ac_account);
        mhandler = new Handler(){
            @Override
            public synchronized void handleMessage(Message msg) {
                super.handleMessage(msg);
            }
        };

    }

    public void doImportInhandler(){
        DataSupport.deleteAll(Account.class);
        int i=0;

        String sql;
        InputStream inStream=null;
        try {
            Log.d("sql",new Date().toString()+"");
            database.beginTransaction();
            inStream = mInputStream;
            BufferedReader reader=new BufferedReader(new InputStreamReader(inStream));
            while (true){
                sql=reader.readLine();
                if(TextUtils.isEmpty(sql)){
                    break;
                }
                //自己创建文件,命名为ac_account,输入如下
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157314', null, null, null, 0, 1, '2016-05-21 01:58:46', '2016-05-21 01:58:34');
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157315', null, null, null, 0, 1, '2016-05-21 01:58:47', '2016-05-21 01:58:35');
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157316', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157317', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157318', null, null, null, 0, 1, '2016-05-21 01:58:49', '2016-05-21 01:58:37');
//                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157319', null, null, null, 0, 1, '2016-05-21 01:58:50', '2016-05-21 01:58:38');

                sql=sql.replace("ac_account","account");
                database.execSQL(sql);
                i++;
//                Log.d("sql",i+"");
            }
            database.setTransactionSuccessful();
            database.endTransaction();
            Log.d("sql",new Date().toString()+"");
        }catch (Exception e){
            e.printStackTrace();
            try {
                if(inStream!=null) {
                    inStream.close();
                }
            } catch (IOException e1) {
                e1.printStackTrace();
            }
        }finally {
            try {
                if(inStream!=null) {
                    inStream.close();
                }
            } catch (IOException e1) {
                e1.printStackTrace();
            }
        }
    }
    public void doImport(View view) {
        mhandler.post(new Runnable() {
            @Override
            public void run() {
                doImportInhandler();
            }
        });
    }


    public void createJsonFile(View view) {
        mhandler.post(new Runnable() {
            @Override
            public void run() {
                try {
                    createJson();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        });
    }

    private void createJson() throws IOException {
        String startTime = new Date().toString();
        Log.d("kodulf","开始:"+ startTime);
        String externalCacheDirPath = getExternalCacheDir().getPath();
        String externalStorageDirectoryPath = Environment.getExternalStorageDirectory().getPath();
//        File newDBfile = new File(externalCacheDirPath+"/hello.db");
        newDBfile = new File(externalStorageDirectoryPath+"/hello.db");
        fileWriter = new FileWriter(newDBfile);
        List<Account> all = new ArrayList<Account>();
        database.beginTransaction();
        try {
             all = DataSupport.findAll(Account.class);
            database.setTransactionSuccessful();
            Log.d("kodulf","执行的时候成功了");
        }catch (Exception e){
            Log.d("kodulf","执行的时候出错了");
        }
        database.endTransaction();

        Log.d("kodulf","开始:"+new Date().toString());
        for (int i = 0; i < all.size(); i++) {
            String s = JSON.toJSONString(all.get(i));
            Log.d("kodulf", s);
            fileWriter.write(s);
            fileWriter.write("\n");
        }
        String endTime = new Date().toString();
        Log.d("kodulf","结束:"+ endTime);
        Log.d("sql","创建JSON开始时间="+startTime+" 结束时间"+endTime);
        fileWriter.close();
    }

    public void startToImportJsonFiles(View view) {
        try {
            mInputStream = new FileInputStream(newDBfile);
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }
        Log.d("kodulf","开始执行新的从JSON格式导入的");
        mhandler.post(new Runnable() {
            @Override
            public void run() {
                doImportJSONInhandler();
            }
        });
    }

private void doImportJSONInhandler() {
        Date startDate = new Date();
        String startTime = startDate.toString();

        database.execSQL("create index account_index on account(uid)");
        DataSupport.deleteAll(Account.class);
        int i=0;

        String sql;
        InputStream inStream=null;
        try {

            Log.d("sql", startTime +"");
            database.beginTransaction();
            inStream = mInputStream;
            BufferedReader reader=new BufferedReader(new InputStreamReader(inStream));
            while (true){
                sql=reader.readLine();

                if(TextUtils.isEmpty(sql)){
                    break;
                }

                Account account = JSON.parseObject(sql, Account.class);
                //————————————————方案一,直接使用save插入,如果数据为16万条的话,大概需要26秒钟——————————————————
                //account.save();

                //+++++++++++++++方案二,在插入之前首先进行查找,如果数据为16万条的话,需要的时间会很长很长+++++++++++++++++
                // 大概需要54分钟,导入开始时间=Thu Jun 30 18:51:41 GMT+08:00 2016 结束时间Thu Jun 30 19:45:50 GMT+08:00 2016
                //int uid = account.getUid();
                //List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);
                //if(accounts.size()==0)
                //    account.save();
                //else
                //   Log.d("json",i+" 已经有了 account"+account.toString());
                //+++++++++++++++++++++++++++++++++

                //****************方案三,使用索引的方式,然后在查找和
                //在方法最上面执行索引的操作:database.execSQL("create index account_index on account(uid)");
                //大概需要2分18秒,比第二种方案块了很多倍,导入开始时间=Thu Jun 30 20:01:45 GMT+08:00 2016 结束时间Thu Jun 30 20:04:03 GMT+08:00 2016
                int uid = account.getUid();
                List<Account> accounts = DataSupport.where("uid=?", String.valueOf(uid)).find(Account.class);
                if(accounts.size()==0)
                    account.save();
                else
                    Log.d("json",i+" 已经有了 account"+account.toString());
                i++;

            }
            database.setTransactionSuccessful();
            database.endTransaction();
            Date endDate = new Date();
            String endTime = endDate.toString();

            Log.d("sql","导入开始时间="+startTime+" 结束时间"+endTime);

        }catch (Exception e){
            e.printStackTrace();
            try {
                if(inStream!=null) {
                    inStream.close();
                }
            } catch (IOException e1) {
                e1.printStackTrace();
            }
        }finally {
            try {
                if(inStream!=null) {
                    inStream.close();
                }
            } catch (IOException e1) {
                e1.printStackTrace();
            }
        }
    }
}




6:自己创建数据内容:

在res下面创建一个文件夹为raw。

自己创建文件,命名为ac_account,输入如下

                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157314', null, null, null, 0, 1, '2016-05-21 01:58:46', '2016-05-21 01:58:34');
                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157315', null, null, null, 0, 1, '2016-05-21 01:58:47', '2016-05-21 01:58:35');
                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157316', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157317', null, null, null, 0, 1, '2016-05-21 01:58:48', '2016-05-21 01:58:36');
                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157318', null, null, null, 0, 1, '2016-05-21 01:58:49', '2016-05-21 01:58:37');
                INSERT INTO ac_account (uid, acc_no, realname, mobile, attribute, status, cdate, edate) VALUES ('157319', null, null, null, 0, 1, '2016-05-21 01:58:50', '2016-05-21 01:58:38');





评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值