CCBIWriter/Reader的研究感触

本文分享了作者将CCB(version4)到CCBI的转换器从Objective-C移植到Java的经验,包括二进制文件处理、基本类型写入、Int编码、Java与Objective-C类型差异、字符编码及优化方案。

2013-3-31 14:38:42

本人已经将CCB(version 4)到CCBI的转换器从Objective-C版本移植成了java版本,积累了一些经验,希望在这里记录 :

1. 二进制文件

2. 基本类型的写入方法

3. Int编码方法

4. Java的基本类型和Objective-C的差异

5. 字符编码的问题

6. 提高效率的优化方案

libs/jline-2.12.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-format-structures-1.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-column-1.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/audience-annotations-0.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-encoding-1.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-jackson-1.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-pool-1.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/zstd-jni-1.4.9-1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/javax.annotation-api-1.3.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/snappy-java-1.1.8.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/parquet-common-1.12.0.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/bigdata/server/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar"],"parent":{"URLs":["file:/bigdata/server/datax/lib/commons-configuration-1.10.jar","file:/bigdata/server/datax/lib/hamcrest-core-1.3.jar","file:/bigdata/server/datax/lib/httpclient-4.5.13.jar","file:/bigdata/server/datax/lib/groovy-all-2.1.9.jar","file:/bigdata/server/datax/lib/commons-logging-1.1.1.jar","file:/bigdata/server/datax/lib/commons-lang3-3.3.2.jar","file:/bigdata/server/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/bigdata/server/datax/lib/logback-core-1.0.13.jar","file:/bigdata/server/datax/lib/janino-2.5.16.jar","file:/bigdata/server/datax/lib/commons-math3-3.1.1.jar","file:/bigdata/server/datax/lib/fluent-hc-4.5.jar","file:/bigdata/server/datax/lib/fastjson2-2.0.23.jar","file:/bigdata/server/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/bigdata/server/datax/lib/slf4j-api-1.7.10.jar","file:/bigdata/server/datax/lib/commons-cli-1.2.jar","file:/bigdata/server/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/bigdata/server/datax/lib/logback-classic-1.0.13.jar","file:/bigdata/server/datax/lib/commons-codec-1.11.jar","file:/bigdata/server/datax/lib/commons-io-2.4.jar","file:/bigdata/server/datax/lib/httpcore-4.4.13.jar","file:/bigdata/server/datax/lib/commons-lang-2.6.jar","file:/bigdata/server/datax/lib/commons-collections-3.2.1.jar","file:/bigdata/server/datax/lib/commons-beanutils-1.9.2.jar","file:/bigdata/server/datax/"],"parent":{"URLs":["file:/root/software/jdk_1.8/jre/lib/ext/cldrdata.jar","file:/root/software/jdk_1.8/jre/lib/ext/jfxrt.jar","file:/root/software/jdk_1.8/jre/lib/ext/jaccess.jar","file:/root/software/jdk_1.8/jre/lib/ext/dnsns.jar","file:/root/software/jdk_1.8/jre/lib/ext/nashorn.jar","file:/root/software/jdk_1.8/jre/lib/ext/sunpkcs11.jar","file:/root/software/jdk_1.8/jre/lib/ext/sunjce_provider.jar","file:/root/software/jdk_1.8/jre/lib/ext/zipfs.jar","file:/root/software/jdk_1.8/jre/lib/ext/sunec.jar","file:/root/software/jdk_1.8/jre/lib/ext/localedata.jar"]}}},"finalParameters":[]} 2025-11-07 17:39:33.225 [job-0] INFO HdfsReader$Job - init() ok and end... 2025-11-07 17:39:33.549 [job-0] ERROR RetryUtil - Exception when calling callable, 异常Msg:Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES) com.alibaba.datax.common.exception.DataXException: Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES) at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.RdbmsException.asConnException(RdbmsException.java:21) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:397) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:387) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil.access$000(DBUtil.java:22) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:322) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:319) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil$Retry.call(RetryUtil.java:164) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:111) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30) [datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:303) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.util.JdbcConnectionFactory.getConnecttion(JdbcConnectionFactory.java:27) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:106) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:147) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:36) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:42) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Job.init(MysqlWriter.java:31) [mysqlwriter-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na] 2025-11-07 17:39:34.550 [job-0] ERROR RetryUtil - Exception when calling callable, 即将尝试执行第1次重试.本次重试计划等待[1000]ms,实际等待[1000]ms, 异常Msg:[Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES)] 2025-11-07 17:39:36.565 [job-0] ERROR RetryUtil - Exception when calling callable, 即将尝试执行第2次重试.本次重试计划等待[2000]ms,实际等待[2000]ms, 异常Msg:[Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES)] 2025-11-07 17:39:40.581 [job-0] ERROR RetryUtil - Exception when calling callable, 即将尝试执行第3次重试.本次重试计划等待[4000]ms,实际等待[4000]ms, 异常Msg:[Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES)] 2025-11-07 17:39:48.595 [job-0] ERROR RetryUtil - Exception when calling callable, 即将尝试执行第4次重试.本次重试计划等待[8000]ms,实际等待[8001]ms, 异常Msg:[Code:[MYSQLErrCode-01], Description:[数据库用户名或者密码错误,请检查填写的账号密码或者联系DBA确认账号和密码是否正确]. - 该数据库用户名为:root 具体错误信息为:java.sql.SQLException: Access denied for user 'root'@'Hadoop02' (using password: YES)]
最新发布
11-08
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值