6.Sqoop Mysql-to-Hive MBeanTrustPermission/JDODataStoreException/GLOBALPRIVILEGEINDEX

使用Sqoop将Mysql里面的数据倒入到Hive报错。

sqoop import \
--connect jdbc:mysql://192.168.5.81:3306/dim?useSSL=false \
--username root --password root  --table ods_agent_da \
--hive-import  \
--hive-table dim.ods_agent_da     \
--target-dir ./ods_agent_da/20220424 \
--hive-partition-key pt      \
--hive-partition-value 20220424                 \
--delete-target-dir  \
--outdir /tmp/MysqlToHive     \
--hive-overwrite    \
--null-non-string '\\N' --null-string '\\N'   \
--columns 'id,create_time,update_time,,nationality,name,status'  \
-m 1

执行报下面三个错误。

1.access denied ("javax.management.MBeanTrustPermission" "register")

2.javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore

3.java.sql.BatchUpdateException: Duplicate entry 'SQL-admin-ROLE-All-admin-ROLE' for key 'GLOBALPRIVILEGEINDEX'

1.2022-04-24 17:11:17,006 main ERROR Could not register mbeans java.security.AccessControlException: 
access denied ("javax.management.MBeanTrustPermission" "register")

解决方法：JDK 里面增加： permission javax.management.MBeanTrustPermission "register";
vi   /data02/jdk1.8.0_261/jre/lib/security/java.policy

// Standard extensions get all permissions by default
grant codeBase "file:${{java.ext.dirs}}/*" {
        permission java.security.AllPermission;
};
// default permissions granted to all domains
grant {
        // Allows any thread to stop itself using the java.lang.Thread.stop()
        // method that takes no argument.
        // Note that this permission is granted by default only to remain
        // backwards compatible.
        // It is strongly recommended that you either remove this permission
        // from this policy file or further restrict it to code sources
        // that you specify, because Thread.stop() is potentially unsafe.
        // See the API specification of java.lang.Thread.stop() for more
        // information.
        permission java.lang.RuntimePermission "stopThread";

        // allows anyone to listen on dynamic ports
        permission java.net.SocketPermission "localhost:0", "listen";

        // "standard" properies that can be read by anyone

        permission java.util.PropertyPermission "java.version", "read";
        permission java.util.PropertyPermission "java.vendor", "read";
        permission java.util.PropertyPermission "java.vendor.url", "read";
        permission java.util.PropertyPermission "java.class.version", "read";
        permission java.util.PropertyPermission "os.name", "read";
        permission java.util.PropertyPermission "os.version", "read";
        permission java.util.PropertyPermission "os.arch", "read";
        permission java.util.PropertyPermission "file.separator", "read";
        permission java.util.PropertyPermission "path.separator", "read";
        permission java.util.PropertyPermission "line.separator", "read";

        permission java.util.PropertyPermission "java.specification.version", "read";
        permission java.util.PropertyPermission "java.specification.vendor", "read";
        permission java.util.PropertyPermission "java.specification.name", "read";

        permission java.util.PropertyPermission "java.vm.specification.version", "read";
        permission java.util.PropertyPermission "java.vm.specification.vendor", "read";
        permission java.util.PropertyPermission "java.vm.specification.name", "read";
        permission java.util.PropertyPermission "java.vm.version", "read";
        permission java.util.PropertyPermission "java.vm.vendor", "read";
        permission java.util.PropertyPermission "java.vm.name", "read";
        permission javax.management.MBeanTrustPermission "register";
};

2.javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
    at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:171)

解决方法：需要增加编码格式的设置：characterEncoding=UTF-8&amp
vi hive-site.xml --对于hive 元数据需要设置编码格式。
<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.5.81:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
</property>

3.sqoop 到导入数据时：
java.sql.BatchUpdateException: Duplicate entry 'SQL-admin-ROLE-All-admin-ROLE' for key 'GLOBALPRIVILEGEINDEX'
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

解决方法：新增参数。

vi hive-site.xml 
  <property>
        <name>datanucleus.fixedDataStore</name>
       <value>true</value>
   </property>
   <property>
        <name>datanecleus.autoCreateSchema</name>
       <value>false</value>
   </property>