Spark 提交.jar程序遇到的问题

1、缺少驱动类包

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)

        ... 92 more

Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

        at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)

        at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)

        ... 94 more

解决方法：spark-submit  --jar  /home /dmp/mysql/lib/mysql-connector.jar  z 指定目录就可以；

2、start-thriftserver.sh 默认启动则--total-executor-cores=所有的；因此从外秒提交jar包没资源，因此需要给-start-thriftserver.sh 少点核数，启动jar 包就可以运行了；（报错如下）

17/10/17 18:50:12 INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_0 tasks to pool default

17/10/17 18:50:27 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

3、从datanode 提交jar包；指定mysql -connector.jar包；把hive.xml 中的mysql  127.0.0.1 改为服务器的ip;

4、bug

17/10/17 18:50:07 INFO metastore.HiveMetaStore: 0: create_database: Database(name:default, description:default database, locationUri:file:/root/test/spark-warehouse, parameters:{})

17/10/17 18:50:07 INFO HiveMetaStore.audit: ugi=root    ip=unknown-ip-addr      cmd=create_database: Database(name:default, description:default database, locationUri:file:/root/test/spark-warehouse, parameters:{})

17/10/17 18:50:07 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)

        at com.sun.proxy.$Proxy22.create_database(Unknown Source)

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:644)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)

        at com.sun.proxy.$Proxy23.createDatabase(Unknown Source)

        at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:306)