空中的鱼

升级需求1.为支持spark kerberos安全机制2.为满足impala trunc函数3.为解决impala import时同时query导致impala hang问题升级步骤参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade.html

loudera/en/documentation/core/latest/topics/installation_upgrade.html优先升级cloudera manager，再升级cdh1.准备工作： 统一集群root密码，需要运维帮忙操作下 agent自动重启关闭 事先下载好parcals包2.CM升级 登录cmserver安装的主机，执行命令： cat /etc/cloudera-scm-server/db.properties 备份CM数据： pg_

ata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/.`; do       export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:/logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/$jarfile"done  hadoop jar bigdata-mapreduce-0.0.1-SNAPSHOT.jar com.yeahmobi.bigdata.mapreduce.G

tiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297) at org.apache.thrift.transport.TSaslClientT

失败时自动选择kerberos安全认证。步骤：关闭防火墙，设置开机不启动防火墙sudo /etc/init.d/iptables statussudo /etc/init.d/iptables stop / sudo service iptables stopsudo chkconfig iptables off安装LDAPyum install db4 db4-utils db4-devel cyrus-sasl* krb5-server-ldap -yyum ins

impalad的机器安装haproxy yum install haproxy编辑/etc/haproxy/haproxy.cfg，参考global # To have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by ad

ion.ldap.url ldap://master-71:389 hive.server2.authentication.ldap.baseDN ou=ndpmedia,dc=yeahmobi,dc=com测试例子：https://github.com/firecodeman/Cloudera-Impala-Hive-JDBC-Example奇怪现象：http://community.cloudera.com/t5/CDH-Manual-Install

hive client直走hive的本地模式，没有经过hiveserver2，所以此种方式能访问所有的数据库，具有超级管理员权限；考虑使用beeline形式。登陆方式例如： beeline -u "jdbc:hive2://172.20.0.74:10000/data_system" -nhive -p111111（ldap账户及密码） 或者 beeline!connect jdbc:hive2://172.20.0.74:10000/da

ANT ALL ON URI 'hdfs://172.20.0.71:8020/user/bi' TO ROLE user_bi_all_role;解决之[b]问题二[/b]：账户bi运行mapreduce需要读取/user/hive/warehouse下的数据。解决：一般/user/hive/warehouse属于hive:hive，根据sentry要求，配置771权限。为了让bi账户对该目录有访问权限，借助aclhadoop fs -setfacl -R -m user:bi:r

，或者写入hbase数据 踩过的坑：在mapreduce的job创建过程中加入如下认证代码UserGroupInformation.setConfiguration(conf); UserGroupInformation.loginUserFromKeytab(conf.get("hbase.master.kerberos.principal"), conf.get("hbase.keytab.path")); a.若此处使用该节点bi账户的认

0/user/hive/.staging/job_1426073522130_3022/libjars/guava-11.0.2.jar(->/data7/yarn/nm/usercache/hive/filecache/84380/guava-11.0.2.jar) transitioned from INIT to LOCALIZED2015-03-26 07:41:00,391 INFO org.apache.hadoop.service.AbstractService: Servi

e.ShuffleSchedulerImpl: assigned 20 of 34 to spark-03:13562 to fetcher#102015-10-23 16:43:54,166 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.S

目的：为impala jdbc提供统一的接口，作用参照http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_proxy.html步骤：安装haproxy选择一台非impalad的机器安装haproxy yum install haproxy编辑/etc/ha...

目的：为解决kerberos安全机制下的impala，resin cache kerberos tgt maxrenewlife天失效问题。说明：impala启用LDAP后，会优先选择LDAP用户密码认证，当LDAP认证失败时自动选择kerberos安全认证。步骤：关闭防火墙，设置开机不启动防火墙sudo /etc/init.d/iptables statuss...

当前cdh版本为5.2.0，且通过cloudera manger来管理集群。选择sentry配置方式，file or db。file形式（sentry-provider.ini），存于hdfs上。选择group mapping方式，有HadoopGroupResourceAuthorizationProvider（正式环境中）、LocalGroupResourceAuthorizatio

cloudera manager hive- sevice wide - advanced-Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml hive.server2.authentication LDAP hive.server2.authenticat...

目标：给各个业务组提供不同用户及用户组，并有限制的访问hdfs路径，及hive数据库。前提：clouderacloudera managerkerberosldapsentry问题与解决：hive client直走hive的本地模式，没有经过hiveserver2，所以此种方式能访问所有的数据库，具有超级管理员权限；考虑使用beeline形式。登...

以账户bi为例[b]问题一[/b]：账户bi beeline ldap后，对于外联表需要外联/user/bi目录下的数据。解决：根据sentry文档，需要给/user/bi授权uri ALL权限。GRANT ALL ON URI 'hdfs://172.20.0.71:8020/user/bi' TO ROLE user_bi_all_role;解决之[b]问题二[/b...

环境：security cdh 5.2.0security hbase启动mapreduce的账户为hive或者others（非hbase），如下假设以bi账户为例 出发点：mapper/reduce程序中读取hbase中数据，或者写入hbase数据 踩过的坑：在mapreduce的job创建过程中加入如下认证代码UserGroupInformation...

yarn重启后，部分nm启动不了，报空指针问题2015-03-26 07:41:00,367 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://:8020/user/hive/.staging/job_142607352...

动态分区问题，如果数据量大或者当动态分区大甚至只有十几个时，会出现如下异常：2015-10-23 16:43:54,165 INFO [fetcher#10] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 20 of 34 to spark-03:13562 to fetcher#102...

mysqldump -hhost -uroot -ppasswd sentry > /tmp/sentry.sql create database sentry DEFAULT CHARACTER SET utf8; grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry'; flush PRIVILEG...

升级需求1.为支持spark kerberos安全机制2.为满足impala trunc函数3.为解决impala import时同时query导致impala hang问题升级步骤参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade...

参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_table_access_mapreduce.htmlhttps://github.com/cloudera/hcatalog-examples.git命令：for jarfile in `ls /logdata/CDH-5....

配置hive gateway机器Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate fa...

执行mapreduce的用户必须是可以访问hdfs相应目录和执行mapreduce的账户，例如hive。指定hive的节点kinit获取执行权限在mapreduce main代码中加入访问hbase的权限，例如：import java.io.IOException;import org.apache.hadoop.hbase.HBaseConfiguration;impor...

ES; source /tmp/sentry.sql mysql参数[code="java"][mysqld]#transaction-isolation=READ-COMMITTED# Disabling symbolic-links is recommended to prevent assorted security risks;# to do so, uncomment this line:# symbolic-links=0so