gdolphinw的专栏

原创 mac安装kafka

bookdeMacBook-Air:~ book$ brew install kafkaUpdating Homebrew...==> Auto-updated Homebrew!Updated 1 tap (homebrew/core).==> New Formulaebcoin ...

原创 C语言字符学习总结

##C语言字符学习总结void main(){ char szStr="A"; //错误的赋值 字符串不能赋值给字符 字符串末尾有结束符\0 [Warning] initialization makes integer from pointer without a cast printf("szStr=[%c]\n",szStr); //打印的不是字符A，说明赋值失败 ...

原创 C语言const修饰的常量如何修改

##C语言const修饰的常量如何修改。void main(){ const int cnum=10; printf("cnum=%d\n",cnum); //cnum=20; 直接修改会报错 *(int*)(&cnum)=30; //可以间接的修改 printf("cnum=%d\n",cnum); }...

原创 python操作Excel

工作需要，列出目录下所有Excel的所有Sheet的名字。目录中有一部分xls文件，一部分xlsx文件。openpyxl不支持读取xls文件，需要先转换成xlsx文件，再操作。from openpyxl import load_workbookimport osimport fnmatchimport win32com.clientfor root, dirs, files i...

原创 ModuleNotFoundError: No module named MySQLdb

python报错：ModuleNotFoundError: No module named ‘MySQLdb’处理办法1，增加 pymysql.install_as_MySQLdb()import pandas as pdfrom sqlalchemy import create_engineimport pymysqlpymysql.install_as_MySQLdb()2...

原创 Spark SQL本地开发调试亲测有效

无hive环境直接运行，会在项目目录生成metastore_db目录，data目录有hive环境,把hive-site.xml放到resources目录直接读取hive进行操作。/* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agree...

原创 Storm实现单词个数统计

Java代码import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.StormSubmitter;import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task...

原创 docker操作

yum install dokcersystemctl start dockerdocker ps -a

原创 oozie相关操作

HUE oozie创建Workflow和Scheduler创建WorkflowHUE查看Jobs通过oozie查看jobs#oozie查看log信息[dev@cdh1 oozie]$ oozie job -oozie http://cdh1:11000/oozie -info 0000063-190620162727859-oozie...

原创 CentOS通过PID找程序路径

#查PIDps -ef|grep rediscd /proc/PIDll or ls -l

原创 CentOS以rpm方式安装JDK8

下载JDKhttps://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.htmldk-8u121-linux-x64.rpmrpm -ivhdk-8u121-linux-x64.rpm

原创 PostgreSQL 11二进制包安装步骤

下载安装包，本文以二进制包方式安装https://www.postgresql.org/download/Binary packagesRed Hatfamily Linux (includingCentOS/Fedora/Scientific/Oraclevariants)选择适合的操作系统，会生成相对应的安装命令。Select version: 11 ...

原创 安装CDH5.14.2步骤

CDH 5 and Cloudera Manager 5 Requirements and Supported Versions可以查看对JDK、数据库版本的要求https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.htmlhttps://blo...

原创 单机Hbase安装

[root@bogon hbase-2.1.4]#wget http://mirror.bit.edu.cn/apache/hbase/2.1.4/hbase-2.1.4-bin.tar.gz[root@bogon hbase-2.1.4]#tar -zvxf hbase-2.1.4-bin.tar.gz[root@bogon hbase]# rpm -ivh jdk-8u211-lin...

原创 CentOS安装Python3.57

#安装python3yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel libffi-devel gcc makehttps://www.python.org/ftp/python/3.5.7/Python-3.5.7.tgztar -zv...

原创 Gitlab安装用rpm方式及版本升级

#CentOS7安装Gitlab#安装基础包yum -y install curl policycoreutils openssh-server openssh-clients#启动sshdsystemctl enable sshdsystemctl start sshd#安装postfixyum -y install postfixsystemctl enable po...

原创 docker环境安装Giblab

#docker pull gitlab/gitlab-ce#创建本地目录mkdir -p /opt/gitlab/etcmkdir -p /opt/gitlab/logmkdir -p /opt/gitlab/datadocker run --detach --publish 8443:443 --publish 8001:80 --name gitlab --r...

原创 CentOS安装管理Nginx

[root@VM_0_10_centos ~]#yum install -y nginx[root@VM_0_10_centos ~]# systemctl start nginx.serviceJob for nginx.service failed because the control process exited with error code. See "systemctl s...

原创 CentOS环境docker安装mysql并上传sql文件

#安装docker[root@VM_0_10_centos ~]# yum install docker[root@VM_0_10_centos ~]# docker psCannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?[root@VM_...

原创 python虚拟环境安装virtualenv

yum install python-virtualenvvirtualenv --python=/usr/bin/python2.7 env2.7[root@VM_0_10_centos env2.7]# pwd/root/gw/python/env2.7[root@VM_0_10_centos env2.7]# source bin/activate(env2.7) [root@...

原创 CentsOS防火墙开端口

vim /etc/sysconfig/iptables-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT-A INPUT -p tcp -m state --state NEW -m tcp --dport 1521 -j ACCEPTsystemctl restart iptables.service...

原创 单机Kafka安装

wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz ps -ef|grep zookeeper|grep -v grep|awk '{print $2}' | xargs kill -9ps -ef|grep kafka|grep -v grep|awk '{print $2}'...

原创 Linux批量杀进程

ps -ef|grep zookeeper|grep -v grep|awk '{print $2}' | xargs kill -9

转载 Django的MTV模式

原创 confluence utf8_general_ci

问题：Collation error The database collation 'utf8_general_ci' is not supported by Confluence. You need to use 'utf8_bin'.解决方法：create database confluence character set utf8 collate utf8_bin;grant all...

原创 pyhton爬取网页数据入库到oracle

from __future__ import print_functionimport requests, re, json, time, osimport randomfrom bs4 import BeautifulSoupimport pandas as pdimport cx_Oracleclass Cwzb(object): def __init__(self)...

原创 mapreduce

[dev@cdh1 python]$ vim genWordCount.pyimport randomlist =[]lineNum=0with open("wordcount.txt","a") as w: for i in range(5000000): apply = random.randint(3, 10) word='' ...

原创 hdfs fsck

fsck(file system check) ,用于检查HDFS上文件和目录的健康状态、获取文件的block块信息和位置信息等。#hdfs dfs -ls /user/gewei/#hdfs dfs -rm /user/gewei/houseApplyTest.csvhadoop fs -put houseApplyTest.csv hdfs:/user/gewei[dev@cd...

原创 find命令过滤权限不够的信息

find / -name filename 2>/dev/null

原创 无网络环境安装python模块

#已安装的模块名称导出到文件pip freeze >requirements.txt#导出已安装的模块到文件夹，指定模块名称或导出所有已安装的模块pip3 download -d D:\python\pip\package elasticsearchpip3 download -d D:\python\pip\package -r requirements.txt...

原创 Kibana ElasticSearch

#kibana 从ElasticSearch中查数据GET ind/ty/_search{ "query": { "match_all": {} }}#kibana 向ElasticSearch生成数据POST ind/ty{ "date" : "2017-09-13", "source" : "优分数据",

原创 htop安装

#CentOS6rpm -ivh https://mirrors.tuna.tsinghua.edu.cn/epel/6/x86_64/epel-release-6-8.noarch.rpmrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPELyum install htop -y#CentOS7yum install epel-relea...

原创 CentOS以yum方式安装jdk

#安装前检查是否已经安装了open-jdkrpm -qa |grep java rpm -qa |grep jdk rpm -qa |grep gcj#批量卸载rpm -qa | grep java | xargs rpm -e –nodeps #yum list java*yum list java-1.8*#安装yum install java-1.8.0-ope...

原创 数据仓库分层设计

根据目前的业务需求，数据仓库分为四层ODS：Operation Data Store原始数据，和源系统数据是同构数据DWD(数据清洗/DWI) data warehouse detail数据明细详情，去除空值，脏数据，超过极限范围的面向主题组织的数据，建设的最细粒度汇总层。DWS(宽表-用户行为，轻度聚合) data warehouse service 轻度聚合对DWD...

原创 CentOS查看端口命令

netstat -naltp |grep 10000-a或--all：显示所有连线中的Socket； -A<网络类型>或--<网络类型>：列出该网络类型连线中的相关地址； -c或--continuous：持续列出网络状态； -C或--cache：显示路由器配置的快取信息； -e或--extend：显示网络其他相关信息； -F或--fib：显示FIB； ...

原创 启动hiveserver2

hive --service hiveserver2netstat -ntulp |grep 10000 

原创 CentOS查看CPU信息

#查看 CPU 物理个数　　grep 'physical id' /proc/cpuinfo | sort -u | wc -l#查看 CPU 核心数量　　grep 'core id' /proc/cpuinfo | sort -u | wc -l#查看 CPU 线程数　　grep 'processor' /proc/cpuinfo | sort -u | wc -l#查看 C...

原创 查看centOS版本

cat /etc/redhat-release 

原创 python访问hdfs

pip install hdfspython 读取hdfs目录或文件import hdfsclient =hdfs.Client("http://10.10.1.4:50070")fileDir="/user/hive/warehouse/house.db/dm_house/dt=201800909"try: status=client.status(fileDir,F...

原创 shell调用sqoop

Mysql 数据导入Hive自动创建hive表 #!/bin/bashsource /etc/profilenum=0list="table1 table2 table3"for i in $list; do echo "$sum" echo "$i" echo "sqoop开始批量导入......" sqoop import --connect jdbc...