<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[dazheng]]></title><description><![CDATA[数据产生价值]]></description><link>https://blog.youkuaiyun.com/dazheng</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; dazheng]]></copyright><item><title><![CDATA[ETL SparkSQL实现]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/105370358</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/105370358</guid><author>dazheng</author><pubDate>Tue, 07 Apr 2020 18:09:00 +0800</pubDate><description><![CDATA[SparkETL
SparkETL主要用SQL方式实现数据仓库ETL，并保持spark的原生多功能、灵活性。采用java对spark功能进行简单封装，对于数据源、目标都是关系型数据库的，从数据抽取、转换、加载完全采用SQL方式，对于SQL不满足的场景，再用spark相关功能实现。
SparkETL是ETL的一个参考实现，实际使用时，需要根据业务需要及模型设计在此基础上增加、修改。
实现背景
Spa...]]></description><category></category></item><item><title><![CDATA[数据湖介绍]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/73499510</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/73499510</guid><author>dazheng</author><pubDate>Tue, 20 Jun 2017 16:02:21 +0800</pubDate><description><![CDATA[数据湖介绍数据湖概念是2011年提出来的,数据湖是数据仓库的补充，是为了解决数据仓库漫长的开发周期，高昂的成本，细节数据丢失等问题出现的。 
数据湖大多是相对于传统基于RDBMS系统的数据仓库，而从2011年前后，也就是数据湖概念出现的时候，很多数据仓库逐渐迁移到以Hadoop为基础的技术栈上，而且除了结构化数据，半结构化、非结构数据也逐渐的存储到数据仓库中，并提供此类服务]]></description><category></category></item><item><title><![CDATA[Go连接Hive]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/53043946</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/53043946</guid><author>dazheng</author><pubDate>Sat, 05 Nov 2016 10:37:46 +0800</pubDate><description><![CDATA[经过两天多的资料查找，反复测试从网上找到的多种方案，加上阿里云售后的支持，终于搞定Go语言连接Apache Hive的问题。 
测试环境，阿里云E-MapReduce 2.1 中的Hive， Golang 1.7，开发环境win10。 
在阿里云hive-site.xml中需要添加： <property>
    <name>hive.server2.authentication</name>]]></description><category></category></item><item><title><![CDATA[Go连接Kafka]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/52995260</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/52995260</guid><author>dazheng</author><pubDate>Tue, 01 Nov 2016 13:59:39 +0800</pubDate><description><![CDATA[golang连接kafka的例子好少，而支持offset追踪者更少，但也是基础的要求。”github.com/bsm/sarama-cluster”结合“github.com/Shopify/sarama”满足了此要求。package main
import (
    "fmt"
    "os"
    "strings"
    "time"    "github.com/Shopify/sa]]></description><category></category></item><item><title><![CDATA[Go语言圣经（中文版）]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/50606930</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/50606930</guid><author>dazheng</author><pubDate>Fri, 29 Jan 2016 14:12:20 +0800</pubDate><description><![CDATA[http://golang-china.github.io/gopl-zh/index.html]]></description><category></category></item><item><title><![CDATA[From scikit-learn to Spark ML]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/50164175</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/50164175</guid><author>dazheng</author><pubDate>Thu, 03 Dec 2015 17:50:42 +0800</pubDate><description><![CDATA[From scikit-learn to Spark ML
Yoann Benoit




PartagerTweeter+
 1E-mail


Dans un récent billet de blog de Databricks et Olivier Girardot, From
 Pandas to Apache Spark’s DataFrame, les aut]]></description><category></category></item><item><title><![CDATA[Pandarize Your Spark Dataframes]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/50161245</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/50161245</guid><author>dazheng</author><pubDate>Thu, 03 Dec 2015 15:16:41 +0800</pubDate><description><![CDATA[DataFrames are a great abstraction for working with structured and semi-structured data. They are basically a collection of rows, organized into named columns. Think of relational database tables: DataFrames are very similar and allow you to do similar ope]]></description><category></category></item><item><title><![CDATA[From Pandas to Apache Spark’s Dataframe]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/50161065</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/50161065</guid><author>dazheng</author><pubDate>Thu, 03 Dec 2015 15:09:30 +0800</pubDate><description><![CDATA[With the introduction in Spark 1.4 of Window operations, you can finally port pretty much any relevant piece of Pandas’ DataFrame computation to Apache Spark parallel computation framework using Spark SQL’s DataFrame. If you’re not yet familiar withSpark’s]]></description><category></category></item><item><title><![CDATA[读《人类简史》有感]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/49746059</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/49746059</guid><author>dazheng</author><pubDate>Mon, 09 Nov 2015 22:33:57 +0800</pubDate><description><![CDATA[读了尤瓦尔.赫拉利(Yuval Noah Harari)的《人类简史 从动物到上帝》，又在昨天下午见了一位对政治极感兴趣的一位老先生，不由得想到了人类未来及共产主义。
在书中，赫拉利先生边传述人类的历史，边对其评价，特别提到人类农业革命，是历史的一个退步，这个观点跟斯塔夫里阿诺斯先生的《全球通史 从史前史到21世纪》观点一样。除此之外，两位先生都在书中对科技进步对人类的影响都是非常的赞赏，读到这些地]]></description><category></category></item><item><title><![CDATA[从机器学习谈起]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/49738749</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/49738749</guid><author>dazheng</author><pubDate>Mon, 09 Nov 2015 15:08:13 +0800</pubDate><description><![CDATA[从机器学习谈起



　　在本篇文章中，我将对机器学习做个概要的介绍。本文的目的是能让即便完全不了解机器学习的人也能了解机器学习，并且上手相关的实践。这篇文档也算是EasyPR开发的番外篇，从这里开始，必须对机器学习了解才能进一步介绍EasyPR的内核。当然，本文也面对一般读者，不会对阅读有相关的前提要求。
　　在进入正题前，我想读者心中可能会有一个疑惑：机器学习有什么重要性，以至于要阅]]></description><category></category></item><item><title><![CDATA[数据标准化的几种方法]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/49738007</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/49738007</guid><author>dazheng</author><pubDate>Mon, 09 Nov 2015 14:24:45 +0800</pubDate><description><![CDATA[常见的数据归一化的方法有：
min-max标准化(Min-max normalization)、log函数转换、atan函数转换、z-score 标准化(zero-mean normalization)]]></description><category></category></item><item><title><![CDATA[gRPC vs Thrift]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/48830511</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/48830511</guid><author>dazheng</author><pubDate>Wed, 30 Sep 2015 17:01:42 +0800</pubDate><description><![CDATA[简单分布式架构
基本问题 
传输什么样的数据，用哪种协议
哪种方式数据交换的效率好
服务端如何处理请求
需要扩展服务端时
当你的服务超过最简单结构时，你想要 
灵活性
可扩展
低延迟
当然，你更想要简单
应该用这些协议吗
SOAP 
XML， XML还是XML
CORBA 
美好的想法，糟糕的实现
过渡设计和臃肿 
DCOM, COM+主要用于windows平台
HTTP/TCP/Socket/Wh]]></description><category></category></item><item><title><![CDATA[Shell常用]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/48630483</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/48630483</guid><author>dazheng</author><pubDate>Mon, 21 Sep 2015 18:33:19 +0800</pubDate><description><![CDATA[1、批量替换文本sed -i '/^$/d' *.sql     #删除空行
sed -i 's/,//g' *.sql 
sed -i 's/^M//g' *.sql       #需要替换^M 或
sed -i 's/\r//g' *.sql
sed -i 's/[ \t]*$//g' *.sql       #删除行尾空白
sed -i 's/$/,/g' *.sql 
sed -i '1 s]]></description><category></category></item><item><title><![CDATA[Spark SQL Relational Data Processing in Spark (学习笔记)]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/48252437</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/48252437</guid><author>dazheng</author><pubDate>Sun, 06 Sep 2015 22:29:50 +0800</pubDate><description><![CDATA[介绍
用户一方面需要关系型操作，另一方面需要过程式的操作，shark只能进行关系型操作。Spark SQL可以将两者结合起来。
Spark SQL由两个重要组成部分 
DataFrame API 
将关系型的处理与过程型处理结合起来，可以对外部数据源和Spark内建的分布式集合进行关系型操作
压缩的列式存储，而不是Java/Scala对象
Catalyst 
提供了一整套性能分析、计划、运行时代码生]]></description><category></category></item><item><title><![CDATA[组织行为学（学习笔记）]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/48239729</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/48239729</guid><author>dazheng</author><pubDate>Sun, 06 Sep 2015 09:12:24 +0800</pubDate><description><![CDATA[大五人格模型 
情绪稳定性
外倾性
经验开发放性
随和性
责任心]]></description><category></category></item><item><title><![CDATA[大规模数据管理系统调查（学习笔记）]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/47702297</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/47702297</guid><author>dazheng</author><pubDate>Sun, 16 Aug 2015 14:41:59 +0800</pubDate><description><![CDATA[介绍
大数据的四个特性 
Volume（体量）、Velocity（速度）、Variety（多样性）、Veracity（精确性）
应对 介绍基本情况 
Data model
System architecture
Consistency model
基本实现原理 
数据模型和数据分区
SEDA/MapReduce架构高扩展性
基于timestamps的并发控制协议
强一致性模型扩展约束
BASE模型及]]></description><category></category></item><item><title><![CDATA[Big Data Lambda Architecture 翻译]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/47282525</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/47282525</guid><author>dazheng</author><pubDate>Tue, 04 Aug 2015 21:06:36 +0800</pubDate><description><![CDATA[Big DataLambda Architecture
Posted on September5, 2012 by dbtube
In orderto meet the challenges of Big Data, you must rethink data systems from theground up. You will discover that some of the most]]></description><category></category></item><item><title><![CDATA[Linux\AIX 常用]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/47256803</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/47256803</guid><author>dazheng</author><pubDate>Mon, 03 Aug 2015 15:20:12 +0800</pubDate><description><![CDATA[1、编辑linux内核 
        1. 在/usr/src下untar这个包.     tar -xjf linux-2.6.25.tar.bz2  2. 配置内核
     cd /usr/src
          mv linux-2.6.25 linux
          cd linux
          make mrproper  #(删除一些stable的文件,针对刚do]]></description><category></category></item><item><title><![CDATA[CentOS6.4上安装配置snappy]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/47256561</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/47256561</guid><author>dazheng</author><pubDate>Mon, 03 Aug 2015 15:08:31 +0800</pubDate><description><![CDATA[1、下载    [hadoop@master pub]$ wget https://snappy.googlecode.com/files/snappy-1.1.0.tar.gz2、安装     [hadoop@master snappy-1.1.0]$ sudo yum install gcc gcc+ gcc-c++
     [hadoop@master bin]$ tar axvf snap]]></description><category></category></item><item><title><![CDATA[Hive常用]]></title><link>https://blog.youkuaiyun.com/dazheng/article/details/47256493</link><guid>https://blog.youkuaiyun.com/dazheng/article/details/47256493</guid><author>dazheng</author><pubDate>Mon, 03 Aug 2015 15:05:40 +0800</pubDate><description><![CDATA[主要整理于2011、2012年1、启动server及元数据    nohup hive --service hiveserver2 10000 > /dw/log/hiveserver.log 2>&1 &
      nohup hive --service metastore 9083 > /dw/log/metastore.log 2>&1 &2、字符集 
     hive元数据存储用mys]]></description><category></category></item></channel></rss>