shuhuai007的专栏

pig的注释格式共有两种： If you place Pig Latin statements in a script, the script can include comments. For multi-line comments use /* …. */ For single line comments use -- /* myscript.pig My

以下为现场实录： 什么是Apache  Pig？ApachePig包括两个部分，首先是PigLatin的语言，是类SQL的数据处理语言。其次，Apache  Pig是在Hadoop这个软件上运行Pig  Latin语言的执行引擎，一部分是语言的部分，还有一部分是实践的部分。 比较一下Pig和Hadoop，跟Hadoop相比Pig具有更快的开发效率，所以可以用更少的代码实现和MapRed

Pig性能优化   1. 尽早去除无用的数据   MapReduce Job的很大一部分开销在于磁盘IO和数据的网络传输，如果能尽早的去除无用的数据，减少数据量，会提升Pig的性能。   1). 尽早的使用Filter   使用Filter可以去除数据中无用的行(Record)，尽早的Filter掉无用的数据，可以减少数据量，提升Pig性能。   2). 尽早的使用Project(F

1. 注释方式     a) . --      b). /*   */ 示例： /*  * Licensed to the Apache Software Foundation (ASF) under one  * or more contributor license agreements.  See the NOTICE file  * distributed with thi

TOKENIZE 分割函数的使用： 1.  假设有如下两个文件：test_tokenize.pig,  log log 内容如下： --------------------------------------- 1 abc,zhoujie 2 mmm,liudehua 3 nnn*zhourunfa 0 nnn()zhouxingchi 5

cd myudfs javac -cp pig.jar UPPER.java cd .. jar -cf myudfs.jar myudfs

We saw an example of this when the Pow function detected overflow: for (int i = 0; i < exponent; i++) { long preresult = result; result *= base; if (preresult > result) { // We overflowed. Give a wa

Memory Management Pig allocates a fix amount of memory to store bags and spills to disk as soon as the memory limit is reached. This is very similar to how Hadoop decides when to spill data accum

A = load 'foreach_input' as (user:chararray, id:long, address:chararray, phone:chararray); --B = foreach A generate user, id; B = foreach A generate user, id, 'good'; --describe A; --describe B;

A = load 'foreach_input' as (user:chararray, id:long, address:chararray, phone:chararray); --B = foreach A generate user, id; B = foreach A generate user, id, 'good'; dump B; 注：'good'是常量

http://www.codelast.com/?p=4550

测试数据文件foreach_bag.log，内容如下： {('b', 55), ('sally', 52), ('john', 25)} {('c', 355), ('msally', 352), ('mjohn', 325)} foreach_bag.pig代码如下： A = LOAD 'foreach_bag.log' as (b:bag{t:(x:chararray, y:int

As discussed elsewhere, Pig can run with or without data type information. In cases where the load function you are using creates data that is already typed, there is little you need to do to optimi

1. 到http://www.vim.org/scripts/script.php?script_id=2186下载pig.vim 2. copy pig.vim file into your ~/.vim/syntax/ directory;  3. add the following three lines to your ~/.vimrc file:  augroup filetype

1. pig中relation名字和field名字不能重新赋值，因为会造成引用丢失 you can write as follows, A = load 'NYSE_dividends' (exchange, symbol, date, dividends); A = filter A by dividends > 0; A = foreach A generate UPPER(symbo

prices = load 'NYSE_daily' as (exchange, symbol, date, open, high, low, close, volume, adj_close); gain = foreach prices generate close - open; gain2 = foreach prices generate $6 - $3; Field

1. divs = load 'NYSE_dividends' using PigStorage(','); NYSE_dividends为相对路径（相对于hdfs空间下/user/yourlogin/），load的input路径可以是目录，也可以是文件 分隔符为",",通过参数传给PigStorage。 2. divs = load 'NYSE_dividends' as (excha

Overview 近期需要用pig做一些统计，由于没有系统学习，总是出现一些问题，且不容易调试，执行效率也不高。所以打算看一些官方文档，在此做些笔记。 pig性能提升 指定类型 如果在load文件时不指定类型，pig在计算时会指定为double类型，而在很多时候，数据本应是整形等，指定为double类型会增加广计算量。另外，指定类型也会使错误提早暴露出来。 --Query