sh脚本的例子=shell脚本

最新推荐文章于 2024-03-11 12:06:01 发布

清，纯一色

最新推荐文章于 2024-03-11 12:06:01 发布

阅读量222

点赞数

CC 4.0 BY-SA版权

文章标签： shell 大数据

原文链接：http://www.cnblogs.com/xjx-user/archive/2013/05/08/3067609.html

本文介绍了一个使用Hadoop进行大规模数据处理的实验流程。实验包括准备数据文件、配置Hadoop环境、运行不同规模的数据处理任务并监控输出结果等步骤。通过对参数调整及准确性测试，展示了如何优化基于Hadoop的大数据处理性能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

copy.sh

for((i=1;i<=45;i=i+1))
do
cat E2006.train >> speedup_E2006
done
echo "OK"

look_state.sh

tail -f thinkpoet_output.txt // output appended data as the file grows

for((block=1;block<=256;block=block*2))
do
        for((iter=1;iter<=4;iter=iter*2))
                do
                       cmd="hadoop jar DLSVM_Huawei_6_test_paper.jar E2006.train 150360 E2006.train.$block.$iter.model 1 0.1 $block $iter 0.1 E2006.test"
                        echo $cmd
                        $cmd
                done
done

run.sh

#1 !/bin/bash
2
3 echo "start experiment"
4 rm thinkpoet_output.txt
5 rm thinkpoet_record.txt
6 #echo "untarball file"
7 #tar xvf data.tar
8 bunzip2 log1p.E2006.train.bz2
9 bunzip2 log1p.E2006.test.bz2
10 bunzip2 YearPredictionMSD_scale.bz2
11 bunzip2 YearPredictionMSD_scale.t.bz2
12 hadoop fs -put log1p.E2006.train .
13 hadoop fs -put log1p.E2006.test .
14 hadoop fs -put YearPredictionMSD_scale .
15 hadoop fs -put YearPredictionMSD_scale.t .
16 #echo "preprocess file"
17 #bash preprocess_file.sh >> thinkpoet_output.txt
18
19 echo "parameter effect test"
20 bash parameter_effect.sh >> thinkpoet_output.txt
21
22 echo "accuracy test"
23 bash accuracy_test.sh >> thinkpoet_output.txt
24
25 echo "experiment finished"
26 echo "please send files \"thinkpoet_output.txt\" and \"thinkpoet_record.txt\" to me, thank you!"

speechup.sh

1 hadoop fs -rm speedup_E2006
2 hadoop fs -put speedup_E2006 .
3 hadoop jar DLSVM_Huawei_6_test_paper.jar speedup_E2006 150360 speedup_E2006.model 1 0.1 256 1 0.1 E2006.test

转载于:https://www.cnblogs.com/xjx-user/archive/2013/05/08/3067609.html