linux/shell:读取包含‘.‘键名的.properties文件

博客主要围绕在Linux Shell中读取properties文件展开。当properties的key名仅由字母数字组成时,读取较简单;若key名含特殊字符,原方法失效。网上文章提供的解决办法有漏洞,作者在此基础上改进,对以特定符号起始的行不处理,并给出实现代码。

如果properties中的key名只是由字母数字组成,那读取properties中的property很简单,示例如下:

假设.properties中内容为:

prop1.properties

username=tom

读取prop1.properties中的username

. prop1.properties
echo $username

但是如果key的名字包含了.(比如redis.port),这个办法就不能用了,因为linux shell中不允许变量名有.符号。

网上找到这篇文章 《linux – 如何读取包含使用Shell脚本具有句点字符的键的.properties文件
,提供了另一种解决办法,就是以文本方式读取properties文件,解析每一行=号两边的内容,将key中的.替换为_.但是这篇文章提供的脚本有一个漏洞:就是没有处理注释符号#.所以容易会带来逻辑隐患。所以在这篇文章提供的脚本的基础我改进了一下,对于以#起始的行不处理。

实现代码如下:

#!/bin/bash
config="$HOME/your.properties"
if [ -f "$config" ]
then
	echo "$config found."
	while IFS='=' read -r key value
	do
		## '.'替换为'-'
		key=$(echo $key | tr '.' '_')
		## 不处理注释符#起始的key
		[[ -z $(echo "$key" | grep -P '\s*#+.*' ) ]] \
			&& eval "${key}='${value}'"  \
			&& echo key=$key value=$value
	done < "$config"
else
	echo "$config not found."
fi

另一个简单的实现是这样的:

#!/bin/bash
config="$HOME/your.properties"
# 定义一个函数从properties文件读取key
function prop {
	[ -f "$config" ] && grep -P "^\s*[^#]?${1}=.*$" $config | cut -d'=' -f2
}

# 调用示例
echo redis.port=$(prop "redis.port")
现在在slave1 [root@slave1 ~]# ssh root@192.168.81.131 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:or9B55pJ+BLGf/sKAzjlX0oEjPZyD4DTCYKgJt6djNk. Please contact your system administrator. Add correct host key in /root/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /root/.ssh/known_hosts:3 ECDSA host key for 192.168.81.131 has changed and you have requested strict checking. Host key verification failed. [root@slave1 ~]# /export/server/hive/bin/hive SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/export/server/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/export/server/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/export/server/jdk/bin:/export/server/hadoop/bin:/export/server/hadoop/sbin:/export/server/hive/bin:/export/server/jdk/bin:/export/server/hadoop/bin:/export/server/hadoop/sbin:/export/server/hive/bin:/root/bin) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/export/server/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/export/server/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = 9982bf98-b1cc-4e2c-aa30-8b3596640a5a Logging initialized using configuration in jar:file:/export/server/hive/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true Exception in thread "main" java.lang.RuntimeException: java.net.ConnectException: Call From slave1/192.168.81.131 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:651) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: java.net.ConnectException: Call From slave1/192.168.81.131 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549) at org.apache.hadoop.ipc.Client.call(Client.java:1491) at org.apache.hadoop.ipc.Client.call(Client.java:1388) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:904) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1661) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683) at org.apache.hadoop.hive.ql.exec.Utilities.ensurePathIsWritable(Utilities.java:4486) at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:760) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:701) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:627) ... 9 more Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804) at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606) at org.apache.hadoop.ipc.Client.call(Client.java:1435) ... 34 more [root@slave1 ~]# SHOW DATABASES; -bash: SHOW: 未找到命令 [root@slave1 ~]#
最新发布
10-31
输入代码 import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { public WordCount() { } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(WordCount.TokenizerMapper.class); job.setCombinerClass(WordCount.IntSumReducer.class); job.setReducerClass(WordCount.IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("hdfs://10.23.175.100:9000/user/wordcount.txt")); FileOutputFormat.setOutputPath(job, new Path("hdfs://10.23.175.100:9000/user/result15")); System.exit(job.waitForCompletion(true)?0:1); } public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private static final IntWritable one = new IntWritable(1); private Text word = new Text(); public TokenizerMapper() { } public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) { this.word.set(itr.nextToken()); context.write(this.word, one); } } } public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public IntSumReducer() { } public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException { int sum = 0; IntWritable val; for(Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) { val = (IntWritable)i$.next(); } this.result.set(sum); context.write(key, this.result); } } } 后报错 D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\bin\java.exe "-javaagent:D:\2025.9\JAVA\IDEA 2023.1\IntelliJ IDEA 2023.1\lib\idea_rt.jar=4323:D:\2025.9\JAVA\IDEA 2023.1\IntelliJ IDEA 2023.1\bin" -Dfile.encoding=UTF-8 -classpath D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\charsets.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\deploy.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\dnsns.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jaccess.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\localedata.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\nashorn.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunec.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\zipfs.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\javaws.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jce.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfr.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfxswt.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jsse.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\management-agent.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\plugin.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\resources.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\rt.jar;D:\2025.9\Hadoop\java\hadoop_project\out\production\hadoop_project;D:\2025.9\Hadoop\test\4\lib2\fst-2.50.jar;D:\2025.9\Hadoop\test\4\lib2\re2j-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\asm-5.0.4.jar;D:\2025.9\Hadoop\test\4\lib2\guice-4.0.jar;D:\2025.9\Hadoop\test\4\lib2\jna-5.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\avro-1.7.7.jar;D:\2025.9\Hadoop\test\4\lib2\gson-2.8.9.jar;D:\2025.9\Hadoop\test\4\lib2\okio-2.8.0.jar;D:\2025.9\Hadoop\test\4\lib2\jline-3.9.0.jar;D:\2025.9\Hadoop\test\4\lib2\jsch-0.1.55.jar;D:\2025.9\Hadoop\test\4\lib2\asm-tree-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\jettison-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jsr305-3.0.2.jar;D:\2025.9\Hadoop\test\4\lib2\log4j-1.2.17.jar;D:\2025.9\Hadoop\test\4\lib2\okhttp-4.9.3.jar;D:\2025.9\Hadoop\test\4\lib2\dnsjava-2.1.7.jar;D:\2025.9\Hadoop\test\4\lib2\ehcache-3.3.1.jar;D:\2025.9\Hadoop\test\4\lib2\json-io-2.5.1.jar;D:\2025.9\Hadoop\test\4\lib2\objenesis-2.6.jar;D:\2025.9\Hadoop\test\4\lib2\paranamer-2.3.jar;D:\2025.9\Hadoop\test\4\lib2\guava-27.0-jre.jar;D:\2025.9\Hadoop\test\4\lib2\javax.inject-1.jar;D:\2025.9\Hadoop\test\4\lib2\snakeyaml-1.26.jar;D:\2025.9\Hadoop\test\4\lib2\aopalliance-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\asm-commons-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\commons-cli-1.2.jar;D:\2025.9\Hadoop\test\4\lib2\commons-net-3.6.jar;D:\2025.9\Hadoop\test\4\lib2\httpcore-4.4.13.jar;D:\2025.9\Hadoop\test\4\lib2\java-util-1.9.0.jar;D:\2025.9\Hadoop\test\4\lib2\jaxb-api-2.2.11.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-core-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-util-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-xdr-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\reload4j-1.2.22.jar;D:\2025.9\Hadoop\test\4\lib2\stax2-api-4.2.1.jar;D:\2025.9\Hadoop\test\4\lib2\zookeeper-3.5.6.jar;D:\2025.9\Hadoop\test\4\lib2\asm-analysis-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\commons-io-2.8.0.jar;D:\2025.9\Hadoop\test\4\lib2\commons-text-1.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-kms-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-nfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-core-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-json-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\json-smart-2.4.7.jar;D:\2025.9\Hadoop\test\4\lib2\jsr311-api-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-admin-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-asn1-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-pkix-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-util-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\slf4j-api-1.7.36.jar;D:\2025.9\Hadoop\test\4\lib2\failureaccess-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\guice-servlet-4.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-auth-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\httpclient-4.5.13.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-xc-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jaxb-impl-2.2.3-1.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-guice-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\json-simple-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-client-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-common-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-crypto-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-server-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\checker-qual-2.5.2.jar;D:\2025.9\Hadoop\test\4\lib2\commons-codec-1.15.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-client-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-server-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-config-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\leveldbjni-all-1.8.jar;D:\2025.9\Hadoop\test\4\lib2\metrics-core-3.2.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-3.10.6.Final.jar;D:\2025.9\Hadoop\test\4\lib2\bcpkix-jdk15on-1.60.jar;D:\2025.9\Hadoop\test\4\lib2\bcprov-jdk15on-1.60.jar;D:\2025.9\Hadoop\test\4\lib2\commons-math3-3.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-core-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-servlet-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-identity-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\protobuf-java-2.5.0.jar;D:\2025.9\Hadoop\test\4\lib2\snappy-java-1.1.8.2.jar;D:\2025.9\Hadoop\test\4\lib2\woodstox-core-5.3.0.jar;D:\2025.9\Hadoop\test\4\lib2\commons-lang3-3.12.0.jar;D:\2025.9\Hadoop\test\4\lib2\curator-client-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-simplekdc-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kotlin-stdlib-1.4.10.jar;D:\2025.9\Hadoop\test\4\lib2\token-provider-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\zookeeper-jute-3.5.6.jar;D:\2025.9\Hadoop\test\4\lib2\accessors-smart-2.4.7.jar;D:\2025.9\Hadoop\test\4\lib2\commons-compress-1.21.jar;D:\2025.9\Hadoop\test\4\lib2\commons-daemon-1.0.13.jar;D:\2025.9\Hadoop\test\4\lib2\commons-logging-1.1.3.jar;D:\2025.9\Hadoop\test\4\lib2\curator-recipes-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-nfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-rbf-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-registry-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-api-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\HikariCP-java7-2.4.12.jar;D:\2025.9\Hadoop\test\4\lib2\mssql-jdbc-6.2.1.jre7.jar;D:\2025.9\Hadoop\test\4\lib2\nimbus-jose-jwt-9.8.1.jar;D:\2025.9\Hadoop\test\4\lib2\slf4j-reload4j-1.7.36.jar;D:\2025.9\Hadoop\test\4\lib2\j2objc-annotations-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jcip-annotations-1.0-1.jar;D:\2025.9\Hadoop\test\4\lib2\netty-all-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-beanutils-1.9.4.jar;D:\2025.9\Hadoop\test\4\lib2\curator-framework-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-core-asl-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-databind-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\javax.servlet-api-3.1.0.jar;D:\2025.9\Hadoop\test\4\lib2\javax.websocket-api-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-annotations-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-httpfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-collections-3.2.2.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-common-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-shaded-guava-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-base-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-mapper-asl-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-io-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-buffer-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-common-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\swagger-annotations-1.5.4.jar;D:\2025.9\Hadoop\test\4\lib2\audience-annotations-0.5.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-registry-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-annotations-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jakarta.xml.bind-api-2.3.2.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-xml-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-handler-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-rbf-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-http-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-jndi-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-plus-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-util-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\kotlin-stdlib-common-1.4.10.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-configuration2-2.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jakarta.activation-api-1.2.1.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-dns-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-xml-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-client-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-server-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-webapp-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-http-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-mqtt-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-smtp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-client-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-tests-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-services-api-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\javax.websocket-client-api-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-servlet-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-http2-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-redis-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-socks-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-stomp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-api-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\animal-sniffer-annotations-1.17.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-native-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-examples-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-router-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-services-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-security-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-hs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-shaded-protobuf_3_7-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-util-ajax-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-haproxy-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-handler-proxy-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-udt-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-app-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-memcache-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-rxtx-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-sctp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-client-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-common-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-server-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-web-proxy-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-json-provider-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-annotations-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-servlet-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\geronimo-jcache_1.0_spec-1.0-alpha-1.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-nodemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-native-client-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-shuffle-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-uploader-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-module-jaxb-annotations-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-jobclient-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-hs-plugins-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-nativetask-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-mawo-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-resourcemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-classes-epoll-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-sharedcachemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-classes-kqueue-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\javax-websocket-client-impl-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\javax-websocket-server-impl-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-jobclient-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-classes-macos-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-distributedshell-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-timeline-pluginstorage-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-unix-common-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-applicationhistoryservice-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-unmanaged-am-launcher-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar WordCount 2025-10-30 01:03:31,804 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2025-10-30 01:03:37,414 WARN [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(136)) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties 2025-10-30 01:03:37,456 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(378)) - Scheduled Metric snapshot period at 10 second(s). 2025-10-30 01:03:37,456 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started Exception in thread "main" ExitCodeException exitCode=-1073741515: at org.apache.hadoop.util.Shell.runCommand(Shell.java:1007) at org.apache.hadoop.util.Shell.run(Shell.java:900) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:677) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:336) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:162) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:148) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1571) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1568) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1568) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1589) at WordCount.main(WordCount.java:28) 进程已结束,退出代码1
10-31
D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\bin\java.exe "-javaagent:D:\2025.9\JAVA\IDEA 2023.1\IntelliJ IDEA 2023.1\lib\idea_rt.jar=11975:D:\2025.9\JAVA\IDEA 2023.1\IntelliJ IDEA 2023.1\bin" -Dfile.encoding=UTF-8 -classpath D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\charsets.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\deploy.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\dnsns.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jaccess.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\localedata.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\nashorn.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunec.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\zipfs.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\javaws.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jce.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfr.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfxswt.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jsse.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\management-agent.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\plugin.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\resources.jar;D:\2025.9\Hadoop\test\4\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\rt.jar;D:\2025.9\Hadoop\java\hadoop_project\out\production\hadoop_project;D:\2025.9\Hadoop\test\4\lib2\fst-2.50.jar;D:\2025.9\Hadoop\test\4\lib2\re2j-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\asm-5.0.4.jar;D:\2025.9\Hadoop\test\4\lib2\guice-4.0.jar;D:\2025.9\Hadoop\test\4\lib2\jna-5.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\avro-1.7.7.jar;D:\2025.9\Hadoop\test\4\lib2\gson-2.8.9.jar;D:\2025.9\Hadoop\test\4\lib2\okio-2.8.0.jar;D:\2025.9\Hadoop\test\4\lib2\jline-3.9.0.jar;D:\2025.9\Hadoop\test\4\lib2\jsch-0.1.55.jar;D:\2025.9\Hadoop\test\4\lib2\asm-tree-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\jettison-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jsr305-3.0.2.jar;D:\2025.9\Hadoop\test\4\lib2\log4j-1.2.17.jar;D:\2025.9\Hadoop\test\4\lib2\okhttp-4.9.3.jar;D:\2025.9\Hadoop\test\4\lib2\dnsjava-2.1.7.jar;D:\2025.9\Hadoop\test\4\lib2\ehcache-3.3.1.jar;D:\2025.9\Hadoop\test\4\lib2\json-io-2.5.1.jar;D:\2025.9\Hadoop\test\4\lib2\objenesis-2.6.jar;D:\2025.9\Hadoop\test\4\lib2\paranamer-2.3.jar;D:\2025.9\Hadoop\test\4\lib2\guava-27.0-jre.jar;D:\2025.9\Hadoop\test\4\lib2\javax.inject-1.jar;D:\2025.9\Hadoop\test\4\lib2\snakeyaml-1.26.jar;D:\2025.9\Hadoop\test\4\lib2\aopalliance-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\asm-commons-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\commons-cli-1.2.jar;D:\2025.9\Hadoop\test\4\lib2\commons-net-3.6.jar;D:\2025.9\Hadoop\test\4\lib2\httpcore-4.4.13.jar;D:\2025.9\Hadoop\test\4\lib2\java-util-1.9.0.jar;D:\2025.9\Hadoop\test\4\lib2\jaxb-api-2.2.11.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-core-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-util-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-xdr-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\reload4j-1.2.22.jar;D:\2025.9\Hadoop\test\4\lib2\stax2-api-4.2.1.jar;D:\2025.9\Hadoop\test\4\lib2\zookeeper-3.5.6.jar;D:\2025.9\Hadoop\test\4\lib2\asm-analysis-9.1.jar;D:\2025.9\Hadoop\test\4\lib2\commons-io-2.8.0.jar;D:\2025.9\Hadoop\test\4\lib2\commons-text-1.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-kms-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-nfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-core-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-json-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\json-smart-2.4.7.jar;D:\2025.9\Hadoop\test\4\lib2\jsr311-api-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-admin-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-asn1-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-pkix-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-util-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\slf4j-api-1.7.36.jar;D:\2025.9\Hadoop\test\4\lib2\failureaccess-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\guice-servlet-4.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-auth-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\httpclient-4.5.13.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-xc-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jaxb-impl-2.2.3-1.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-guice-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\json-simple-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-client-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-common-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-crypto-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-server-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\checker-qual-2.5.2.jar;D:\2025.9\Hadoop\test\4\lib2\commons-codec-1.15.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-client-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-server-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\kerby-config-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\leveldbjni-all-1.8.jar;D:\2025.9\Hadoop\test\4\lib2\metrics-core-3.2.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-3.10.6.Final.jar;D:\2025.9\Hadoop\test\4\lib2\bcpkix-jdk15on-1.60.jar;D:\2025.9\Hadoop\test\4\lib2\bcprov-jdk15on-1.60.jar;D:\2025.9\Hadoop\test\4\lib2\commons-math3-3.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-core-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jersey-servlet-1.19.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-identity-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\protobuf-java-2.5.0.jar;D:\2025.9\Hadoop\test\4\lib2\snappy-java-1.1.8.2.jar;D:\2025.9\Hadoop\test\4\lib2\woodstox-core-5.3.0.jar;D:\2025.9\Hadoop\test\4\lib2\commons-lang3-3.12.0.jar;D:\2025.9\Hadoop\test\4\lib2\curator-client-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\kerb-simplekdc-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\kotlin-stdlib-1.4.10.jar;D:\2025.9\Hadoop\test\4\lib2\token-provider-1.0.1.jar;D:\2025.9\Hadoop\test\4\lib2\zookeeper-jute-3.5.6.jar;D:\2025.9\Hadoop\test\4\lib2\accessors-smart-2.4.7.jar;D:\2025.9\Hadoop\test\4\lib2\commons-compress-1.21.jar;D:\2025.9\Hadoop\test\4\lib2\commons-daemon-1.0.13.jar;D:\2025.9\Hadoop\test\4\lib2\commons-logging-1.1.3.jar;D:\2025.9\Hadoop\test\4\lib2\curator-recipes-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-nfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-rbf-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-registry-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-api-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\HikariCP-java7-2.4.12.jar;D:\2025.9\Hadoop\test\4\lib2\mssql-jdbc-6.2.1.jre7.jar;D:\2025.9\Hadoop\test\4\lib2\nimbus-jose-jwt-9.8.1.jar;D:\2025.9\Hadoop\test\4\lib2\slf4j-reload4j-1.7.36.jar;D:\2025.9\Hadoop\test\4\lib2\j2objc-annotations-1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jcip-annotations-1.0-1.jar;D:\2025.9\Hadoop\test\4\lib2\netty-all-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-beanutils-1.9.4.jar;D:\2025.9\Hadoop\test\4\lib2\curator-framework-4.2.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-core-asl-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-databind-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\javax.servlet-api-3.1.0.jar;D:\2025.9\Hadoop\test\4\lib2\javax.websocket-api-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-annotations-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-httpfs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-collections-3.2.2.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-common-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-shaded-guava-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-base-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-mapper-asl-1.9.13.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-io-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-buffer-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-common-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\swagger-annotations-1.5.4.jar;D:\2025.9\Hadoop\test\4\lib2\audience-annotations-0.5.0.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-registry-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-annotations-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jakarta.xml.bind-api-2.3.2.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-xml-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-handler-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-rbf-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-http-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-jndi-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-plus-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-util-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\kotlin-stdlib-common-1.4.10.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\commons-configuration2-2.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jakarta.activation-api-1.2.1.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-dns-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-xml-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-client-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-server-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-webapp-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-http-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-mqtt-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-smtp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-client-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-tests-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-services-api-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\javax.websocket-client-api-1.0.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-servlet-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-http2-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-redis-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-socks-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-stomp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-api-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\animal-sniffer-annotations-1.17.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-native-client-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-examples-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-router-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-services-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-security-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-hs-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-shaded-protobuf_3_7-1.1.1.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-util-ajax-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-haproxy-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-handler-proxy-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-udt-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-app-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-codec-memcache-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-rxtx-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-sctp-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-client-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-common-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-server-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-web-proxy-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-jaxrs-json-provider-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\jetty-annotations-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\websocket-servlet-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\geronimo-jcache_1.0_spec-1.0-alpha-1.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-common-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-nodemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-hdfs-native-client-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-shuffle-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-uploader-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\jackson-module-jaxb-annotations-2.12.7.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-jobclient-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-hs-plugins-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-nativetask-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-mawo-core-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-resourcemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-classes-epoll-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-sharedcachemanager-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-classes-kqueue-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\javax-websocket-client-impl-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\javax-websocket-server-impl-9.4.43.v20210629.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-mapreduce-client-jobclient-3.3.4-tests.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-classes-macos-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-distributedshell-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-timeline-pluginstorage-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-unix-common-4.1.77.Final.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-server-applicationhistoryservice-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\hadoop-yarn-applications-unmanaged-am-launcher-3.3.4.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar;D:\2025.9\Hadoop\test\4\lib2\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar WordCount 2025-10-30 00:59:38,619 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2025-10-30 00:59:44,218 WARN [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(136)) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties 2025-10-30 00:59:44,263 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(378)) - Scheduled Metric snapshot period at 10 second(s). 2025-10-30 00:59:44,263 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started Exception in thread "main" ExitCodeException exitCode=-1073741515: at org.apache.hadoop.util.Shell.runCommand(Shell.java:1007) at org.apache.hadoop.util.Shell.run(Shell.java:900) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:677) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:336) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:162) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:148) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1571) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1568) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1568) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1589) at WordCount.main(WordCount.java:28)
10-31
本关任务:配置 Flume 相关配置内容,完成采集目录下所有文件至 HDFS 中。 相关知识 为了完成本关任务,你需要掌握: Exec Source; Memory Channel; HDFS Sink; 采集示例。 Exec Source Fluem Exec Source 介绍及属性详解如下所述: Exec Source 在启动时运行一个给定的 Unix 命令,并期望该进程在标准输出上不断地生成数据(除非将属性 logStdErr 设置为 true,否则将直接丢弃 stderr)。如果进程出于任何原因退出,源也将退出,并且不会产生进一步的数据。这意味着像cat [named pipe]或tail - f [file]这样的配置将产生预期的结果,而as date可能不会产生这种结果——前两个命令将产生数据流,而后者将产生单个事件并退出。 属性详解 必要属性: agent.sources=xx //当前source的名称 agent.sources.xx.channels=xxx //当前source连接的channel的名称,因为一个source可以连接多个channel,所以这的属性为channels agent.sources.xx.type=exec //当前source的类型 agent.sources.xx.command=xxxx //当前source在exec类型下的执行命令 其他属性: shell //用于运行命令的shell调用。例如/bin/sh - c。仅适用于依赖于通配符、反勾号、管道等shell特性的命令。 restartThrottle //尝试重新启动前等待的时间(以毫秒为单位) 默认值:10000 restart //如果执行的cmd死亡,是否应该重新启动它 默认值:false logStdErr //是否应该记录命令的stderr 默认值:false batchSize //一次读取和发送到通道的最大行数 默认值:20 batchTimeout //如果没有达到缓冲区大小,则在数据被向下推送之前等待的时间(以毫秒为单位) 默认值:3000 selector.type //replicating(复用) 或者 multiplexing(多路复用) interceptors //以空格分隔的拦截器列表 ExecSource和其他异步源的问题是,如果无法将事件放入客户机知道的通道中,则源不能保证该事件。在这种情况下,数据将丢失。例如,最常见的请求特性之一是tail -F [file]类用例,在这种用例中,应用程序向磁盘上的日志文件写入数据,Flume 跟踪文件,将每一行作为一个事件发送。虽然这是可能的,但有一个明显的问题;如果通道被填满,Flume 不能发送事件,会发生什么? Flume 无法向编写日志文件的应用程序表明它需要保留日志,或者由于某种原因没有发送事件。如果这没有意义,您只需要知道这一点:当使用单向异步接口(如 ExecSource )时,您的应用程序永远不能保证数据已被接收!作为此警告的扩展(而且要完全清楚),在使用此源时绝对没有事件交付的保证。要获得更强的可靠性保证,可以考虑使用假脱机目录源、Taildir 源或通过 SDK 与 Flume 直接集成。 Memory Channel Memory Channel 介绍及属性详解如下所述: Events 存储在 Java Heap,即内存队列中(内存的大小是可以指定的)。对于流量较高和由于 Agent 故障而准备丢失数据的流程来说,这是一个理想的选择; 属性详解 //必有属性 agent.channels=xx //指定当前channel的名称 agent.channels.xx.type=memory //其他属性 capacity //存储在通道中的最大事件数量 默认值:100 transactionCapacity //通道在每个事务中将从源获取或提供给接收器的最大事件数 默认值:100 keep-alive //超时(以秒为单位),用于添加或删除事件 默认值:3 byteCapacityBufferPercentage //定义字节码容量与估计的通道中所有事件的总大小之间的缓冲区百分比,以解释报头中的数据。 默认值:20 byteCapacity //允许的最大内存字节数,作为该通道中所有事件的总和。实现只计算事件体,这也是提供bytecapacitybufferpercent配置参数的原因。默认计算值等于JVM可用的最大内存的80%(即在命令行上传递的-Xmx值的80%)。请注意,如果您在一个JVM上有多个内存通道,并且它们碰巧包含相同的物理事件(例如,如果您使用来自单个源的复制通道选择器),那么出于通道字节码的目的,这些事件大小可能会被重复计算。将该值设置为0将导致该值回落到大约200gb的硬内部限制。 HDFS Sink File Channel 介绍及属性详解如下所述: 这个接收器将事件写入 Hadoop 分布式文件系统( HDFS )。它目前支持创建文本和序列文件。它支持这两种文件类型的压缩。可以根据经过的时间、数据大小或事件数量定期滚动文件(关闭当前文件并创建一个新文件)。它还根据时间戳或事件起源的机器等属性对数据进行存储/分区。 HDFS 目录路径可能包含格式化转义序列,这些转义序列将被 HDFS 接收器替换,以生成一个目录/文件名来存储事件。使用这个接收器需要安装 Hadoop ,以便 Flume 可以使用Hadoop jar 与 HDFS 集群通信。注意,需要一个支持 sync() 调用的 Hadoop 版本。 属性详解 必有属性: agent.sinks=xx //当前sink的名称 agent.sinks.xx.type = hdfs //当前sink的类型 agent.sinks.xx.channel=xxx //与当前sink连接的channel的名称,因为一个sink只能连接一个channel,所以当前的属性为channel agent.sinks.xx.hdfs.path=$HDFS_PATH //hdfs上的目录路径 其他属性: hdfs.filePrefix //Flume在hdfs创建的目录中创建的文件路径的上一级 hdfs.fileSuffix //附加到文件的后缀(例如:.avro -注意:句点不会自动添加) hdfs.inUsePrefix //前缀,用于flume主动写入的临时文件 hdfs.inUseSuffix //后缀,用于flume主动写入的临时文件 默认值:.tmp hdfs.emptyInUseSuffix //if false,在编写输出时使用inUseSuffix中设置的参数,关闭输出hdfs之后。从输出文件名中删除inUseSuffix设置的参数。if true,则inUseSuffix参数被忽略,取而代之的是一个空字符串。 默认值:false hdfs.rollInterval //在滚动当前文件之前等待的秒数(0 =根据时间间隔永不滚动) 默认值:30 hdfs.rollSize //要触发滚动的文件大小,以字节为单位(0:永远不要基于文件大小进行滚动) 默认值:1024 hdfs.rollCount //滚前写入文件的事件数(0 =从不基于事件数滚) 默认值:10 hdfs.idleTimeout //关闭非活动文件的超时(0 =禁用自动关闭空闲文件) 默认值:0 hdfs.batchSize //在将文件刷新到HDFS之前写入文件的事件数 默认值:100 hdfs.codeC //压缩编解码器:gzip, bzip2, lzo, lzop, snappy hdfs.fileType //文件格式:SequenceFile, DataStream或CompressedStream (1)DataStream不会压缩输出文件,请不要设置codeC (2)CompressedStream需要设置hdfs。带有可用编解码器的编解码器 默认值:SequenceFile hdfs.maxnOpenFiles //只允许这个开放文件的数量。如果超过这个数字,则关闭最古老的文件。 默认值:5000 hdfs.minBlockReplicas //指定每个HDFS块的最小复制数。如果没有指定,它来自类路径中的默认Hadoop配置 hdfs.writeFormat //序列文件(SequenceFile)记录的格式。Text或Writable在使用Flume创建数据文件之前设置为Text,否则这些文件不能被Apache Impala(孵化)或Apache Hive读取。 默认值:Writable hdfs.threadsPoolSize //用于HDFS IO操作(打开、写入等)的每个HDFS接收器的线程数 默认值:10 hdfs.rollTimerPoolSize //每个HDFS接收器用于调度定时文件滚动的线程数 默认值:1 hdfs.kerberosPrincipal //访问安全HDFS的Kerberos用户主体 hdfs.kerberosKeytab //Kerberos keytab用于访问安全的HDFS hdfs.round //是否应该将时间戳四舍五入(如果为真,则影响除%t之外的所有基于时间的转义序列) 默认值:false hdfs.roundUnit //四舍五入的单位,时间,分钟,或小时 默认值:second hdfs.roundValue //将此(在使用hdfs.roundUnit配置的单元中)分为以上多个(在单元配置的单元中),小于当前时间 默认值:1 hdfs.timeZone //用于解析目录路径的时区的名称,例如:America/Los Angeles。 默认值:Local Time hdfs.useLocalTimeStamp //在替换转义序列时,使用本地时间(而不是事件头的时间戳) 默认值:false hdfs.closeTries //设置为0:有节点挂掉后,不会给file重命名,并且文件可能一直处于打开状态,并且后缀为自己设置的后缀;设置为1:有节点挂掉后,会尝试给file重命名,但是次数有限制;如果尝试失败,文件会保持开启,但会在flume再次启动时关闭. 默认值:0 hdfs.retryInterval //在连续试图关闭文件之间的时间。每个关闭调用都要花费多个RPC往返于Namenode,因此设置太低会导致名称节点上的大量负载。如果设置为0或更少,sink将不会试图关闭该文件 默认值:180 serializer //其他可能的选项包括avro_event或EventSerializer.BUilder实现的完全限定类名 默认值:TEXT 采集示例 接下来让我们通过一个简单采集示例来了解本实训 Flume 采集方案案例的编写。 案例分析 案例流程图如下: 案例流程如下: 创建符合条件的flume配置文件; 执行配置文件,开启监控; 向flumedata目录中文件写入数据; exec source 实时监控文件是否有追加数据写入; 将读取到的数据,通过 Hdfs Sink 方式输出到 HDFS。 Exec Source 常用配置如下: Property Name Default Description channels - type - exec command - 运行的linux命令 shell - 指定shell类型 batchSize 20 每次读取并传送给channel的最大行数 selector.type replicating replicating or multiplexing;选择器类型,默认将源数据复制发送给所有channel 配置文件内容如下: # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = exec a1.sources.r1.spoolDir = tail -F /opt/flume/flumedata/test.log # Describe the sink a1.sinks.k1.type = hdfs a1.sinks.k1.hdfs.path = hdfs://hadoop101:9000/user/root/flume/splogs/%Y%m%d #上传文件的前缀 a1.sinks.k1.hdfs.filePrefix = logs- a1.sinks.k1.hdfs.fileSuffix = hdfs #是否按照时间滚动文件夹 a1.sinks.k1.hdfs.round = true #多少时间单位创建一个新的文件夹 a1.sinks.k1.hdfs.roundValue = 1 #重新定义时间单位 a1.sinks.k1.hdfs.roundUnit = hour #是否使用本地时间戳 a1.sinks.k1.hdfs.useLocalTimeStamp = true #积攒多少个 Event 才 flush 到 HDFS 一次,到了一定时间了也会去flush a1.sinks.k1.hdfs.batchSize = 1000 #设置文件类型,可支持压缩 a1.sinks.k1.hdfs.fileType = DataStream #多久生成一个新的文件,60s a1.sinks.k1.hdfs.rollInterval = 60 #设置每个文件的滚动大小 a1.sinks.k1.hdfs.rollSize = 134117700 #文件的滚动与 Event 数量无关 a1.sinks.k1.hdfs.rollCount = 0 # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 注意:对于所有与时间相关的转义序列,Event Header 中必须存在以 timestamp 的 key(除非 hdfs.useLocalTimeStamp 设置为 true,此方法会使用 TimestampInterceptor 自动添 timestamp)。 启动监控文件夹命令: bin/flume-ng agent --conf conf/ --name a1 --conf-file job/flume-file-hdfs.conf 参数说明: --conf/-c:表示配置文件存储在 conf/目录 --name/-n:表示给 agent 起名为 a1 --conf-file/-f:Flume 本次启动读取的配置文件是在 job 文件夹下的 flume-file-hdfs.conf 文件。 说明:在使用 Spooling Directory Source 时,不要在监控目录中创建并持续修改文件(无法动态读取文件内容);上传完成的文件会以.COMPLETED 结尾(默认,可以修改);被监控文件夹每 500 毫秒扫描一次文件变动。 向 data 文件夹中添加文件 在/opt/flume 目录下创建 data 文件夹 [educoder@hadoop101 flume]$ mkdir data 向 data 文件夹中添加文件 [educoder@hadoop101 flume]$ touch 1.txt [educoder@hadoop101 flume]$ touch 2.txt [educoder@hadoop101 flume]$ touch 3.txt 查看 HDFS 上的数据 打开 HDFS 的 web 界面,到配置文件中指定的目录下查看 编程要求 根据提示,将 /opt/flume/flumedata 目录下的 aa.txt (平台已经帮你创建完毕) 采集到 Hdfs,你只需在Begin-End区域内配置配置文件即可,其余步骤后台会自动完成(平台后台向 aa.txt 文件中发送一条消息 From introduction to mastery,因为本实训中 source 源为 exec,学员在自己电脑环境中可以使用命令行发送,命令为echo " From introduction to mastery" >> /opt/flume/flumedata/aa.txt)。 aa.txt文件中的内容如下: Hadoop JAVA Flume 配置文件要求 : Agent命名为 a1 ; source选用 exec 采集所有文件; Channel选用 memery; 文件保存到 hdfs有如下要求 : hdfs路径名称: hdfs://localhost:9000; 保存到 flume 目录下; 文件前缀命名为 wcm; 文件后缀命名为 .wsy; 每 6s 回滚一次文件文件格式使用DataStream。 测试说明 本次评测最后查看 HDFS 中 /flume 目录是否存在传递过来的数据。 预期输出: Hadoop JAVA Flume From introduction to mastery
10-28
评论 1
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

10km

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值