hadoop job error-1

最新推荐文章于 2024-03-16 20:06:47 发布

原创最新推荐文章于 2024-03-16 20:06:47 发布 · 4.1k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #pig #file

hadoop 同时被 2 个专栏收录

63 篇文章

订阅专栏

pig

3 篇文章

订阅专栏

本文记录了一次使用Pig处理数据时遇到的错误情况：在尝试执行存储操作时发生异常，提示文件创建失败，因为已有客户端尝试创建同一文件。但稍后再次执行相同的Pig查询时，却能够成功运行。

在使用pig处理数据时，red程序报错：
job_201110171502_1932

org.apache.pig.backend.executionengine.ExecException: ERROR 2081: Unable to setup the store function.
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:111)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:385)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:261)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:648)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /data/dm/uid_active_vv/user_active_vv_20110901_30/_temporary/_attempt_201110171502_1932_r_000000_0/part-r-00000 for DFSClient_attempt_201110171502_1932_r_000000_0 on client 111.161.25.189 because current leaseholder is trying to recreate file.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1193)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1123)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:551)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1383)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1379)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1377)

    at org.apache.hadoop.ipc.Client.call(Client.java:1030)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
    at $Proxy2.create(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy2.create(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2862)
    at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:539)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:206)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:528)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:509)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:416)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter(PigTextOutputFormat.java:98)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createStoreFunc(MapReducePOStoreImpl.java:85)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:103)
    ... 9 more

但是稍后再次执行同样的pig查询时，却执行成功了，问题原因尚未定位？