flink系列1 程序win上启动

最新推荐文章于 2024-11-02 22:14:34 发布
dataastron
最新推荐文章于 2024-11-02 22:14:34 发布
阅读量1k
点赞数
CC 4.0 BY-SA版权
分类专栏：大数据
本文链接：https://blog.youkuaiyun.com/dataastron/article/details/79035107
大数据专栏收录该内容
22 篇文章
订阅专栏
该日志记录了Apache Flink JobManager在本地模式下启动的过程，包括配置加载、资源分配、网络组件初始化等关键步骤。启动过程中遇到Hadoop winutils二进制文件缺失错误。
摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >
这里写图片描述
2018-01-11 14:30:24,595 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
2018-01-11 14:30:24,596 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Starting JobManager (Version: 1.4.0, Rev:3a9d9f2, Date:06.12.2017 @ 11:08:40 UTC)
2018-01-11 14:30:24,596 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  OS current user: astron
2018-01-11 14:30:24,797 ERROR org.apache.hadoop.util.Shell                                  - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.runtime.util.EnvironmentInformation.getHadoopUser(EnvironmentInformation.java:96)
    at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:285)
    at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1904)
    at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2018-01-11 14:30:24,808 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-01-11 14:30:24,863 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Current Hadoop/Kerberos user: astron
2018-01-11 14:30:24,863 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.151-b12
2018-01-11 14:30:24,863 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Maximum heap size: 736 MiBytes
2018-01-11 14:30:24,863 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JAVA_HOME: C:\Program Files\Java\jdk1.8.0_151
2018-01-11 14:30:24,865 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Hadoop version: 2.7.3
2018-01-11 14:30:24,865 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM Options:
2018-01-11 14:30:24,865 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xms768m
2018-01-11 14:30:24,865 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xmx768m
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog.file=D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\log\flink-astron-jobmanager-CDYJY-astronPC.log
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlogback.configurationFile=file:D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\conf/logback.xml
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog4j.configuration=file:D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\conf/log4j.properties
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Program Arguments:
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --configDir
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\conf
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --executionMode
2018-01-11 14:30:24,866 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     local
2018-01-11 14:30:24,867 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Classpath: D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\lib\flink-dist_2.11-1.4.0.jar;D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\lib\flink-python_2.11-1.4.0.jar;D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\lib\flink-shaded-hadoop2-uber-1.4.0.jar;D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\lib\log4j-1.2.17.jar;D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\lib\slf4j-log4j12-1.7.7.jar;
2018-01-11 14:30:24,867 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
2018-01-11 14:30:24,868 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Registered UNIX signal handlers for [TERM, INT]
2018-01-11 14:30:24,962 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Loading configuration from D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\conf
2018-01-11 14:30:24,965 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-01-11 14:30:24,965 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-01-11 14:30:24,965 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 1024
2018-01-11 14:30:24,965 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 1024
2018-01-11 14:30:24,965 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-01-11 14:30:24,966 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2018-01-11 14:30:24,966 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-01-11 14:30:24,966 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: web.port, 8081
2018-01-11 14:30:24,988 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager without high-availability
2018-01-11 14:30:24,990 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager on localhost:6123 with execution mode LOCAL
2018-01-11 14:30:25,020 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to astron (auth:SIMPLE)
2018-01-11 14:30:25,154 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Trying to start actor system at localhost:6123
2018-01-11 14:30:25,421 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2018-01-11 14:30:25,749 INFO  akka.remote.Remoting                                          - Starting remoting
2018-01-11 14:30:25,875 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@localhost:6123]
2018-01-11 14:30:25,880 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Actor system started at akka.tcp://flink@localhost:6123
2018-01-11 14:30:25,893 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
2018-01-11 14:30:25,897 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager web frontend
2018-01-11 14:30:25,906 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of JobManager log file: D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\log\flink-astron-jobmanager-CDYJY-astronPC.log
2018-01-11 14:30:25,906 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of JobManager stdout file: D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0\bin\..\log\flink-astron-jobmanager-CDYJY-astronPC.out
2018-01-11 14:30:25,907 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory C:\Users\astron\AppData\Local\Temp\flink-web-ad3b3ad8-6065-492a-a629-3285bdb7048b for the web interface files
2018-01-11 14:30:25,908 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory C:\Users\astron\AppData\Local\Temp\flink-web-f04857c0-59e1-4be8-b594-af61a0926bf5 for web frontend JAR file uploads
2018-01-11 14:30:26,121 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend listening at localhost:8081
2018-01-11 14:30:26,122 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor
2018-01-11 14:30:26,128 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory C:\Users\astron\AppData\Local\Temp\blobStore-bb1ca34d-2f58-44e1-97dc-2ca033c0d4a1
2018-01-11 14:30:26,130 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:62101 - max concurrent requests: 50 - max backlog: 1000
2018-01-11 14:30:26,171 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting embedded TaskManager for JobManager's LOCAL execution mode
2018-01-11 14:30:26,172 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started memory archivist akka://flink/user/archive
2018-01-11 14:30:26,174 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager at akka.tcp://flink@localhost:6123/user/jobmanager.
2018-01-11 14:30:26,180 INFO  org.apache.flink.runtime.jobmanager.JobManager                - JobManager akka.tcp://flink@localhost:6123/user/jobmanager was granted leadership with leader session ID Some(00000000-0000-0000-0000-000000000000).
2018-01-11 14:30:26,181 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig [server address: localhost/127.0.0.1, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 1 (manual), number of client threads: 1 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2018-01-11 14:30:26,184 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration  - Messages have a max timeout of 10000 ms
2018-01-11 14:30:26,187 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Temporary file directory 'C:\Users\astron\AppData\Local\Temp': total 118 GB, usable 52 GB (44.07% usable)
2018-01-11 14:30:26,322 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 72 MB for network buffer pool (number of memory segments: 2306, bytes per segment: 32768).
2018-01-11 14:30:26,336 WARN  org.apache.flink.runtime.query.QueryableStateUtils            - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. Please put the corresponding jar from the opt to the lib folder.
2018-01-11 14:30:26,336 WARN  org.apache.flink.runtime.query.QueryableStateUtils            - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. Please put the corresponding jar from the opt to the lib folder.
2018-01-11 14:30:26,337 INFO  org.apache.flink.runtime.io.network.NetworkEnvironment        - Starting the network environment and its components.
2018-01-11 14:30:26,345 INFO  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful initialization (took 4 ms).
2018-01-11 14:30:26,348 INFO  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful initialization (took 3 ms). Listening on SocketAddress /127.0.0.1:62106.
2018-01-11 14:30:26,489 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Limiting managed memory to 0.7 of the currently free heap space (460 MB), memory will be allocated lazily.
2018-01-11 14:30:26,494 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory C:\Users\astron\AppData\Local\Temp\flink-io-4371bfee-efa7-4c35-b7bf-4952bbbb49b1 for spill files.
2018-01-11 14:30:26,499 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory C:\Users\astron\AppData\Local\Temp\flink-dist-cache-90a4b18a-a40f-4633-a27a-6c84dacd1ceb
2018-01-11 14:30:26,510 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory C:\Users\astron\AppData\Local\Temp\flink-dist-cache-93b68834-c782-4766-aea7-6882e40cc0e5
2018-01-11 14:30:26,516 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor at akka://flink/user/taskmanager#-1317514981.
2018-01-11 14:30:26,516 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager data connection information: 48b153b53a217da7885d2ca8f8a102bb @ localscen.risk.jd.com (dataPort=62106)
2018-01-11 14:30:26,517 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager has 1 task slot(s).
2018-01-11 14:30:26,517 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Memory usage stats: [HEAP: 96/736/736 MB, NON HEAP: 39/39/-1 MB (used/committed/max)]
2018-01-11 14:30:26,518 INFO  org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager  - Trying to associate with JobManager leader akka.tcp://flink@localhost:6123/user/jobmanager
2018-01-11 14:30:26,520 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Trying to register at JobManager akka.tcp://flink@localhost:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2018-01-11 14:30:26,522 INFO  org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager  - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#-1749419493] - leader session 00000000-0000-0000-0000-000000000000
2018-01-11 14:30:26,522 INFO  org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager  - TaskManager 48b153b53a217da7885d2ca8f8a102bb has started.
2018-01-11 14:30:26,524 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at localscen (akka://flink/user/taskmanager) as 5e42b32cf44ae20c6400962296261621. Current number of registered hosts is 1. Current number of alive task slots is 1.
2018-01-11 14:30:26,525 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Successful registration at JobManager (akka://flink/user/jobmanager), starting network stack and library cache.
2018-01-11 14:30:26,527 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Determined BLOB server address to be localhost/127.0.0.1:62101. Starting BLOB cache.
2018-01-11 14:30:26,529 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Created BLOB cache storage directory C:\Users\astron\AppData\Local\Temp\blobStore-2a3d34f5-5506-4fcd-9c22-c2186ea19952
2018-01-11 14:30:26,531 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory C:\Users\astron\AppData\Local\Temp\blobStore-7b11a823-bb0b-4961-a2a6-01f4cf602195