UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘s3‘

最新推荐文章于 2022-12-20 19:28:03 发布

原创最新推荐文章于 2022-12-20 19:28:03 发布 · 4.6k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#flink #hadoop #hdfs

Flink 专栏收录该内容

15 篇文章

订阅专栏

在使用Flink的yarn-session模式运行任务时遇到UnsupportedFileSystemSchemeException，问题在于缺少对S3的支持。解决方案是添加flink-s3-fs-hadoop和flink-s3-fs-presto插件到plugins目录下，并重启yarn-session。按照提示，将两个插件的jar文件分别复制到对应目录后，问题得到解决。

flink UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘s3’

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugins: flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:513)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
	at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:214)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)

yarn-session模式提交task，出现UnsupportedFileSystemSchemeException，
查看 flink-conf.yaml文件，参数fs.allowed-fallback-filesystems已经配置。
在这里插入图片描述
回看错误，发现以下提示信息

The scheme is directly supported by Flink through the following plugins: 
flink-s3-fs-hadoop, flink-s3-fs-presto.
 Please ensure that each plugin resides within its own subfolder within the plugins directory. 
 See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html

应该是在升级flink版本的时候，忘记配置flink的plugins，查看plugins目录，果然没有flink-s3-fs-hadoop, flink-s3-fs-presto
在这里插入图片描述
如何配置这些插件，好在提示信息中给了参考网址，打开对应的网址，我得到了如何配置的信息。

按提示配置了s3-fs-hadoop和 flink-s3-fs-presto插件，命令如下：

# s3-fs-hadoop
mkdir ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.13.1.jar ./plugins/s3-fs-hadoop/

#flink-s3-fs-presto
mkdir ./plugins/s3-fs-presto
cp ./opt/flink-s3-fs-presto-1.13.1.jar ./plugins/s3-fs-presto/

重启一下yarn-session, 然后重新提交task，没有错误了。

在这里插入图片描述