英文:
How solve java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem in aws lambda?
问题
在设置了hadoop配置之后,我得到了java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem
的错误,即使org.apache.hadoop.fs.LocalFileSystem
在jar文件中存在。
英文:
After setup the hadoop configuration I got java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem
even when org.apache.hadoop.fs.LocalFileSystem
is present in the jar file .
答案1
得分: 0
解决方案是从getClassByName
函数中“移除”classloader
。例如,创建带有重写方法getClass
的org.apache.hadoop.conf.Configuration
,如下所示:
Configuration hadoopConfig = new Configuration() {
public Class<?> getClassByName(String name) throws ClassNotFoundException {
return Class.forName(name);
}
};
注:
- 这只是工作实现。原始实现使用
Class.forName(name, true, classLoader);
,因此如果需要,新的重写代码可能无法工作,如果需要,请添加if
语句,仅为org.apache.hadoop.fs.LocalFileSystem
更改逻辑。 - 错误仅在AWS Lambda上发生并得到修复。在其他环境(例如真实的Hadoop集群)中,这可能不是一个合适的解决方案。
英文:
Solution is to "remove" classloader
from getClassByName
function. E.g. create org.apache.hadoop.conf.Configuration
with getClass
overrided method, like:
Configuration hadoopConfig = new Configuration() {
public Class<?> getClassByName(String name) throws ClassNotFoundException {
return Class.forName(name);
}
};
Notes
- This is just work implementation. Original implementation uses
Class.forName(name, true, classLoader);
so new overrided
code may not work if so addif
statement to change logic obly fororg.apache.hadoop.fs.LocalFileSystem
- The error happened and fixed only for AWS lambda. In other environments (like true Hadoop cluster) this may
not be an appropriate solution
专注分享java语言的经验与见解,让所有开发者获益!
评论