How solve java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem in aws lambda?

huangapple 未分类评论45阅读模式
英文:

How solve java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem in aws lambda?

问题

在设置了hadoop配置之后,我得到了java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem的错误,即使org.apache.hadoop.fs.LocalFileSystem在jar文件中存在。

英文:

After setup the hadoop configuration I got java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem even when org.apache.hadoop.fs.LocalFileSystem is present in the jar file .

答案1

得分: 0

解决方案是从getClassByName函数中“移除”classloader。例如,创建带有重写方法getClassorg.apache.hadoop.conf.Configuration,如下所示:

Configuration hadoopConfig = new Configuration() {
    public Class<?> getClassByName(String name) throws ClassNotFoundException {
        return Class.forName(name);
    }
};

注:

  1. 这只是工作实现。原始实现使用Class.forName(name, true, classLoader);,因此如果需要,新的重写代码可能无法工作,如果需要,请添加if语句,仅为org.apache.hadoop.fs.LocalFileSystem更改逻辑。
  2. 错误仅在AWS Lambda上发生并得到修复。在其他环境(例如真实的Hadoop集群)中,这可能不是一个合适的解决方案。
英文:

Solution is to "remove" classloader from getClassByName function. E.g. create org.apache.hadoop.conf.Configuration with getClass overrided method, like:

Configuration hadoopConfig = new Configuration() {
    public Class&lt;?&gt; getClassByName(String name) throws ClassNotFoundException {
      return Class.forName(name);
    }

};

Notes

  1. This is just work implementation. Original implementation uses Class.forName(name, true, classLoader); so new overrided
    code may not work if so add if statement to change logic obly for org.apache.hadoop.fs.LocalFileSystem
  2. The error happened and fixed only for AWS lambda. In other environments (like true Hadoop cluster) this may
    not be an appropriate solution

huangapple
  • 本文由 发表于 2020年7月26日 16:42:03
  • 转载请务必保留本文链接:https://java.coder-hub.com/63097976.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定