英文:
JSONPATH - java.lang.NoSuchFieldError: defaultReader
问题
我正在尝试在我的Spark RDD操作中使用Jayaway JSONPATH。（Spark版本1.6.3）
JsonPath版本2.0.0
在运行时，我遇到了以下堆栈跟踪。
java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:179)
at com.virtualpairprogrammers.JavaIntroduction.returnJsonData(JavaIntroduction.java:115)
at com.virtualpairprogrammers.JavaIntroduction$1.call(JavaIntroduction.java:58)
at com.virtualpairprogrammers.JavaIntroduction$1.call(JavaIntroduction.java:1)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1015)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
...
当我在Eclipse中运行它时，只有使用JSONPATH进行json数据提取时，结果是正确的。
当我使用Spark submit时，我会收到上述错误消息。
链接已经阅读了，但与我的用例不太相关。
如果有人遇到过这个问题并解决了它，请协助我继续前进。
以下是我正在使用的代码片段，
JavaRDD<Document> rdd = MongoSpark.load(jsc);
JavaRDD<String> fullFile = rdd.map(new Function<Document, String>() {
public String call(Document s) {
String values="";
Configuration conf1 = Configuration.defaultConfiguration().addOptions(Option.SUPPRESS_EXCEPTIONS);
ReadContext ctx = JsonPath.using(conf1).parse(s.toJson());
for (int i = 0; i < charsetNameArray.length;i++) {
values=values+"|"+ctx.read("$."+charsetNameArray[i]);
}
return values;
}
});
此外，我还尝试从Eclipse中运行Json path，对我来说一切正常。但通过Spark Submit运行时会失败。
提前感谢
致意
英文:
I am trying to use Jayaway JSONPATH with my Spark RDD operations. (spark version 1.6.3)
JsonPath version 2.0.0
I am getting the following stack trace when running.
java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:179)
at com.virtualpairprogrammers.JavaIntroduction.returnJsonData(JavaIntroduction.java:115)
at com.virtualpairprogrammers.JavaIntroduction$1.call(JavaIntroduction.java:58)
at com.virtualpairprogrammers.JavaIntroduction$1.call(JavaIntroduction.java:1)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1015)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1335)
at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1335)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1857)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1857)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
When i run it in my eclipse , only the json data extraction using the JSONPATH , i am getting the results properly.
When i am using Spark submit, i am getting the above error message.
Link read it , but was not that related to my usecase.
If anyone has faced this issue and fixed it please do assist me to move forward.
Following is the Code Snippet , i am using,
JavaRDD<Document> rdd = MongoSpark.load(jsc);
JavaRDD<String> fullFile = rdd.map(new Function<Document, String>() {
public String call(Document s) {
String values="";
Configuration conf1 = Configuration.defaultConfiguration().addOptions(Option.SUPPRESS_EXCEPTIONS);
ReadContext ctx = JsonPath.using(conf1).parse(s.toJson());
for (int i = 0; i < charsetNameArray.length;i++) {
values=values+"|"+ctx.read("$."+charsetNameArray[i]);
}
return values;
}
});
Also i have tried running it Json path from eclipse, it is working fine for me. When running through Spark Submit is when it fails.
Thanks in Advance
Regards
答案1
得分: 0
我尝试了很多，我认为 Jsonpath 与 Hadoop 2.7 及更低版本不兼容。以下是相关的链接：Reference
使用 JsonFlatten 进行扁平化，然后在 RDD 中使用 Map 函数将其转换为可读或可用的 RDD。
以下是相应的 Maven 依赖项：
<dependency>
<groupId>com.github.wnameless</groupId>
<artifactId>json-flattener</artifactId>
<version>0.2.2</version>
</dependency>
英文:
I tried a lot , and i think Jsonpath is not compatible with Hadoop 2.7 and below.
Below is the Link for the reference :-
Reference
Used the JsonFlatten to flatten and then use Map function in RDD to convert into readable or usable RDD.
Below is the Maven Dependency for the same.
<dependency>
<groupId>com.github.wnameless</groupId>
<artifactId>json-flattener</artifactId>
<version>0.2.2</version>
</dependency>
专注分享java语言的经验与见解，让所有开发者获益！
评论