Pyspark:尝试连接Java服务器时出现错误。

huangapple 未分类评论43阅读模式
英文:

Pyspark: An error occurred while trying to connect to the Java server

问题

我在Jupyter Notebook中启动了我的Pyspark会话:

spark = (SparkSession.builder.appName('myApp')
         .config("spark.executor.memory", "4g")
         .config("spark.executor.instances", "20")
         .config("spark.master", "yarn")
         .config("spark.driver.memory", "8g")
         .config("spark.ui.port", "4373")
         .config("spark.driver.maxResultSize", "1g")
         .enableHiveSupport()
        .getOrCreate())

但是后来出现了这个错误:

Py4JNetworkError: 在尝试连接到Java服务器时发生错误 (127.0.0.1:34032)

有时候 SparkSession 可能会正常启动,但之后当我在脚本上工作时会显示这个错误。
想知道如何解决这个问题。

英文:

I am starting my Pyspark session in JupyterNotebook:

spark = (SparkSession.builder.appName('myApp')
         .config("spark.executor.memory", "4g")
         .config("spark.executor.instances", "20")
         .config("spark.master", "yarn")
         .config("spark.driver.memory", "8g")
         .config("spark.ui.port", "4373")
         .config("spark.driver.maxResultSize", "1g")
         .enableHiveSupport()
        .getOrCreate())

But then got this error:

Py4JNetworkError: An error occurred while trying 
to connect to the Java server (127.0.0.1:34032)

Sometimes the SparkSession would start fine but then later on when I am working on the script it shows the error.
Would like to know how to fix it.

huangapple
  • 本文由 发表于 2020年7月26日 09:18:56
  • 转载请务必保留本文链接:https://java.coder-hub.com/63095187.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定