英文:
Pyspark: An error occurred while trying to connect to the Java server
问题
我在Jupyter Notebook中启动了我的Pyspark会话：
spark = (SparkSession.builder.appName('myApp')
.config("spark.executor.memory", "4g")
.config("spark.executor.instances", "20")
.config("spark.master", "yarn")
.config("spark.driver.memory", "8g")
.config("spark.ui.port", "4373")
.config("spark.driver.maxResultSize", "1g")
.enableHiveSupport()
.getOrCreate())
但是后来出现了这个错误：
Py4JNetworkError: 在尝试连接到Java服务器时发生错误 (127.0.0.1:34032)
有时候 SparkSession 可能会正常启动，但之后当我在脚本上工作时会显示这个错误。
想知道如何解决这个问题。
英文:
I am starting my Pyspark session in JupyterNotebook:
spark = (SparkSession.builder.appName('myApp')
.config("spark.executor.memory", "4g")
.config("spark.executor.instances", "20")
.config("spark.master", "yarn")
.config("spark.driver.memory", "8g")
.config("spark.ui.port", "4373")
.config("spark.driver.maxResultSize", "1g")
.enableHiveSupport()
.getOrCreate())
But then got this error:
Py4JNetworkError: An error occurred while trying
to connect to the Java server (127.0.0.1:34032)
Sometimes the SparkSession would start fine but then later on when I am working on the script it shows the error.
Would like to know how to fix it.
专注分享java语言的经验与见解，让所有开发者获益！
评论