英文:
Pyspark execution issue
问题
当我尝试运行pyspark脚本时，出现以下错误：
(base) 🦁 spark-submit SparkLinearRegression.py
20/05/05 06:52:01 WARN NativeCodeLoader: 无法为您的平台加载native-hadoop库...在适用的情况下使用内置的Java类
Traceback (most recent call last):
File "/Users/koke/anaconda3/bin/jupyter", line 10, in <module>
sys.exit(main())
File "/Users/koke/anaconda3/lib/python3.7/site-packages/jupyter_core/command.py", line 230, in main
command = _jupyter_abspath(subcommand)
File "/Users/koke/anaconda3/lib/python3.7/site-packages/jupyter_core/command.py", line 133, in _jupyter_abspath
'找不到Jupyter命令`{}`。'.format(jupyter_subcommand)
Exception: 找不到Jupyter命令`jupyter-/Users/koke/Data/MLCourse/SparkLinearRegression.py`。
log4j:WARN 无法找到记录器的附加器（org.apache.spark.util.ShutdownHookManager）。
log4j:WARN 请正确初始化log4j系统。
log4j:WARN 详细信息请参阅http://logging.apache.org/log4j/1.2/faq.html#noconfig。
我已经将bash_profile中的默认JDK从java10更改为java8，但没有起作用。请帮忙解决。
英文:
when I am trying to run pyspark script it gives me following error
(base) 🦁 spark-submit SparkLinearRegression.py
20/05/05 06:52:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
File "/Users/koke/anaconda3/bin/jupyter", line 10, in <module>
sys.exit(main())
File "/Users/koke/anaconda3/lib/python3.7/site-packages/jupyter_core/command.py", line 230, in main
command = _jupyter_abspath(subcommand)
File "/Users/koke/anaconda3/lib/python3.7/site-packages/jupyter_core/command.py", line 133, in _jupyter_abspath
'Jupyter command `{}` not found.'.format(jupyter_subcommand)
Exception: Jupyter command `jupyter-/Users/koke/Data/MLCourse/SparkLinearRegression.py` not found.
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I have changed to java8 instead of java10 as default jdk in bash_profile, didn't work.
Please help..
专注分享java语言的经验与见解，让所有开发者获益！
评论