首页 分享 Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, maste

Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, maste

来源:花匠小妙招 时间:2024-12-12 13:50

最新推荐文章于 2024-09-09 07:30:00 发布

Rachel_nana 于 2019-08-28 16:13:00 发布

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。

运行SparkContext报错:

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /usr/local/spark/python/pyspark/shell.py:59 

hadoop@rachel-virtual-machine:/usr/local/spark/bin$ ./pyspark

./pyspark: 行 45: python: 未找到命令

Python 3.6.8 (default, Jan 14 2019, 11:02:34)

[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux

Type "help", "copyright", "credits" or "license" for more information.

2019-08-28 15:27:12 WARN Utils:66 - Your hostname, rachel-virtual-machine resolves to a loopback address: 127.0.1.1; using 192.168.80.128 instead (on interface ens33)

2019-08-28 15:27:12 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address

2019-08-28 15:27:23 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Welcome to

____ __

/ __/__ ___ _____/ /__

_ / _ / _ `/ __/ '_/

/__ / .__/_,_/_/ /_/_ version 2.3.3

/_/

Using Python version 3.6.8 (default, Jan 14 2019 11:02:34)

SparkSession available as 'spark'.

>>> from pyspark import SparkContext

>>> sc = SparkContext( 'local', 'test')

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/usr/local/spark/python/pyspark/context.py", line 129, in __init__

SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)

File "/usr/local/spark/python/pyspark/context.py", line 328, in _ensure_initialized

callsite.function, callsite.file, callsite.linenum))

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /usr/local/spark/python/pyspark/shell.py:59

出现这个错误是因为之前已经启动了SparkContext,所以需要先关闭spark,然后再启动。

>>> sc.stop()

>>> sc=SparkContext("local","test")

>>> logFile = "file:///usr/local/spark/README.md"

>>> logData = sc.textFile(logFile, 2).cache()

>>> numAs = logData.filter(lambda line: 'a' in line).count()

>>> numBs = logData.filter(lambda line: 'b' in line).count()

>>> print('Lines with a: %s, Lines with b: %s' % (numAs, numBs))

Lines with a: 61, Lines with b: 30

相关知识

成功解决报错Cannot find class [com.alibaba.druid.pool.DruidDataSource] for bean with name ‘dataSource‘
Operator '==' cannot be applied to错误
cannot open resource self.font = core.getfont(font, size, index, encoding, layout
no such file or directory
python中x y的意思,在Python中x ='y''z'的含义是什么?
【深度学习TPU+Keras+Tensorflow+EfficientNetB7】kaggle竞赛 使用TPU对104种花朵进行分类 第十八次尝试 99.9%准确率
考研英语作文:温室花朵经不起考验
Honeywell Air Purifier
2003年考研英语一作文温室里地花朵经不起风雨.pdf
基于神经结构搜索的多种植物叶片病害识别

网址: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, maste https://www.huajiangbk.com/newsview1055193.html

所属分类:花卉
上一篇: 幼儿园活动计划方案(15篇)
下一篇: 动物传播的植物有哪些

推荐分享