spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Debug Spark Streaming in PyCharm
Date Sat, 11 Jul 2015 02:34:26 GMT
spark-submit does a lot of magic configurations (classpaths etc) underneath
the covers to enable pyspark to find Spark JARs, etc. I am not sure how you
can start running things directly from the PyCharm IDE. Others in the
community may be able to answer. For now the main way to run pyspark stuff
is through spark-submit, or pyspark (which uses spark-submit underneath).

On Fri, Jul 10, 2015 at 6:28 AM, blbradley <> wrote:

> Hello,
> I'm trying to debug a PySpark app with Kafka Streaming in PyCharm. However,
> PySpark cannot find the jar dependencies for Kafka Streaming without
> editing
> the program. I can temporarily use SparkConf to set 'spark.jars', but I'm
> using Mesos for production and don't want to edit my program everytime I
> want to debug. I'd like to find a way to debug without editing the source.
> Here's what my PyCharm debug execution command looks like:
> home/brandon/.pyenv/versions/coinspark/bin/python2.7
> /opt/pycharm-community/helpers/pydev/ --multiproc --client
> --port 59042 --file
> /home/brandon/src/coins/coinspark/
> I might be able to use spark-submit has the command PyCharm runs, but I'm
> not sure if that will work with the debugger.
> Thoughts?
> Cheers!
> Brandon Bradley
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message