spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: Performance - Python streaming v/s Scala streaming
Date Mon, 24 Aug 2015 20:35:15 GMT
The scala version of the Kafka  is something that we have been working on
for a while, and is likely to be more optimized than the python one. The
python one definitely requires pass the data back and forth between JVM and
Python VM and decoding the raw bytes to the Python strings (probably less
efficient that Java's Byte to UTF8 decoder), so that may cause some extra
overheads compared to scala.

Also consider trying the direct API. Read more in the Kafka integration
guide - http://spark.apache.org/docs/latest/streaming-kafka-integration.html
That overall has a much higher throughput that the earlier receiver based
approach.

BTW, disclaimer. Do not consider this difference as generalization of the
performance difference between Scala and Python for all of Spark, For
example, DataFrames provide performance parity between Scala and Python
APIs.


On Mon, Aug 24, 2015 at 5:22 AM, utk.pat <utkarsh.patkar@gmail.com> wrote:

> I am new to SPARK streaming. I was running the "kafka_wordcount" example
> with a local KAFKA and SPARK instance. It was very easy to set this up and
> get going :) I tried running both SCALA and Python versions of the word
> count example. Python versions seems to be extremely slow. Sometimes it has
> delays of more than couple of minutes. On the other hand SCALA versions
> seems to be way better. I am running on a windows machine. I am trying to
> understand what is the cause slowness in python streaming? Is there
> anything that I am missing? For real time streaming analysis should I
> prefer SCALA?
> ------------------------------
> View this message in context: Performance - Python streaming v/s Scala
> streaming
> <http://apache-spark-user-list.1001560.n3.nabble.com/Performance-Python-streaming-v-s-Scala-streaming-tp24415.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>

Mime
View raw message