spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: Support for Second level of concurrency
Date Tue, 25 Sep 2018 16:53:30 GMT
What is the ultimate goal of this algorithm?  There could be already algorithms that can do
this within Spark. You could also put a message on Kafka (or another broker) and have spark
applications listen to them to trigger further computation. This would be also more controlled
and can be done already now.

> On 25. Sep 2018, at 17:31, sandeep mehandru <> wrote:
> Hi Folks,
>   There is a use-case , where we are doing large computation on two large
> vectors. It is basically a scenario, where we run a flatmap operation on the
> Left vector and run co-relation logic by comparing it with all the rows of
> the second vector. When this flatmap operation is running on an executor,
> this compares row 1 from left vector with all rows of the second vector. The
> goal is that from this flatmap operation, we want to start another remote
> map operation that compares a portion of right vector rows. This enables a
> second level of concurrent operation, thereby increasing throughput and
> utilizing other nodes. But to achieve this we need access to spark context
> from within the Flatmap operation.
> I have attached a snapshot describing the limitation.
> <>

> In simple words, this boils down to having access to  a spark context from
> within an executor , so that the next level of map or concurrent operations
> can be spun on the partitions on other machines. I have some experience with
> other in-memory compute grids technologies like Coherence, Hazelcast. This
> frameworks do allow to trigger next level of concurrent operations from
> within a task being executed on one node.
> Regards,
> Sandeep.
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

To unsubscribe e-mail:

View raw message