spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: How do you perform blocking IO in apache spark job?
Date Mon, 08 Sep 2014 18:22:13 GMT

So the external service itself creates threads and blocks until they
finished execution? In this case you should not do threading but include it
via jni directly in spark - it will take care about threading for you.

Vest regards
Hi, Jörn, first of all, thanks for you intent to help.

This one external service is a native component, that is stateless and that
performs the calculation based on the data I provide. The data is in RDD.

That one component I have on each worker node and I would like to get as
much parallelism as possible on a single worker node.
Using scala future I can get it, at least as much threads, as my machine
allows me. But how to do the same on spark? Is there a possibility to cal
that native component on each worker in multiple threads?

Thanks in advance.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message