spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Walker <>
Subject Re: bulk upload to Elasticsearch and shuffle behavior
Date Tue, 01 Sep 2015 03:05:19 GMT
I think I have found out what was causing me difficulties.  It seems I was
reading too much into the stage description shown in the "Stages" tab of
the Spark application UI.  While it said "repartition at", I can infer from the network traffic and
from its response to changes I subsequently made that the actual code that
was running was the code doing the HBase lookups.  I suspect the actual
shuffle, once it occurred, required on the same order of network IO as the
upload to Elasticsearch that followed.


On Mon, Aug 31, 2015 at 6:09 PM, Eric Walker <> wrote:

> Hi,
> I am working on a pipeline that carries out a number of stages, the last
> of which is to build some large JSON objects from information in the
> preceding stages.  The JSON objects are then uploaded to Elasticsearch in
> bulk.
> If I carry out a shuffle via a `repartition` call after the JSON documents
> have been created, the upload to ES is fast.  But the shuffle itself takes
> many tens of minutes and is IO-bound.
> If I omit the repartition, the upload to ES takes a long time due to a
> complete lack of parallelism.
> Currently, the step that precedes the assembling of the JSON documents,
> which goes into the final repartition call, is the querying of pairs of
> object ids.  In a mapper the ids are resolved to documents by querying
> HBase.  The initial pairs of ids are obtained via a query against the SQL
> context, and the query result is repartitioned before going into the mapper
> that resolves the ids into documents.
> It's not clear to me why the final repartition preceding the upload to ES
> is required.  I would like to omit it, since it is so expensive and
> involves so much network IO, but have not found a way to do this yet.  If I
> omit the repartition, the job takes much longer.
> Does anyone know what might be going on here, and what I might be able to
> do to get rid of the last `repartition` call before the upload to ES?
> Eric

View raw message