flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: questions about Flink's HashJoin performance
Date Mon, 15 May 2017 10:20:43 GMT
Hi Weijie,

it might be the case that batching the processing of multiple rows can give
you an improved performance compared to single row processing.

Maybe you could share the exact benchmark base line results and the code
you use to test Flink's MutableHashTable with us. Also the Flink
configuration and how you run it would be of interest. That way we might be
able to see if we can tune Flink a bit more.


On Sun, May 14, 2017 at 5:23 AM, weijie tong <tongweijie178@gmail.com>

> I has a test case to use Flink's MutableHashTable class to do a hash join
> on a local machine with 64g memory, 64cores. The test case is one build
> table with 14w rows ,one probe table with 320w rows ,the matched result
> rows is 12 w.
> It takes 2.2 seconds to complete the join.The performance seems bad. I
> ensure there's no overflow, the smaller table is the build side. The
> MutableObjectIterator is a sequence of Rows. The Row is composed of several
> fields which are byte[]. Through my log,I find the open() method takes
> 1.560 seconds. The probe iterates phase takes 680ms.  And my Jprofiler's
> profile shows the MutableObjectIterator's next() method call is the
> hotspot.
> I want to know how to tune this scenario. I find Drill's HashJoin is batch
> model. Its build side's input is a RecordBatch which holds batch of rows
> and memory size is approach to L2 cache. Through this strategy it will gain
> less method calls (that means call to next() ) and much efficient to cpu
> calculation.  I also find SQL server's paper noticed the batch model's
> performance gains (https://www.microsoft.com/en-us/research/wp-content/
> uploads/2013/06/Apollo3-Sigmod-2013-final.pdf)  .   I guess the
> performance's down is due to the single row iterate model.
> Hope someone to correct my opinion. Also maybe I have a wrong use  of the
> MutableHashTable. wait for someone to give an advice.

View raw message