spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rico Bergmann <i...@ricobergmann.de>
Subject Strange codegen error for SortMergeJoin in Spark 2.2.1
Date Tue, 05 Jun 2018 10:58:41 GMT
Hi!

I get a strange error when executing a complex SQL-query involving 4
tables that are left-outer-joined:

Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37,
Column 18: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java',
Line 37, Column 18: No applicable constructor/method found for actual parameters "int"; candidates
are: "org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(org.apache.spark.memory.TaskMemoryManager,
org.apache.spark.storage.BlockManager, org.apache.spark.serializer.SerializerManager, org.apache.spark.TaskContext,
int, long, int, int)", "org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(int,
int)"

...

/* 037 */     smj_matches = new org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(2147483647);

The same query works with Spark 2.2.0.

I checked the Spark source code and saw that in
ExternalAppendOnlyUnsafeRowArray a second int was introduced into the
constructor in 2.2.1

But looking at the codegeneration part of SortMergeJoinExec:

// A list to hold all matched rows from right side. val matches = ctx.freshName("matches")
val clsName = classOf[ExternalAppendOnlyUnsafeRowArray].getName

val spillThreshold = getSpillThreshold
val inMemoryThreshold = getInMemoryThreshold

ctx.addMutableState(clsName, matches, s"$matches= new $clsName($inMemoryThreshold, $spillThreshold);")

it should get 2 parameters, not just one.


May be anyone has an idea?


Best,

Rico.


Mime
View raw message