I am still working on making a minimal test that I can share without my work-specific code being in there. However, the problem occurs with a dataframe with several hundred columns being asked to do a tension split. Random split works with up to about 350 columns so far. It breaks in my code with 600 columns, but it's a converted dataset of case classes to dataframe. This is deterministically causing the error in Scala 2.11.
Once I can get a deterministically breaking test without work code I will try to file a Jira bug.On Tue, Aug 16, 2016, 04:17 Ted Yu <firstname.lastname@example.org> wrote:I think we should reopen it.
On Aug 16, 2016, at 1:48 AM, Kazuaki Ishizaki <ISHIZAKI@jp.ibm.com> wrote:I just realized it since it broken a build with Scala 2.10.
spark/commit/ fa244e5a90690d6a31be50f2aa203a e1a2e9a1cf
I can reproduce the problem in SPARK-15285 with master branch.
Should we reopen SPARK-15285?
From: Ted Yu <email@example.com>
To: dhruve ashar <firstname.lastname@example.org>
Cc: Aris <email@example.com>, "firstname.lastname@example.org" <email@example.com>
Date: 2016/08/15 06:19
Subject: Re: Spark 2.0.0 JaninoRuntimeException
Looks like the proposed fix was reverted:
Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB"
This reverts commit fa244e5a90690d6a31be50f2aa203a
Maybe this was fixed in some other JIRA ?
On Fri, Aug 12, 2016 at 2:30 PM, dhruve ashar <firstname.lastname@example.org> wrote:
I see a similar issue being resolved recently: https://issues.
On Fri, Aug 12, 2016 at 3:33 PM, Aris <email@example.com> wrote:
I'm on Spark 2.0.0 working with Datasets -- and despite the fact that smaller data unit tests work on my laptop, when I'm on a cluster, I get cryptic error messages:
Caused by: org.codehaus.janino.
JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/ catalyst/InternalRow;Lorg/ apache/spark/sql/catalyst/ InternalRow;)I" of class "org.apache.spark.sql. catalyst.expressions. GeneratedClass$ SpecificOrdering" grows beyond 64 KB
Unfortunately I'm not clear on how to even isolate the source of this problem. I didn't have this problem in Spark 1.6.1.