spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Spark 2.0.0 JaninoRuntimeException
Date Tue, 16 Aug 2016 15:56:20 GMT
Can you take a look at commit fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf ?

There was a test:
SPARK-15285 Generated SpecificSafeProjection.apply method grows beyond 64KB

See if it matches your use case.

On Tue, Aug 16, 2016 at 8:41 AM, Aris <arisofalaska@gmail.com> wrote:

> I am still working on making a minimal test that I can share without my
> work-specific code being in there. However, the problem occurs with a
> dataframe with several hundred columns being asked to do a tension split.
> Random split works with up to about 350 columns so far. It breaks in my
> code with 600 columns, but it's a converted dataset of case classes to
> dataframe. This is deterministically causing the error in Scala 2.11.
>
> Once I can get a deterministically breaking test without work code I will
> try to file a Jira bug.
>
> On Tue, Aug 16, 2016, 04:17 Ted Yu <yuzhihong@gmail.com> wrote:
>
>> I think we should reopen it.
>>
>> On Aug 16, 2016, at 1:48 AM, Kazuaki Ishizaki <ISHIZAKI@jp.ibm.com>
>> wrote:
>>
>> I just realized it since it broken a build with Scala 2.10.
>> https://github.com/apache/spark/commit/fa244e5a90690d6a31be50f2aa203a
>> e1a2e9a1cf
>>
>> I can reproduce the problem in SPARK-15285 with master branch.
>> Should we reopen SPARK-15285?
>>
>> Best Regards,
>> Kazuaki Ishizaki,
>>
>>
>>
>> From:        Ted Yu <yuzhihong@gmail.com>
>> To:        dhruve ashar <dhruveashar@gmail.com>
>> Cc:        Aris <arisofalaska@gmail.com>, "user@spark.apache.org" <
>> user@spark.apache.org>
>> Date:        2016/08/15 06:19
>> Subject:        Re: Spark 2.0.0 JaninoRuntimeException
>> ------------------------------
>>
>>
>>
>> Looks like the proposed fix was reverted:
>>
>>     Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply
>> method grows beyond 64 KB"
>>
>>     This reverts commit fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf.
>>
>> Maybe this was fixed in some other JIRA ?
>>
>> On Fri, Aug 12, 2016 at 2:30 PM, dhruve ashar <*dhruveashar@gmail.com*
>> <dhruveashar@gmail.com>> wrote:
>> I see a similar issue being resolved recently:
>> *https://issues.apache.org/jira/browse/SPARK-15285*
>> <https://issues.apache.org/jira/browse/SPARK-15285>
>>
>> On Fri, Aug 12, 2016 at 3:33 PM, Aris <*arisofalaska@gmail.com*
>> <arisofalaska@gmail.com>> wrote:
>> Hello folks,
>>
>> I'm on Spark 2.0.0 working with Datasets -- and despite the fact that
>> smaller data unit tests work on my laptop, when I'm on a cluster, I get
>> cryptic error messages:
>>
>> Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method
>> "(Lorg/apache/spark/sql/catalyst/InternalRow;Lorg/
>> apache/spark/sql/catalyst/InternalRow;)I" of class "org.apache.spark.sql.
>> catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB
>>
>> Unfortunately I'm not clear on how to even isolate the source of this
>> problem. I didn't have this problem in Spark 1.6.1.
>>
>> Any clues?
>>
>>
>>
>> --
>> -Dhruve Ashar
>>
>>
>>
>>

Mime
View raw message