spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksander Eskilson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17131) Code generation fails when running SQL expressions against a wide dataset (thousands of columns)
Date Wed, 19 Oct 2016 22:12:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590024#comment-15590024
] 

Aleksander Eskilson commented on SPARK-17131:
---------------------------------------------

[~sowen], [~melentye]
I'm not so certain this error is the same as SPARK-16845. It seems like there have been several
classes of errors all related to the sizes of individual methods growing beyond the 64 KB
limit (SPARK-16845, SPARK-17702). I think this one is of a different class of error, or at
least {code}Constant pool has grown past JVM limit of 0xFFFF{code} marks a different class
of error. I was able to produce similar to the one first documented when trying to encode
a Java object with a very wide and deeply nested schema. I've gone ahead and created a bug
report for that, SPARK-18016, and in its description I've attached a small project that can
reproduce the error.

> Code generation fails when running SQL expressions against a wide dataset (thousands
of columns)
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-17131
>                 URL: https://issues.apache.org/jira/browse/SPARK-17131
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Iaroslav Zeigerman
>         Attachments: _SPARK_17131__add_a_test_case_with_1000_column_DF_where_describe___fails.patch
>
>
> When reading the CSV file that contains 1776 columns Spark and Janino fail to generate
the code with message:
> {noformat}
> Constant pool has grown past JVM limit of 0xFFFF
> {noformat}
> When running a common select with all columns it's fine:
> {code}
>       val allCols = df.columns.map(c => col(c).as(c + "_alias"))
>       val newDf = df.select(allCols: _*)
>       newDf.show()
> {code}
> But when I invoke the describe method:
> {code}
> newDf.describe(allCols: _*)
> {code}
> it fails with the following stack trace:
> {noformat}
> 	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:889)
> 	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:941)
> 	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:938)
> 	at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> 	at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> 	... 30 more
> Caused by: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM
limit of 0xFFFF
> 	at org.codehaus.janino.util.ClassFile.addToConstantPool(ClassFile.java:402)
> 	at org.codehaus.janino.util.ClassFile.addConstantIntegerInfo(ClassFile.java:300)
> 	at org.codehaus.janino.UnitCompiler.addConstantIntegerInfo(UnitCompiler.java:10307)
> 	at org.codehaus.janino.UnitCompiler.pushConstant(UnitCompiler.java:8868)
> 	at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4346)
> 	at org.codehaus.janino.UnitCompiler.access$7100(UnitCompiler.java:185)
> 	at org.codehaus.janino.UnitCompiler$10.visitIntegerLiteral(UnitCompiler.java:3265)
> 	at org.codehaus.janino.Java$IntegerLiteral.accept(Java.java:4321)
> 	at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3290)
> 	at org.codehaus.janino.UnitCompiler.fakeCompile(UnitCompiler.java:2605)
> 	at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4362)
> 	at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3975)
> 	at org.codehaus.janino.UnitCompiler.access$6900(UnitCompiler.java:185)
> 	at org.codehaus.janino.UnitCompiler$10.visitMethodInvocation(UnitCompiler.java:3263)
> 	at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:3974)
> 	at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3290)
> 	at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4368)
> 	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2662)
> 	at org.codehaus.janino.UnitCompiler.access$4400(UnitCompiler.java:185)
> 	at org.codehaus.janino.UnitCompiler$7.visitMethodInvocation(UnitCompiler.java:2627)
> 	at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:3974)
> 	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2654)
> 	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1643)
> ....
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message