spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-15392) The default value of size estimation is not good
Date Thu, 19 May 2016 00:22:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-15392:
------------------------------------

    Assignee: Apache Spark  (was: Davies Liu)

> The default value of size estimation is not good
> ------------------------------------------------
>
>                 Key: SPARK-15392
>                 URL: https://issues.apache.org/jira/browse/SPARK-15392
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Davies Liu
>            Assignee: Apache Spark
>             Fix For: 2.0.0
>
>
> We use  autoBroadcastJoinThreshold + 1L as the default value of size estimation, that
is not good in 2.0, because we will calculate the size based on size of schema, then the estimation
could be less than autoBroadcastJoinThreshold if you have an SELECT on top of an DataFrame
created from RDD.
> We should use an even bigger default value, for example, MaxLong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message