[ https://issues.apache.org/jira/browse/SPARK-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-15392:
------------------------------------
Assignee: Apache Spark (was: Davies Liu)
> The default value of size estimation is not good
> ------------------------------------------------
>
> Key: SPARK-15392
> URL: https://issues.apache.org/jira/browse/SPARK-15392
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Davies Liu
> Assignee: Apache Spark
> Fix For: 2.0.0
>
>
> We use autoBroadcastJoinThreshold + 1L as the default value of size estimation, that
is not good in 2.0, because we will calculate the size based on size of schema, then the estimation
could be less than autoBroadcastJoinThreshold if you have an SELECT on top of an DataFrame
created from RDD.
> We should use an even bigger default value, for example, MaxLong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|