spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (SPARK-15157) String fields in Dataframe behaves weirdly when executor-memory >= 32GB
Date Thu, 05 May 2016 13:03:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen reopened SPARK-15157:
-------------------------------

Please resolve as duplicate, not fixed

> String fields in Dataframe behaves weirdly when executor-memory >= 32GB
> -----------------------------------------------------------------------
>
>                 Key: SPARK-15157
>                 URL: https://issues.apache.org/jira/browse/SPARK-15157
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Varadharajan
>            Priority: Critical
>
> Consider the below snippet. This issue was faced on an infrastructure with CDH 5.5.2
(which ships with Spark 1.5.0).
> {code:title=test.scala|borderStyle=solid}
> case class A(id: Int, name: String)
> sc.parallelize(List(A(1, "varadharajan"), A(2, "thiyagu"))).toDF.show
> {code}
> When i execute the above snippet with the below command line arguments, it works as expected.
> {code}
> spark-shell --master yarn  --num-executors 4 --executor-cores 10 --executor-memory 31g
> {code}
> {quote}
> | id|        name|
> |  1|varadharajan|
> |  2|     thiyagu|
> {quote}
> But things become weird, when i increase executor memory beyond 32GB. 
> {code}
> spark-shell --master yarn  --num-executors 4 --executor-cores 10 --executor-memory 32g
> {code}
> {quote}
> | id|        name|
> |  1|ajan|
> |  2|     |
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message