spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version
Date Mon, 23 May 2016 13:38:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-3828.
------------------------------
    Resolution: Won't Fix

Defunct now that Hadoop 1 is not supported

> Spark returns inconsistent results when building with different Hadoop version 
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-3828
>                 URL: https://issues.apache.org/jira/browse/SPARK-3828
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0
>         Environment: OSX 10.9, Spark master branch
>            Reporter: Liquan Pei
>
> For text8 data at http://mattmahoney.net/dc/text8.zip. To reproduce, please unzip first.

> Spark build with different Hadoop version returns different result. 
> {code}
> val data = sc.textFile("text8")
> data.count()
> {code}
> returns 1 when built with SPARK_HADOOP_VERSION=1.0.4 and return 2 when built with SPARK_HADOOP_VERSION=2.4.0.

> Looking through the rdd code, it seems that textFile uses hadoopFile which creates HadoopRDD,
we should probably create newHadoopRDD when building spark with SPARK_HADOOP_VERSION >=
2.0.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message