spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ashesh_28 <asheshro...@gmail.com>
Subject RE: Unable to Access files in Hadoop HA enabled from using Spark
Date Wed, 13 Apr 2016 07:21:34 GMT
Are you running from eclipse ?
If so add the *Hadoop_conf_dir* path to the classpath

And then you can access your hdfs directory as below 

object sparkExample {
  def main(args: Array[String]){ 
    val logname = "///user/hduser/input/sample.txt"
    val conf = new
SparkConf().setAppName("SimpleApp").setMaster("local[2]").set("spark.executor.memory",
"1g")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logname, 2)
    val numAs = logData.filter(line => line.contains("hadoop")).count()
    val numBs = logData.filter(line => line.contains("spark")).count()
    println("Lines with Hadoop : %s, Lines with Spark: %s".format(numAs,
numBs))
  }
}




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-Access-files-in-Hadoop-HA-enabled-from-using-Spark-tp26768p26771.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message