spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <abdul.h.huss...@bt.com>
Subject [Spark DataFrame]: Passing DataFrame to custom method results in NullPointerException
Date Mon, 15 Jan 2018 11:56:01 GMT
Hi,

My Spark app is mapping lines from a text file to case classes stored within an RDD.

When I run the following code on this rdd:
.collect.map(line => if(validate_hostname(line, data_frame)) line).foreach(println)

It correctly calls the method validate_hostname by passing the case class and another data_frame
defined within the main method. Unfortunately the above map only returns a TraversableLike
collection so I can't do transformations and joins on this data structure so I'm tried to
apply a filter on the rdd with the following code:
.filter(line => validate_hostname(line, data_frame)).count()

Unfortunately the above method with filtering the rdd does not pass the data_frame so I get
a NullPointerException though it correctly passes the case class which I print within the
method.

Where am I going wrong?

When

Regards,
Abdul Haseeb Hussain

Mime
View raw message