My Spark app is mapping lines from a text file to case classes stored within an RDD.
When I run the following code on this rdd:
.collect.map(line => if(validate_hostname(line, data_frame)) line).foreach(println)
It correctly calls the method validate_hostname by passing the case class and another data_frame defined within the main method. Unfortunately the above map only returns a TraversableLike collection so I can’t do transformations and joins on this data structure so I’m tried to apply a filter on the rdd with the following code:
.filter(line => validate_hostname(line, data_frame)).count()
Unfortunately the above method with filtering the rdd does not pass the data_frame so I get a NullPointerException though it correctly passes the case class which I print within the method.
Where am I going wrong?
Abdul Haseeb Hussain