spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghavendra Pandey <raghavendra.pan...@gmail.com>
Subject Re: DataFrame Filter Inside Another Data Frame Map
Date Thu, 02 Jul 2015 17:06:41 GMT
You can collect the dataframe as array n then create map out of it...,
On Jul 2, 2015 9:23 AM, <asoni.learn@gmail.com> wrote:

> Any example how can i return a Hashmap from data frame ?
>
> Thanks ,
> Ashish
>
> On Jul 1, 2015, at 11:34 PM, Holden Karau <holden@pigscanfly.ca> wrote:
>
> Collecting it as a regular (Java/scala/Python) map. You can also broadcast
> the map if your going to use it multiple times.
>
> On Wednesday, July 1, 2015, Ashish Soni <asoni.learn@gmail.com> wrote:
>
>> Thanks , So if i load some static data from database and then i need to
>> use than in my map function to filter records what will be the best way to
>> do it,
>>
>> Ashish
>>
>> On Wed, Jul 1, 2015 at 10:45 PM, Raghavendra Pandey <
>> raghavendra.pandey@gmail.com> wrote:
>>
>>> You cannot refer to one rdd inside another rdd.map function...
>>> Rdd object is not serialiable. Whatever objects you use inside map
>>> function  should be serializable as they get transferred to executor nodes.
>>> On Jul 2, 2015 6:13 AM, "Ashish Soni" <asoni.learn@gmail.com> wrote:
>>>
>>>> Hi All  ,
>>>>
>>>> I am not sure what is the wrong with below code as it give below error
>>>> when i access inside the map but it works outside
>>>>
>>>> JavaRDD<Charge> rdd2 = rdd.map(new Function<Charge, Charge>()
{
>>>>
>>>>
>>>>             @Override
>>>>             public Charge call(Charge ch) throws Exception {
>>>>
>>>>
>>>>                * DataFrame df = accountRdd.filter("login=test");*
>>>>
>>>>                 return ch;
>>>>             }
>>>>
>>>>         });
>>>>
>>>> 5/07/01 20:38:08 ERROR Executor: Exception in task 0.0 in stage 0.0
>>>> (TID 0)
>>>> java.lang.NullPointerException
>>>>     at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:129)
>>>>     at org.apache.spark.sql.DataFrame.org
>>>> $apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
>>>>
>>>
>>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
> Linked In: https://www.linkedin.com/in/holdenkarau
>
>

Mime
View raw message