spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: append rows to dataframe
Date Mon, 14 Mar 2016 04:31:51 GMT
 dffiltered = unionAll(dfresult.filter(dfFilterSQLs(
i)).select("Col1","Col2","Col3","Col4","Col5"))

FYI

On Sun, Mar 13, 2016 at 8:50 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Have you tried unionAll() method of DataFrame ?
>
> On Sun, Mar 13, 2016 at 8:44 PM, Divya Gehlot <divya.htconex@gmail.com>
> wrote:
>
>> Hi,
>>
>> Please bear me for asking such a naive question
>> I have list of conditions (dynamic sqls) sitting in hbase table .
>> I need to iterate through those dynamic sqls and add the data to
>> dataframes.
>> As we know dataframes are immutable ,when I try to iterate in for loop as
>> shown below I get only last dynamic sql result set .
>>
>> var dffiltered : DataFrame = sqlContext.emptyDataFrame
>>  for ( i <- 0 to (dfFilterSQLs.length - 1)) {
>>  println("Condition============="+dfFilterSQLs(i))
>>  dffiltered =
>> dfresult.filter(dfFilterSQLs(i)).select("Col1","Col2","Col3","Col4","Col5")
>>   dffiltered.show
>>   }
>>
>>
>> How can I keep on appending data to dataframe and get the final result
>> having all the sql conditions.
>>
>> Thanks in advance for the help.
>>
>> Thanks,
>> Divya
>>
>
>

Mime
View raw message