spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: append rows to dataframe
Date Mon, 14 Mar 2016 03:50:19 GMT
Have you tried unionAll() method of DataFrame ?

On Sun, Mar 13, 2016 at 8:44 PM, Divya Gehlot <divya.htconex@gmail.com>
wrote:

> Hi,
>
> Please bear me for asking such a naive question
> I have list of conditions (dynamic sqls) sitting in hbase table .
> I need to iterate through those dynamic sqls and add the data to
> dataframes.
> As we know dataframes are immutable ,when I try to iterate in for loop as
> shown below I get only last dynamic sql result set .
>
> var dffiltered : DataFrame = sqlContext.emptyDataFrame
>  for ( i <- 0 to (dfFilterSQLs.length - 1)) {
>  println("Condition============="+dfFilterSQLs(i))
>  dffiltered =
> dfresult.filter(dfFilterSQLs(i)).select("Col1","Col2","Col3","Col4","Col5")
>   dffiltered.show
>   }
>
>
> How can I keep on appending data to dataframe and get the final result
> having all the sql conditions.
>
> Thanks in advance for the help.
>
> Thanks,
> Divya
>

Mime
View raw message