spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Nist <tsind...@gmail.com>
Subject Re: Load selected rows with sqlContext in the dataframe
Date Thu, 21 Jul 2016 15:48:09 GMT
You can set the dbtable to this:

.option("dbtable", "(select * from master_schema where 'TID' = '100_0')")

HTH,

Todd


On Thu, Jul 21, 2016 at 10:59 AM, sujeet jog <sujeet.jog@gmail.com> wrote:

> I have a table of size 5GB, and want to load selective rows into dataframe
> instead of loading the entire table in memory,
>
>
> For me memory is a constraint hence , and i would like to peridically load
> few set of rows and perform dataframe operations on it,
>
> ,
> for the "dbtable"  is there a way to perform select * from master_schema
> where 'TID' = '100_0';
> which can load only this to memory as dataframe .
>
>
>
> Currently  I'm using code as below
>     val df          =  sqlContext.read .format("jdbc")
>                       .option("url", url)
>                       .option("dbtable", "master_schema").load()
>
>
> Thansk,
> Sujeet
>

Mime
View raw message