spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sujeet jog <sujeet....@gmail.com>
Subject Re: Load selected rows with sqlContext in the dataframe
Date Fri, 22 Jul 2016 18:32:30 GMT
Thanks Todd.

On Thu, Jul 21, 2016 at 9:18 PM, Todd Nist <tsindotg@gmail.com> wrote:

> You can set the dbtable to this:
>
> .option("dbtable", "(select * from master_schema where 'TID' = '100_0')")
>
> HTH,
>
> Todd
>
>
> On Thu, Jul 21, 2016 at 10:59 AM, sujeet jog <sujeet.jog@gmail.com> wrote:
>
>> I have a table of size 5GB, and want to load selective rows into
>> dataframe instead of loading the entire table in memory,
>>
>>
>> For me memory is a constraint hence , and i would like to peridically
>> load few set of rows and perform dataframe operations on it,
>>
>> ,
>> for the "dbtable"  is there a way to perform select * from master_schema
>> where 'TID' = '100_0';
>> which can load only this to memory as dataframe .
>>
>>
>>
>> Currently  I'm using code as below
>>     val df          =  sqlContext.read .format("jdbc")
>>                       .option("url", url)
>>                       .option("dbtable", "master_schema").load()
>>
>>
>> Thansk,
>> Sujeet
>>
>
>

Mime
View raw message