spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sujeet jog <sujeet....@gmail.com>
Subject Load selected rows with sqlContext in the dataframe
Date Thu, 21 Jul 2016 14:59:15 GMT
I have a table of size 5GB, and want to load selective rows into dataframe
instead of loading the entire table in memory,


For me memory is a constraint hence , and i would like to peridically load
few set of rows and perform dataframe operations on it,

,
for the "dbtable"  is there a way to perform select * from master_schema
where 'TID' = '100_0';
which can load only this to memory as dataframe .



Currently  I'm using code as below
    val df          =  sqlContext.read .format("jdbc")
                      .option("url", url)
                      .option("dbtable", "master_schema").load()


Thansk,
Sujeet

Mime
View raw message