spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Takeshi Yamamuro <linguin....@gmail.com>
Subject Re: SparkSession for RDBMS
Date Wed, 03 Aug 2016 16:43:16 GMT
Hi,

If these bounaries are not given, spark tries to read all the data as a
single parition.
See:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala#L56

// maropu


On Wed, Aug 3, 2016 at 11:19 PM, Selvam Raman <selmna@gmail.com> wrote:

> Hi All,
>
> I would like to read the data from RDBMS to spark (2.0) using
> sparksession. How can i decide upper boundary, lower boundary and
> partitions.
> is there any specific approach available.
>
> How Sqoop2 decides number of partitions, upper and lower boundary if we
> are not specifying anything.
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>



-- 
---
Takeshi Yamamuro

Mime
View raw message