spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <ravishankar.n...@gmail.com>
Subject Question on using pseudo columns in spark jdbc options
Date Sun, 03 Dec 2017 06:39:40 GMT
Hi all,

I am using a query to fetch data from MYSQL as follows:

var df = spark.read.
format("jdbc").
option("url", "jdbc:mysql://10.0.0.192:3306/retail_db").
option("driver" ,"com.mysql.jdbc.Driver").
option("user", "retail_dba").
option("password", "cloudera").
option("dbtable", "orders").
option("partitionColumn", "order_id").
option("lowerBound", "1").
option("upperBound", "68883").
option("numPartitions", "4").
load()

Question is, can I use a pseudo column (like ROWNUM in Oracle or
RRN(employeeno) in DB2) in option where I specify the "partitionColumn" ?
If not, can we specify a partition column which is not a primary key ?

Best,
Ravion

Mime
View raw message