spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <>
Subject Question on using pseudo columns in spark jdbc options
Date Sun, 03 Dec 2017 06:39:40 GMT
Hi all,

I am using a query to fetch data from MYSQL as follows:

var df =
option("url", "jdbc:mysql://").
option("driver" ,"com.mysql.jdbc.Driver").
option("user", "retail_dba").
option("password", "cloudera").
option("dbtable", "orders").
option("partitionColumn", "order_id").
option("lowerBound", "1").
option("upperBound", "68883").
option("numPartitions", "4").

Question is, can I use a pseudo column (like ROWNUM in Oracle or
RRN(employeeno) in DB2) in option where I specify the "partitionColumn" ?
If not, can we specify a partition column which is not a primary key ?


View raw message