spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khare, Ankit" <ankit.kh...@eon.com>
Subject Re: writing into oracle database is very slow
Date Fri, 19 Apr 2019 13:58:05 GMT
Hi Jiang

We faced similar issue so we write the file and then use sqoop to export data to mssql.

We achieved a great time benefit with this strategy.

Sent from my iPhone

On 19. Apr 2019, at 10:47, spark receiver <spark.receiver@gmail.com<mailto:spark.receiver@gmail.com>>
wrote:

hi Jiang,

i was facing the very same issue ,the solution is write to file and using oracle external
table to do the insert.

hope this could help.

Dalin

On Thu, Apr 18, 2019 at 11:43 AM Jörn Franke <jornfranke@gmail.com<mailto:jornfranke@gmail.com>>
wrote:
What is the size of the data? How much time does it need on HDFS and how much on Oracle? How
many partitions do you have on Oracle side?

Am 06.04.2019 um 16:59 schrieb Lian Jiang <jiangok2006@gmail.com<mailto:jiangok2006@gmail.com>>:

Hi,

My spark job writes into oracle db using:

df.coalesce(10).write.format("jdbc").option("url", url)
  .option("driver", driver).option("user", user)
  .option("batchsize", 2000)
  .option("password", password).option("dbtable", tableName).mode("append").save()

It is much slow than writting into HDFS. The data to write is small.

Is this expected? Thanks for any clue.

Mime
View raw message