spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subhajit Purkayastha" <>
Subject Spark Save mode "Overwrite" -Lock wait timeout exceeded; try restarting transaction Error
Date Mon, 12 Sep 2016 04:08:53 GMT
I am using spark 1.5.2 with Memsql Database as a persistent repository


I am trying to update rows (based on the primary key), if it is appears more
than 1 time (basically run the save load as a Upsert operation)


val UpSertConf = SaveToMemSQLConf(msc.memSQLConf, 




                                         "duplicate_key_behavior" ->

                                         //,"insertBatchSize" -> "100"






When I set the SaveMode to "Overwrite", I get the following errors, bcos of
object locking



[cloudera@quickstart scala-2.10]$ spark-submit GLBalance-assembly-1.0.jar

(number of records ,204919)

[Stage 7:>                                                          (0 + 4)
/ 7]16/09/09 21:26:01 ERROR Executor: Exception in task 1.0 in stage 7.0
(TID 22)

java.sql.SQLException: Leaf Error ( Lock wait timeout
exceeded; try restarting transaction

    at com.mysql.jdbc.SQLError.createSQLException(

    at com.mysql.jdbc.MysqlIO.checkErrorPacket(

    at com.mysql.jdbc.MysqlIO.checkErrorPacket(

    at com.mysql.jdbc.MysqlIO.checkErrorPacket(

    at com.mysql.jdbc.MysqlIO.sendFileToServer(

    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(

    at com.mysql.jdbc.MysqlIO.readAllResults(

    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(

    at com.mysql.jdbc.ConnectionImpl.execSQL(

    at com.mysql.jdbc.StatementImpl.executeUpdate(

    at com.mysql.jdbc.StatementImpl.executeUpdate(





If I change the SaveMode to "append", the load completes but all the
duplicate records gets rejected. I was told that there is a known bug (edited)


If there another way for me to save the data (in an Upsert mode) ?



View raw message