spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Spark SQL Transaction
Date Wed, 20 Apr 2016 15:09:29 GMT
Assuming that you are using JDBC for putting data into any ACID compliant
database (MSSQL, Sybase, Oracle etc), you are implicitly or explicitly
 adding BEGIN TRAN to INSERT statement in a distributed transaction. MSSQL
does not know or care where data is coming from. If your connection
completes OK a COMMIT TRAN will be sent and that will tell MSQL to commit
transaction. If yoy kill Spark transaction before MSSQL receive COMMIT
TRAN, the transaction will be rolled back.

The only option is that if you don't care about full data getting to
MSSQL,to break your insert into chunks at source and send data to MSSQL in
small batches. In that way you will not lose all data in MSSQL because of
rollback.

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 20 April 2016 at 07:33, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> Are you using JDBC to push data to MSSQL?
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 19 April 2016 at 23:41, Andrés Ivaldi <iaivaldi@gmail.com> wrote:
>
>> I mean local transaction, We've ran a Job that writes into SQLServer then
>> we killed spark JVM just for testing purpose and we realized that SQLServer
>> did a rollback.
>>
>> Regards
>>
>> On Tue, Apr 19, 2016 at 5:27 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> What do you mean by *without transaction*? do you mean forcing SQL
>>> Server to accept a non logged operation?
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 19 April 2016 at 21:18, Andrés Ivaldi <iaivaldi@gmail.com> wrote:
>>>
>>>> Hello, is possible to execute a SQL write without Transaction? we dont
>>>> need transactions to save our data and this adds an overhead to the
>>>> SQLServer.
>>>>
>>>> Regards.
>>>>
>>>> --
>>>> Ing. Ivaldi Andres
>>>>
>>>
>>>
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>

Mime
View raw message