spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrés Ivaldi <iaiva...@gmail.com>
Subject Re: Bit(N) on create Table with MSSQLServer
Date Wed, 04 May 2016 19:54:52 GMT
Ok, so I did that, create a database and then insert data, but spark drop
database and try to create it again, I'm using
Dataframe.write(SaveMode.Overwrite), documentation said :
"when performing a Overwrite, the data will be deleted before writing out
the new data."

why is dropping the table?


On Wed, May 4, 2016 at 6:44 AM, Andrés Ivaldi <iaivaldi@gmail.com> wrote:

> Yes, I can do that, it's what we are doing now, but I think the best
> approach would be delegate the create table action to spark.
>
> On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> Can you create the MSSQL (target) table first with the correct column
>> setting and insert data from Spark to it with JDBC as opposed to JDBC
>> creating target table itself?
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 3 May 2016 at 22:19, Andrés Ivaldi <iaivaldi@gmail.com> wrote:
>>
>>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
>>> instead of varchar(MAX) , so how can I override default SQL Mapping?
>>>
>>> regards.
>>>
>>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Well if MSSQL cannot create that column then it is more like
>>>> compatibility between Spark and RDBMS.
>>>>
>>>> What value that column has in MSSQL. Can you create table the table in
>>>> MSSQL database or map it in Spark to a valid column before opening JDBC
>>>> connection?
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 29 April 2016 at 16:16, Andrés Ivaldi <iaivaldi@gmail.com> wrote:
>>>>
>>>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>>>> types, This create table cannot be executed on MSSQLServer.
>>>>>
>>>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>>>> right?
>>>>>
>>>>> Anyway it´s possible to override that mapping in Spark?
>>>>>
>>>>> Regards
>>>>>
>>>>> --
>>>>> Ing. Ivaldi Andres
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>
>>
>
>
> --
> Ing. Ivaldi Andres
>



-- 
Ing. Ivaldi Andres

Mime
View raw message