sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Kemper <mar...@cloudera.com>
Subject Re: Upsert option in Sqoop export command.
Date Tue, 05 Dec 2017 05:03:42 GMT
Hello Ajana,

*re --update-key column_name*

Is "column_name" the actual name of the column having the  unique key in
the Oracle table.

*re: --columns "id,START_DT,EXP_DT,QUALITY,VAL,PROCES_DT"*

With Oracle options that reference column names are most likely going to be
case-sensitive, is the "id" column actually "ID" in your Oracle table
schema?


Markus Kemper
Customer Operations Engineer
[image: www.cloudera.com] <http://www.cloudera.com>


On Mon, Dec 4, 2017 at 10:05 PM, Ajana Chandiruthil Sathian <
ajanacs@gmail.com> wrote:

> Hello Markus,
>
> I tried with --input-lines-terminated-by instead of lines-terminated-by
> still, the --column argument is working as it supposed to work.  The
> --column option worked perfectly when  I did only the insert option( the
> initial load) when I am doing the upsert option, it is not considering the
> order as I mentioned in the --column sqoop argument.  Please see the
> attached sqoop argument and the CSV file screenshot. I could not understand
> why --column option is not working when I am doing upsert. Andy kind of
> help is really appreciated.
>
> Column order in ods: id,START_DT,EXP_DT,VAL,QUALITY,PROCES_DT
> value order in csv file; id,START_DT,EXP_DT,QUALITY,VAL,PROCES_DT
>
> --Sqoop command.
> sqoop export --connect connection_string \
> --username xxx \
> --password xxxx \
> --table xxx \
> --export-dir /location/file.csv \
> --input-fields-terminated-by ',' \
> --input-lines-terminated-by '\n' \
> --update-key column_name \
> --update-mode allowinsert \
> --columns "id,START_DT,EXP_DT,QUALITY,VAL,PROCES_DT" \
> -m 1
>
> Best,
> Ajana Sathian
> Big Data Intern
> Cars.com
> http://ajanacs.com/
>
> On Sun, Dec 3, 2017 at 3:41 PM, Markus Kemper <markus@cloudera.com> wrote:
>
>> Hey Alana,
>>
>> —lines-* and —fields-* are for (sqoop import) and —input-lines-* and
>> —input-fields-* are for (sqoop export).
>>
>> If you do not specify it will use the default which may or may not be
>> compatible with your data files.
>>
>> As a best practice I always recommend explicitly setting all options to
>> avoid any confusion or assumptions.
>>
>> Thanks, Markus
>>
>> On Dec 3, 2017, at 12:46, Ajana Chandiruthil Sathian <ajanacs@gmail.com>
>> wrote:
>>
>> Hello Markus,
>>
>> Can I ask you one question? Does that make any change? Both are for the
>> same purpose, right? Please correct me if I am wrong.
>>
>> On Sun, Dec 3, 2017 at 10:39 AM, Markus Kemper <markus@cloudera.com>
>> wrote:
>>
>>> Hello Ajana,
>>>
>>> Have you tried using (--input-lines-terminated-by '\n') instead of (--lines-terminated-by
>>> '\n')?
>>>
>>>
>>> Markus Kemper
>>> Customer Operations Engineer
>>> [image: www.cloudera.com] <http://www.cloudera.com>
>>>
>>>
>>> On Sun, Dec 3, 2017 at 8:45 AM, Ajana Chandiruthil Sathian <
>>> ajanacs@gmail.com> wrote:
>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Ajana Chandiruthil Sathian <ajanacs@gmail.com>
>>>> Date: Wed, Nov 15, 2017 at 11:09 AM
>>>> Subject: Upsert option in Sqoop export command.
>>>> To: user@sqoop.apache.org
>>>>
>>>>
>>>> To whom ever it may concern,
>>>>
>>>> I have a csv file in Hadoop and I did Sqoop export to Oracle. The
>>>> column data type order in ODS is number,date, date,float,varchar and the
>>>> column data type in the csv file is number,date, float,date,varchar and I
>>>> used the --columns sqoop argument to get control in column ordering and it
>>>> worked. But I could not control the column ordering when I was doing the
>>>> upsert operation in sqoop export. It is giving  me misalignment in ODS
>>>> after Sqoop( please see the attached image). The below given is the sqoop
>>>> command:
>>>>
>>>> sqoop export --connect ConnectionString \
>>>> --username xxx \
>>>> --password xxxx \
>>>> --table tableName \
>>>> --export-dir /dir/TestUpdate.txt \
>>>> --input-fields-terminated-by ',' \
>>>> --lines-terminated-by '\n' \
>>>> --update-key column_name \
>>>> --update-mode allowinsert \
>>>> --columns "id,START_DT,VAL,end_DT,QUALITY" \
>>>> -m 4
>>>>
>>>>
>>>
>>
>

Mime
View raw message