sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Antoine Pieretti" <anpieretti-...@airfrance.fr>
Subject Re: SQOOP AVRO ORAOOP some questions
Date Fri, 09 Dec 2011 14:19:23 GMT

Hello,

Thanks for your help.

sorry for the delay.

I have created the JIRA : https://issues.apache.org/jira/browse/SQOOP-406
for the Avro update issue.

Regards

Antoine



                                                                           
             Jarek Jarcec                                                  
             Cecho                                                         
             <jarcec@apache.or                                           A 
             g>                        sqoop-user@incubator.apache.org     
                                                                        cc 
             07/12/2011 12:39                                              
                                                                     Objet 
                                       Re: SQOOP AVRO ORAOOP some          
             Veuillez répondre         questions                           
                     à                                                     
             sqoop-user@incuba                                             
              tor.apache.org                                               
                                                                           
                                                                           
                                                                           




Hmm... I've just marked problem with default AVRO mapping of type DATE as
"Won't fix" in SQOOP-399 because it would break backward compatibility, but
I'm starting to think that it needs to be addressed anyway. I'll look on it
later this week. Can you please file JIRA for that?

We've recently added option to force sqoop use different than default
mapping, however version 1.4.0 do not support that for AVRO output. But
there is working patch attached to SQOOP-362 that has been committed to the
trunk and will be included in next release. Meantime you might consider
applying this patch to the base 1.4.0 release and rebuilding sqoop for you.

Can you also file JIRA for your second issue? You can also submit your
patch for review in case that you managed to fix that issue.

Thank you very much for your time during investigating the issues!

Jarcec

SQOOP-399 https://issues.apache.org/jira/browse/SQOOP-399
SQOOP-362 https://issues.apache.org/jira/browse/SQOOP-362

On Wed, Dec 07, 2011 at 11:48:24AM +0100, Antoine Pieretti wrote:
>
> Hi Kate,
>
> here the versions :
> Sqoop :  1.4.0
> Oraoop : oraoop-1.4.0.149
> Hadoop : hadoop-core-0.20.2-cdh3u2
>
> The first thing we found, is that the AvroSchemaGenerator, return a LONG
> type for a DATE in toAvroType, but ORAOOP always works with String.
>
> Please, find here the console log during an Update
> (See attached file: SQOOP_UPDATE_LOG-txt.zip)
>
> In debug, I found that in TextExportMapper, he call parse method, wich
> __loadFromFields with a List<String>, with the first value equals to the
> text "Objavro.schema?{"type":"record".
>
> If in the JdbcUpdateExportJob I implement the AvroExportMapper for an
Avro
> fyleType, it works.
>
> Thanks for your help.
>
> Please, also find an avro file example.
>
>
> (See attached file: part-m-00000-avro.zip)
>
> Regards, Antoine
>
>
>
>
>
>
>

>              Kate Ting

>              <kate@cloudera.co

>              m>
A
>                                        sqoop-user@incubator.apache.org

>              07/12/2011 09:27
cc
>

>
Objet
>              Veuillez répondre         Re: SQOOP AVRO ORAOOP some

>                      à                 questions

>              sqoop-user@incuba

>               tor.apache.org

>

>

>

>

>
>
>
>
> Hi Antoine,
>
> The following would be helpful in figuring out what could be causing the
> failed job:
>
> What Sqoop, OraOop, and Hadoop versions are you running?
> Please run Sqoop job with --verbose flag and then attach console log.
> Also, provide a representative input data set that triggers this problem.
>
> Regards, Kate
>
> On Tue, Dec 6, 2011 at 12:15 AM, Antoine Pieretti <
> anpieretti-ext@airfrance.fr> wrote:
>
>
>       Hi folks,
>
>       i am a newbie within Sqoop and Avro.
>
>       I try to manipulate data from an Oracle Database, import, and
export
>       Avro files.
>
>       The import works well, the export (INSERT) also, but when i am
trying
>       to Update data from an Avro file to the DB the job failed.
>
>       Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx
>       --username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table
>       MY_TABLE --verbose --export-dir D:\\tmp\extraction\Rep --update-key
>       My_COLUMN_KEY.
>
>       The job calls the TextExportMapper (not the AvropExportMapper), and
>       try to parse the file, wich contains the Avro schemas and parse it
as
>       a data line.
>
>       Is it a common issue ?
>
>       Thanks for your help
>
>       Antoine PIERETTI
>
>
>
>       Accédez aux meilleurs tarifs Air France, gérez vos réservations et
>       enregistrez-vous en ligne sur http://www.airfrance.com
>       Find best Air France fares, manage your reservations and check in
>       online at http://www.airfrance.com
>
>
>       Les données et renseignements contenus dans ce message peuvent être
>       de nature confidentielle et soumis au secret professionnel et sont
>       destinés à l'usage exclusif du destinataire dont les coordonnées
>       figurent ci-dessus. Si vous recevez cette communication par erreur,
>       nous vous demandons de ne pas la copier, l'utiliser ou la
divulguer.
>       Nous vous prions de notifier cette erreur à l'expéditeur et
d'effacer
>       immédiatement cette communication de votre système. Société Air
>       France - Société anonyme au capital de 1 901 231 625 euros - RCS
>       Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
>       CEDEX
>       The data and information contained in this message may be
>       confidential and subject to professional secrecy and are intended
for
>       the exclusive use of the recipient at the address shown above. If
you
>       receive this message by mistake, we ask you not to copy, use or
>       disclose it. Please notify this error to the sender immediately and
>       delete this message from your system. Société Air France - Limited
>       company with capital of 1,901,231,625 euros - Bobigny register of
>       companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy
CDG
>       CEDEX Pensez à l'environnement avant d'imprimer ce message.
>       Think of the environment before printing this mail.
>
>
>
>
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et
enregistrez-vous en ligne sur  http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in online
at  http://www.airfrance.com  Les données et renseignements contenus dans
ce message peuvent être de nature confidentielle et soumis au secret
professionnel et sont destinés à l'usage exclusif du destinataire dont les
coordonnées figurent ci-dessus. Si vous recevez cette communication par
erreur, nous vous demandons de ne pas la copier, l'utiliser ou la
divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et
d'effacer immédiatement cette communication de votre système. Société Air
France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny
(France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
> The data and information contained in this message may be confidential
and subject to professional secrecy and are intended for the exclusive use
of the recipient at the address shown above. If you receive this message by
mistake, we ask you not to copy, use or disclose it. Please notify this
error to the sender immediately and delete this message from your system.
Société Air France - Limited company with capital of 1,901,231,625 euros -
Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95
747 Roissy CDG CEDEX  Pensez à l'environnement avant d'imprimer ce message.

> Think of the environment before printing this mail.







(See attached file: signature.asc)
Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en
ligne sur  http://www.airfrance.com  
Find best Air France fares, manage your reservations and check in online at  http://www.airfrance.com
 Les données et renseignements contenus dans ce message peuvent être de nature confidentielle
et soumis au secret professionnel et sont destinés à l'usage exclusif du destinataire dont
les coordonnées figurent ci-dessus. Si vous recevez cette communication par erreur, nous
vous demandons de ne pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier
cette erreur à l'expéditeur et d'effacer immédiatement cette communication de votre système.
Société Air France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny (France)
420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX  
The data and information contained in this message may be confidential and subject to professional
secrecy and are intended for the exclusive use of the recipient at the address shown above.
If you receive this message by mistake, we ask you not to copy, use or disclose it. Please
notify this error to the sender immediately and delete this message from your system. Société
Air France - Limited company with capital of 1,901,231,625 euros - Bobigny register of companies
(France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX  Pensez à l'environnement
avant d'imprimer ce message.  
Think of the environment before printing this mail.   
Mime
View raw message