sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henrique Andrade <chmandr...@gmail.com>
Subject Re: Error exporting Avro File
Date Wed, 06 May 2015 12:21:37 GMT
This is the first time that I am using since I am having several problems
with files with different characters. I was wondering that with Avro I
could solve that. Other option that I thought was to change the code/decode
on base64.

Since there are tickets like this:

https://issues.apache.org/jira/browse/SQOOP-1283
https://issues.apache.org/jira/browse/SQOOP-1282

I think this is supposed to be working.


2015-05-05 21:13 GMT-03:00 Gwen Shapira <gshapira@cloudera.com>:

> Looks like Sqoop never figured out that the file is Avro (you can see it
> calls TextExportMapper and not AvroExportMapper).
>
> Sqoop guesses that the file is Avro based on the first 3 bytes (should be
> 'Obj'), which actually look correct in your case... so, while I can tell
> what is the issue, I can't help solve it :(
>
> Does it work on other Avro files? Did it work on older versions?
>
> Gwen
>
>
> On Tue, May 5, 2015 at 1:50 PM, Henrique Andrade <chmandrade@gmail.com>
> wrote:
>
>> I am trying to export a file to a PostgreSQL that is in Avro format but I
>> am getting the following errors:
>>
>>
>> I am using the 1.4.6 RC2 version. Any help?
>>
>> 2015-05-05 17:47:09,793 DEBUG - Fetching password from job credentials
>> store
>> 2015-05-05 17:47:10,271 ERROR -
>> 2015-05-05 17:47:10,271 ERROR - Exception raised during data export
>> 2015-05-05 17:47:10,271 ERROR -
>> 2015-05-05 17:47:10,271 ERROR - Exception:
>> java.lang.RuntimeException: Can't parse input data: '"sqlType":"93"}'
>> at vw_vacation.__loadFromFields(vw_vacation.java:3096)
>> at vw_vacation.parse(vw_vacation.java:2709)
>> at
>> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
>> at
>> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>> at
>> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> at
>> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NumberFormatException: For input string:
>> ""sqlType":"93"}"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Integer.parseInt(Integer.java:481)
>> at java.lang.Integer.valueOf(Integer.java:582)
>> at vw_vacation.__loadFromFields(vw_vacation.java:2798)
>> ... 13 more
>> 2015-05-05 17:47:10,272 ERROR - On input: Obj
>> avro.schema�.{"type":"record","name":"sqoop_import_VW_VENDAS_ITEMS_DETALHE","doc":"Sqoop
>> import of
>> VW_VENDAS_ITEMS_DETALHE","fields":[{"name":"CreatedAt","type":["null","long"],"default":null,"columnName":"CreatedAt","sqlType":"93"},{"name":"StoreId","type":["null","int"],"default":null,"columnName":"StoreId","sqlType":"5"},{"name":"ItemId","type":["null","long"],"default":null,"columnName":"ItemId","sqlType":"-5"},{"name":"StoreName","type":["null","string"],"default":null,"columnName":"StoreName","sqlType":"12"},{"name":"StoreCode","type":["null","string"],"default":null,"columnName":"StoreCode","sqlType":"12"},{"name":"OrderId","type":["null","string"],"default":null,"columnName":"OrderId","sqlType":"12"},{"name":"Status","type":["null","string"],"default":null,"columnName":"Status","sqlType":"12"},{"name":"CustomerId","type":["null","long"],"default":null,"columnName":"CustomerId","sqlType":"-5"},{"name":"CustomerFirstName","type":["null","string"],"default":null,"columnName":"CustomerFirstName","sqlType":"12"},{"name":"CustomerLastName","type":["null","string"],"default":null,"columnName":"CustomerLastName","sqlType":"12"},{"name":"CustomerEmail","type":["null","string"],"default":null,"columnName":"CustomerEmail","sqlType":"12"},{"name":"CustomerAddressType","type":["null","string"],"default":null,"columnName":"CustomerAddressType","sqlType":"12"},{"name":"CustomerStreet","type":["null","string"],"default":null,"columnName":"CustomerStreet","sqlType":"12"},{"name":"CustomerCity","type":["null","string"],"default":null,"columnName":"CustomerCity","sqlType":"12"},{"name":"CustomerZipCode","type":["null","string"],"default":null,"columnName":"CustomerZipCode","sqlType":"12"},{"name":"CustomerRegion","type":["null","string"],"default":null,"columnName":"CustomerRegion","sqlType":"12"},{"name":"CustomerCountry","type":["null","string"],"default":null,"columnName":"CustomerCountry","sqlType":"12"},{"name":"CustomerPhone","type":["null","string"],"default":null,"columnName":"CustomerPhone","sqlType":"12"},{"name":"ProductId","type":["null","long"],"default":null,"columnName":"ProductId","sqlType":"-5"},{"name":"SkuId","type":["null","string"],"default":null,"columnName":"SkuId","sqlType":"12"},{"name":"ProductName","type":["null","string"],"default":null,"columnName":"ProductName","sqlType":"12"},{"name":"CurrencyCode","type":["null","string"],"default":null,"columnName":"CurrencyCode","sqlType":"12"},{"name":"QtdInvoice","type":["null","string"],"default":null,"columnName":"QtdInvoice","sqlType":"3"},{"name":"QtdOrdered","type":["null","string"],"default":null,"columnName":"QtdOrdered","sqlType":"3"},{"name":"QtdReturned","type":["null","string"],"default":null,"columnName":"QtdReturned","sqlType":"3"},{"name":"OriginalPrice","type":["null","string"],"default":null,"columnName":"OriginalPrice","sqlType":"3"},{"name":"InvoicedTotalRow","type":["null","string"],"default":null,"columnName":"InvoicedTotalRow","sqlType":"3"}],"tableName":"VW_VENDAS_ITEMS_DETALHE"}��aV5jc
>> � ۨ cu)p�� ����P \ BBeach Park
>> 2015-05-05 17:47:10,272 ERROR - On input file:
>> file:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring Data/Chapter
>> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro
>> 2015-05-05 17:47:10,272 ERROR - At position 0
>> 2015-05-05 17:47:10,272 ERROR -
>> 2015-05-05 17:47:10,272 ERROR - Currently processing split:
>> 2015-05-05 17:47:10,272 ERROR -
>> Paths:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring
>> Data/Chapter
>> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro:0+217032,/Users/henriqueandrade/Documents/Workspace/sandbox/Spring
>> Data/Chapter
>> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00001.avro:0+217032
>> 2015-05-05 17:47:10,272 ERROR -
>> 2015-05-05 17:47:10,273 ERROR - This issue might not necessarily be
>> caused by current input
>> 2015-05-05 17:47:10,273 ERROR - due to the batching nature of export.
>> 2015-05-05 17:47:10,273 ERROR -
>> 2015-05-05 17:47:10,273 DEBUG - Instructing auto-progress thread to quit.
>> 2015-05-05 17:47:10,273 DEBUG - Waiting for progress thread shutdown...
>> 2015-05-05 17:47:10,273 INFO  - Auto-progress thread is finished.
>> keepGoing=false
>> 2015-05-05 17:47:10,273 DEBUG - Progress thread shutdown detected.
>> 2015-05-05 17:47:10,274 DEBUG - Committing transaction of 0 statements
>> 2015-05-05 17:47:10,275 INFO  - map task executor complete.
>> 2015-05-05 17:47:10,275 WARN  - job_local1317842886_0001
>> java.lang.Exception: java.io.IOException: Can't export data, please check
>> failed map task logs
>> at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>> at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
>> Caused by: java.io.IOException: Can't export data, please check failed
>> map task logs
>> at
>> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
>> at
>> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>> at
>> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> at
>> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.RuntimeException: Can't parse input data:
>> '"sqlType":"93"}'
>> at vw_vacation.__loadFromFields(vw_vacation.java:3096)
>> at vw_vacation.parse(vw_vacation.java:2709)
>> at
>> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
>> ... 11 more
>> Caused by: java.lang.NumberFormatException: For input string:
>> ""sqlType":"93"}"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Integer.parseInt(Integer.java:481)
>> at java.lang.Integer.valueOf(Integer.java:582)
>> at vw_vacation.__loadFromFields(vw_vacation.java:2798)
>> ... 13 more
>> 2015-05-05 17:47:10,280 DEBUG - PrivilegedAction as:henriqueandrade
>> (auth:SIMPLE)
>> from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331)
>> 2015-05-05 17:47:10,679 DEBUG - PrivilegedAction as:henriqueandrade
>> (auth:SIMPLE)
>> from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
>> 2015-05-05 17:47:10,680 INFO  - Job job_local1317842886_0001 running in
>> uber mode : false
>> 2015-05-05 17:47:10,681 INFO  -  map 0% reduce 0%
>>
>
>

Mime
View raw message