sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: Error exporting Avro File
Date Wed, 06 May 2015 00:13:15 GMT
Looks like Sqoop never figured out that the file is Avro (you can see it
calls TextExportMapper and not AvroExportMapper).

Sqoop guesses that the file is Avro based on the first 3 bytes (should be
'Obj'), which actually look correct in your case... so, while I can tell
what is the issue, I can't help solve it :(

Does it work on other Avro files? Did it work on older versions?

Gwen


On Tue, May 5, 2015 at 1:50 PM, Henrique Andrade <chmandrade@gmail.com>
wrote:

> I am trying to export a file to a PostgreSQL that is in Avro format but I
> am getting the following errors:
>
>
> I am using the 1.4.6 RC2 version. Any help?
>
> 2015-05-05 17:47:09,793 DEBUG - Fetching password from job credentials
> store
> 2015-05-05 17:47:10,271 ERROR -
> 2015-05-05 17:47:10,271 ERROR - Exception raised during data export
> 2015-05-05 17:47:10,271 ERROR -
> 2015-05-05 17:47:10,271 ERROR - Exception:
> java.lang.RuntimeException: Can't parse input data: '"sqlType":"93"}'
> at vw_vacation.__loadFromFields(vw_vacation.java:3096)
> at vw_vacation.parse(vw_vacation.java:2709)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NumberFormatException: For input string:
> ""sqlType":"93"}"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:481)
> at java.lang.Integer.valueOf(Integer.java:582)
> at vw_vacation.__loadFromFields(vw_vacation.java:2798)
> ... 13 more
> 2015-05-05 17:47:10,272 ERROR - On input: Obj
> avro.schema�.{"type":"record","name":"sqoop_import_VW_VENDAS_ITEMS_DETALHE","doc":"Sqoop
> import of
> VW_VENDAS_ITEMS_DETALHE","fields":[{"name":"CreatedAt","type":["null","long"],"default":null,"columnName":"CreatedAt","sqlType":"93"},{"name":"StoreId","type":["null","int"],"default":null,"columnName":"StoreId","sqlType":"5"},{"name":"ItemId","type":["null","long"],"default":null,"columnName":"ItemId","sqlType":"-5"},{"name":"StoreName","type":["null","string"],"default":null,"columnName":"StoreName","sqlType":"12"},{"name":"StoreCode","type":["null","string"],"default":null,"columnName":"StoreCode","sqlType":"12"},{"name":"OrderId","type":["null","string"],"default":null,"columnName":"OrderId","sqlType":"12"},{"name":"Status","type":["null","string"],"default":null,"columnName":"Status","sqlType":"12"},{"name":"CustomerId","type":["null","long"],"default":null,"columnName":"CustomerId","sqlType":"-5"},{"name":"CustomerFirstName","type":["null","string"],"default":null,"columnName":"CustomerFirstName","sqlType":"12"},{"name":"CustomerLastName","type":["null","string"],"default":null,"columnName":"CustomerLastName","sqlType":"12"},{"name":"CustomerEmail","type":["null","string"],"default":null,"columnName":"CustomerEmail","sqlType":"12"},{"name":"CustomerAddressType","type":["null","string"],"default":null,"columnName":"CustomerAddressType","sqlType":"12"},{"name":"CustomerStreet","type":["null","string"],"default":null,"columnName":"CustomerStreet","sqlType":"12"},{"name":"CustomerCity","type":["null","string"],"default":null,"columnName":"CustomerCity","sqlType":"12"},{"name":"CustomerZipCode","type":["null","string"],"default":null,"columnName":"CustomerZipCode","sqlType":"12"},{"name":"CustomerRegion","type":["null","string"],"default":null,"columnName":"CustomerRegion","sqlType":"12"},{"name":"CustomerCountry","type":["null","string"],"default":null,"columnName":"CustomerCountry","sqlType":"12"},{"name":"CustomerPhone","type":["null","string"],"default":null,"columnName":"CustomerPhone","sqlType":"12"},{"name":"ProductId","type":["null","long"],"default":null,"columnName":"ProductId","sqlType":"-5"},{"name":"SkuId","type":["null","string"],"default":null,"columnName":"SkuId","sqlType":"12"},{"name":"ProductName","type":["null","string"],"default":null,"columnName":"ProductName","sqlType":"12"},{"name":"CurrencyCode","type":["null","string"],"default":null,"columnName":"CurrencyCode","sqlType":"12"},{"name":"QtdInvoice","type":["null","string"],"default":null,"columnName":"QtdInvoice","sqlType":"3"},{"name":"QtdOrdered","type":["null","string"],"default":null,"columnName":"QtdOrdered","sqlType":"3"},{"name":"QtdReturned","type":["null","string"],"default":null,"columnName":"QtdReturned","sqlType":"3"},{"name":"OriginalPrice","type":["null","string"],"default":null,"columnName":"OriginalPrice","sqlType":"3"},{"name":"InvoicedTotalRow","type":["null","string"],"default":null,"columnName":"InvoicedTotalRow","sqlType":"3"}],"tableName":"VW_VENDAS_ITEMS_DETALHE"}��aV5jc
> � ۨ cu)p�� ����P \ BBeach Park
> 2015-05-05 17:47:10,272 ERROR - On input file:
> file:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring Data/Chapter
> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro
> 2015-05-05 17:47:10,272 ERROR - At position 0
> 2015-05-05 17:47:10,272 ERROR -
> 2015-05-05 17:47:10,272 ERROR - Currently processing split:
> 2015-05-05 17:47:10,272 ERROR -
> Paths:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring
> Data/Chapter
> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro:0+217032,/Users/henriqueandrade/Documents/Workspace/sandbox/Spring
> Data/Chapter
> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00001.avro:0+217032
> 2015-05-05 17:47:10,272 ERROR -
> 2015-05-05 17:47:10,273 ERROR - This issue might not necessarily be caused
> by current input
> 2015-05-05 17:47:10,273 ERROR - due to the batching nature of export.
> 2015-05-05 17:47:10,273 ERROR -
> 2015-05-05 17:47:10,273 DEBUG - Instructing auto-progress thread to quit.
> 2015-05-05 17:47:10,273 DEBUG - Waiting for progress thread shutdown...
> 2015-05-05 17:47:10,273 INFO  - Auto-progress thread is finished.
> keepGoing=false
> 2015-05-05 17:47:10,273 DEBUG - Progress thread shutdown detected.
> 2015-05-05 17:47:10,274 DEBUG - Committing transaction of 0 statements
> 2015-05-05 17:47:10,275 INFO  - map task executor complete.
> 2015-05-05 17:47:10,275 WARN  - job_local1317842886_0001
> java.lang.Exception: java.io.IOException: Can't export data, please check
> failed map task logs
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: java.io.IOException: Can't export data, please check failed map
> task logs
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Can't parse input data:
> '"sqlType":"93"}'
> at vw_vacation.__loadFromFields(vw_vacation.java:3096)
> at vw_vacation.parse(vw_vacation.java:2709)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
> ... 11 more
> Caused by: java.lang.NumberFormatException: For input string:
> ""sqlType":"93"}"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:481)
> at java.lang.Integer.valueOf(Integer.java:582)
> at vw_vacation.__loadFromFields(vw_vacation.java:2798)
> ... 13 more
> 2015-05-05 17:47:10,280 DEBUG - PrivilegedAction as:henriqueandrade
> (auth:SIMPLE)
> from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331)
> 2015-05-05 17:47:10,679 DEBUG - PrivilegedAction as:henriqueandrade
> (auth:SIMPLE)
> from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
> 2015-05-05 17:47:10,680 INFO  - Job job_local1317842886_0001 running in
> uber mode : false
> 2015-05-05 17:47:10,681 INFO  -  map 0% reduce 0%
>

Mime
View raw message