sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muthu Pandi <muthu1...@gmail.com>
Subject Re: Exporting to non standard databases
Date Mon, 25 Aug 2014 05:49:10 GMT
Pls Ignore the previous mail Thanks guys for the help and for info found
that "--batch" option is working which support batch export without  "
 -Dsqoop.export.records.per.statement=1".



*Regards Muthupandi.K*

 [image: Picture (Device Independent Bitmap)]



On Mon, Aug 25, 2014 at 11:18 AM, Muthu Pandi <muthu1086@gmail.com> wrote:

> Thanks guys for the help and for info found that "--direct" option is
> working which support batch export without  " -Dsqoop.export.records.per.
> statement=1".
>
>
>
> *Regards Muthupandi.K*
>
>  [image: Picture (Device Independent Bitmap)]
>
>
>
> On Mon, Aug 25, 2014 at 11:04 AM, Gwen Shapira <gshapira@cloudera.com>
> wrote:
>
>> It won't be super fast since its row-by-row insert, but 2GB is not huge,
>> so I'll expect this will take only few minutes.
>>
>>
>> On Sun, Aug 24, 2014 at 10:23 PM, Muthu Pandi <muthu1086@gmail.com>
>> wrote:
>>
>>> Spot on!!! Jarcec
>>>
>>>        Thanks for the quick reply, you have saved my lot of time.
>>>
>>>        Can this be used for exporting larger data since we are exporting
>>> in bulk. My data will be of size 2GB Per day and will export at end of the
>>> day.
>>>
>>>
>>>
>>>
>>> *Regards Muthupandi.K*
>>>
>>>  [image: Picture (Device Independent Bitmap)]
>>>
>>>
>>>
>>> On Mon, Aug 25, 2014 at 10:40 AM, Jarek Jarcec Cecho <jarcec@apache.org>
>>> wrote:
>>>
>>>> Hi Muthu,
>>>> the given message is a “Warning” and not an “Error”. Sqoop is warning
>>>> you that you are using —driver option that will force Sqoop to use Generic
>>>> JDBC connector. That is fine in your case as Sqoop don’t have specialized
>>>> connector for Vertica. You can always use --connect-manager parameter to
>>>> get rid of the message as is the warning suggesting:
>>>>
>>>>         sqoop export --connect-manager
>>>> org.apache.sqoop.manager.GenericJdbcManager ...
>>>>
>>>> You are however hitting a different problem, my best guess that Vertica
>>>> is not supporting multi value insert statements that Sqoop is generating
by
>>>> default. Try to disable this feature using property
>>>> sqoop.export.records.per.statement:
>>>>
>>>>         sqoop export -Dsqoop.export.records.per.statement=1 —connect ...
>>>>
>>>> Jarcec
>>>>
>>>> On Aug 25, 2014, at 6:55 AM, Muthu Pandi <muthu1086@gmail.com> wrote:
>>>>
>>>> > Hi every one,
>>>> >                Is it possible to export data from HDFS to vetica.I
>>>> successfully imported data from vertica to HDFS using sqoop.When i Export
i
>>>> get the following error.Please give me solution as soon as possible.this
is
>>>> the error which am getting,
>>>> >
>>>> > WARN sqoop.ConnFactory: Parameter --driver is set to an explicit
>>>> driver however appropriate connection manager is not being set (via
>>>> --connection-manager). Sqoop is going to fall back to
>>>> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
>>>> which connection manager should be used next time.
>>>> >
>>>> > will above will have impact.How do i create connection manager for
>>>> vertica.
>>>> >
>>>> > My sqoop query is
>>>> >
>>>> > sqoop export  --driver com.vertica.jdbc.Driver --connect
>>>> "jdbc:vertica://X.X.X.X:5433/dbname" --username username -P --table
>>>> tablename  -m 1 --export-dir /ver.txt  --input-fields-terminated-by ','
>>>> --lines-terminated-by '\n'
>>>> >
>>>> >
>>>> > and am using hadoop2.4.1 and sqoop1.4.4 version
>>>> >
>>>> >
>>>> >
>>>> > 14/08/25 10:12:57 WARN tool.BaseSqoopTool: Setting your password on
>>>> the command-line is insecure. Consider using -P instead.
>>>> > 14/08/25 10:12:57 WARN sqoop.ConnFactory: Parameter --driver is set
>>>> to an explicit driver however appropriate connection manager is not being
>>>> set (via --connection-manager). Sqoop is going to fall back to
>>>> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
>>>> which connection manager should be used next time.
>>>> > 14/08/25 10:12:57 INFO manager.SqlManager: Using default fetchSize of
>>>> 1000
>>>> > 14/08/25 10:12:57 INFO tool.CodeGenTool: Beginning code generation
>>>> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement:
>>>> SELECT t.* FROM AGE AS t WHERE 1=0
>>>> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement:
>>>> SELECT t.* FROM AGE AS t WHERE 1=0
>>>> > 14/08/25 10:12:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>>>> /opt/hadoop-2.4.1
>>>> > Note:
>>>> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.java uses
>>>> or overrides a deprecated API.
>>>> > Note: Recompile with -Xlint:deprecation for details.
>>>> > 14/08/25 10:13:02 INFO orm.CompilationManager: Writing jar file:
>>>> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.jar
>>>> > 14/08/25 10:13:03 INFO mapreduce.ExportJobBase: Beginning export of
>>>> AGE
>>>> > SLF4J: Class path contains multiple SLF4J bindings.
>>>> > SLF4J: Found binding in
>>>> [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> > SLF4J: Found binding in
>>>> [jar:file:/opt/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>> explanation.
>>>> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>> > 14/08/25 10:13:03 INFO Configuration.deprecation: mapred.jar is
>>>> deprecated. Instead, use mapreduce.job.jar
>>>> > 14/08/25 10:13:06 INFO manager.SqlManager: Executing SQL statement:
>>>> SELECT t.* FROM AGE AS t WHERE 1=0
>>>> > 14/08/25 10:13:06 INFO Configuration.deprecation:
>>>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
>>>> mapreduce.reduce.speculative
>>>> > 14/08/25 10:13:06 INFO Configuration.deprecation:
>>>> mapred.map.tasks.speculative.execution is deprecated. Instead, use
>>>> mapreduce.map.speculative
>>>> > 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks is
>>>> deprecated. Instead, use mapreduce.job.maps
>>>> > 14/08/25 10:13:06 INFO client.RMProxy: Connecting to ResourceManager
>>>> at nn01/10.10.200.74:8032
>>>> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to
>>>> process : 1
>>>> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to
>>>> process : 1
>>>> > 14/08/25 10:13:27 INFO mapreduce.JobSubmitter: number of splits:1
>>>> > 14/08/25 10:13:28 INFO mapreduce.JobSubmitter: Submitting tokens for
>>>> job: job_1408941947869_0001
>>>> > 14/08/25 10:13:30 INFO impl.YarnClientImpl: Submitted application
>>>> application_1408941947869_0001
>>>> > 14/08/25 10:13:30 INFO mapreduce.Job: The url to track the job:
>>>> http://nn01:8088/proxy/application_1408941947869_0001/
>>>> > 14/08/25 10:13:30 INFO mapreduce.Job: Running job:
>>>> job_1408941947869_0001
>>>> > 14/08/25 10:13:44 INFO mapreduce.Job: Job job_1408941947869_0001
>>>> running in uber mode : false
>>>> > 14/08/25 10:13:44 INFO mapreduce.Job:  map 0% reduce 0%
>>>> > 14/08/25 10:14:11 INFO mapreduce.Job:  map 100% reduce 0%
>>>> > 14/08/25 10:14:11 INFO mapreduce.Job: Task Id :
>>>> attempt_1408941947869_0001_m_000000_0, Status : FAILED
>>>> > Error: java.io.IOException: java.sql.SQLException:
>>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
>>>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>>>> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>>> >       at java.security.AccessController.doPrivileged(Native Method)
>>>> >       at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> >       at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>>>> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>>> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR:
>>>> Syntax error at or near ","
>>>> >       at com.vertica.util.ServerErrorData.buildException(Unknown
>>>> Source)
>>>> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
>>>> >       at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown
>>>> Source)
>>>> >       at
>>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
>>>> >       at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
>>>> >       ... 8 more
>>>> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856)
>>>> ERROR: Syntax error at or near ","
>>>> >       ... 20 more
>>>> >
>>>> > 14/08/25 10:14:12 INFO mapreduce.Job:  map 0% reduce 0%
>>>> > 14/08/25 10:14:25 INFO mapreduce.Job: Task Id :
>>>> attempt_1408941947869_0001_m_000000_1, Status : FAILED
>>>> > Error: java.io.IOException: java.sql.SQLException:
>>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
>>>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>>>> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>>> >       at java.security.AccessController.doPrivileged(Native Method)
>>>> >       at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> >       at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>>>> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>>> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR:
>>>> Syntax error at or near ","
>>>> >       at com.vertica.util.ServerErrorData.buildException(Unknown
>>>> Source)
>>>> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
>>>> >       at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown
>>>> Source)
>>>> >       at
>>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
>>>> >       at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
>>>> >       ... 8 more
>>>> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856)
>>>> ERROR: Syntax error at or near ","
>>>> >       ... 20 more
>>>> >
>>>> > 14/08/25 10:14:34 INFO mapreduce.Job: Task Id :
>>>> attempt_1408941947869_0001_m_000000_2, Status : FAILED
>>>> > Error: java.io.IOException: java.sql.SQLSyntaxErrorException:
>>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
>>>> >       at
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
>>>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>>>> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>>> >       at java.security.AccessController.doPrivileged(Native Method)
>>>> >       at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> >       at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>>>> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>>> > Caused by: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856)
>>>> ERROR: Syntax error at or near ","
>>>> >       at com.vertica.util.ServerErrorData.buildException(Unknown
>>>> Source)
>>>> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown
>>>> Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
>>>> >       at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown
>>>> Source)
>>>> >       at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown
>>>> Source)
>>>> >       at
>>>> com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown Source)
>>>> >       at
>>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
>>>> >       at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown
>>>> Source)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
>>>> >       at
>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
>>>> >       ... 8 more
>>>> > Caused by: com.vertica.support.exceptions.SyntaxErrorException:
>>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
>>>> >       ... 21 more
>>>> >
>>>> > 14/08/25 10:14:43 INFO mapreduce.Job:  map 100% reduce 0%
>>>> > 14/08/25 10:14:44 INFO mapreduce.Job: Job job_1408941947869_0001
>>>> failed with state FAILED due to: Task failed
>>>> task_1408941947869_0001_m_000000
>>>> > Job failed as tasks failed. failedMaps:1 failedReduces:0
>>>> >
>>>> > 14/08/25 10:14:45 INFO mapreduce.Job: Counters: 9
>>>> >       Job Counters
>>>> >               Failed map tasks=4
>>>> >               Launched map tasks=4
>>>> >               Other local map tasks=3
>>>> >               Rack-local map tasks=1
>>>> >               Total time spent by all maps in occupied slots
>>>> (ms)=53177
>>>> >               Total time spent by all reduces in occupied slots (ms)=0
>>>> >               Total time spent by all map tasks (ms)=53177
>>>> >               Total vcore-seconds taken by all map tasks=53177
>>>> >               Total megabyte-seconds taken by all map tasks=54453248
>>>> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group FileSystemCounters
>>>> is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
>>>> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes
>>>> in 99.0548 seconds (0 bytes/sec)
>>>> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group
>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>>> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Exported 0 records.
>>>> > 14/08/25 10:14:45 ERROR tool.ExportTool: Error during export: Export
>>>> job failed!
>>>> > Regards
>>>> > Muthupandi.K
>>>> >
>>>> >
>>>> >
>>>> > <image002.png>
>>>> >
>>>>
>>>>
>>>
>>
>

Mime
View raw message