sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muthu Pandi <muthu1...@gmail.com>
Subject Re: Exporting to non standard databases
Date Mon, 25 Aug 2014 05:23:09 GMT
Spot on!!! Jarcec

       Thanks for the quick reply, you have saved my lot of time.

       Can this be used for exporting larger data since we are exporting in
bulk. My data will be of size 2GB Per day and will export at end of the day.




*Regards Muthupandi.K*

 [image: Picture (Device Independent Bitmap)]



On Mon, Aug 25, 2014 at 10:40 AM, Jarek Jarcec Cecho <jarcec@apache.org>
wrote:

> Hi Muthu,
> the given message is a “Warning” and not an “Error”. Sqoop is warning you
> that you are using —driver option that will force Sqoop to use Generic JDBC
> connector. That is fine in your case as Sqoop don’t have specialized
> connector for Vertica. You can always use --connect-manager parameter to
> get rid of the message as is the warning suggesting:
>
>         sqoop export --connect-manager
> org.apache.sqoop.manager.GenericJdbcManager ...
>
> You are however hitting a different problem, my best guess that Vertica is
> not supporting multi value insert statements that Sqoop is generating by
> default. Try to disable this feature using property
> sqoop.export.records.per.statement:
>
>         sqoop export -Dsqoop.export.records.per.statement=1 —connect ...
>
> Jarcec
>
> On Aug 25, 2014, at 6:55 AM, Muthu Pandi <muthu1086@gmail.com> wrote:
>
> > Hi every one,
> >                Is it possible to export data from HDFS to vetica.I
> successfully imported data from vertica to HDFS using sqoop.When i Export i
> get the following error.Please give me solution as soon as possible.this is
> the error which am getting,
> >
> > WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver
> however appropriate connection manager is not being set (via
> --connection-manager). Sqoop is going to fall back to
> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
> which connection manager should be used next time.
> >
> > will above will have impact.How do i create connection manager for
> vertica.
> >
> > My sqoop query is
> >
> > sqoop export  --driver com.vertica.jdbc.Driver --connect
> "jdbc:vertica://X.X.X.X:5433/dbname" --username username -P --table
> tablename  -m 1 --export-dir /ver.txt  --input-fields-terminated-by ','
> --lines-terminated-by '\n'
> >
> >
> > and am using hadoop2.4.1 and sqoop1.4.4 version
> >
> >
> >
> > 14/08/25 10:12:57 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> > 14/08/25 10:12:57 WARN sqoop.ConnFactory: Parameter --driver is set to
> an explicit driver however appropriate connection manager is not being set
> (via --connection-manager). Sqoop is going to fall back to
> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
> which connection manager should be used next time.
> > 14/08/25 10:12:57 INFO manager.SqlManager: Using default fetchSize of
> 1000
> > 14/08/25 10:12:57 INFO tool.CodeGenTool: Beginning code generation
> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM AGE AS t WHERE 1=0
> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM AGE AS t WHERE 1=0
> > 14/08/25 10:12:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /opt/hadoop-2.4.1
> > Note:
> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.java uses
> or overrides a deprecated API.
> > Note: Recompile with -Xlint:deprecation for details.
> > 14/08/25 10:13:02 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.jar
> > 14/08/25 10:13:03 INFO mapreduce.ExportJobBase: Beginning export of AGE
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> [jar:file:/opt/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > 14/08/25 10:13:03 INFO Configuration.deprecation: mapred.jar is
> deprecated. Instead, use mapreduce.job.jar
> > 14/08/25 10:13:06 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM AGE AS t WHERE 1=0
> > 14/08/25 10:13:06 INFO Configuration.deprecation:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> > 14/08/25 10:13:06 INFO Configuration.deprecation:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> > 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> > 14/08/25 10:13:06 INFO client.RMProxy: Connecting to ResourceManager at
> nn01/10.10.200.74:8032
> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to
> process : 1
> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to
> process : 1
> > 14/08/25 10:13:27 INFO mapreduce.JobSubmitter: number of splits:1
> > 14/08/25 10:13:28 INFO mapreduce.JobSubmitter: Submitting tokens for
> job: job_1408941947869_0001
> > 14/08/25 10:13:30 INFO impl.YarnClientImpl: Submitted application
> application_1408941947869_0001
> > 14/08/25 10:13:30 INFO mapreduce.Job: The url to track the job:
> http://nn01:8088/proxy/application_1408941947869_0001/
> > 14/08/25 10:13:30 INFO mapreduce.Job: Running job: job_1408941947869_0001
> > 14/08/25 10:13:44 INFO mapreduce.Job: Job job_1408941947869_0001 running
> in uber mode : false
> > 14/08/25 10:13:44 INFO mapreduce.Job:  map 0% reduce 0%
> > 14/08/25 10:14:11 INFO mapreduce.Job:  map 100% reduce 0%
> > 14/08/25 10:14:11 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_0, Status : FAILED
> > Error: java.io.IOException: java.sql.SQLException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> >       at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> >       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at javax.security.auth.Subject.doAs(Subject.java:415)
> >       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax
> error at or near ","
> >       at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
> Source)
> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> >       at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown
> Source)
> >       at
> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> >       at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> >       at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> >       ... 8 more
> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856)
> ERROR: Syntax error at or near ","
> >       ... 20 more
> >
> > 14/08/25 10:14:12 INFO mapreduce.Job:  map 0% reduce 0%
> > 14/08/25 10:14:25 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_1, Status : FAILED
> > Error: java.io.IOException: java.sql.SQLException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> >       at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> >       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at javax.security.auth.Subject.doAs(Subject.java:415)
> >       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax
> error at or near ","
> >       at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
> Source)
> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> >       at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown
> Source)
> >       at
> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> >       at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> >       at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> >       ... 8 more
> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856)
> ERROR: Syntax error at or near ","
> >       ... 20 more
> >
> > 14/08/25 10:14:34 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_2, Status : FAILED
> > Error: java.io.IOException: java.sql.SQLSyntaxErrorException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> >       at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> >       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> >       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at javax.security.auth.Subject.doAs(Subject.java:415)
> >       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> >       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > Caused by: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856)
> ERROR: Syntax error at or near ","
> >       at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> >       at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown
> Source)
> >       at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> >       at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown
> Source)
> >       at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown
> Source)
> >       at
> com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown Source)
> >       at
> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> >       at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown
> Source)
> >       at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> >       at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> >       ... 8 more
> > Caused by: com.vertica.support.exceptions.SyntaxErrorException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> >       ... 21 more
> >
> > 14/08/25 10:14:43 INFO mapreduce.Job:  map 100% reduce 0%
> > 14/08/25 10:14:44 INFO mapreduce.Job: Job job_1408941947869_0001 failed
> with state FAILED due to: Task failed task_1408941947869_0001_m_000000
> > Job failed as tasks failed. failedMaps:1 failedReduces:0
> >
> > 14/08/25 10:14:45 INFO mapreduce.Job: Counters: 9
> >       Job Counters
> >               Failed map tasks=4
> >               Launched map tasks=4
> >               Other local map tasks=3
> >               Rack-local map tasks=1
> >               Total time spent by all maps in occupied slots (ms)=53177
> >               Total time spent by all reduces in occupied slots (ms)=0
> >               Total time spent by all map tasks (ms)=53177
> >               Total vcore-seconds taken by all map tasks=53177
> >               Total megabyte-seconds taken by all map tasks=54453248
> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group FileSystemCounters is
> deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes in
> 99.0548 seconds (0 bytes/sec)
> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Exported 0 records.
> > 14/08/25 10:14:45 ERROR tool.ExportTool: Error during export: Export job
> failed!
> > Regards
> > Muthupandi.K
> >
> >
> >
> > <image002.png>
> >
>
>

Mime
View raw message