sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Exporting to non standard databases
Date Mon, 25 Aug 2014 05:10:13 GMT
Hi Muthu,
the given message is a “Warning” and not an “Error”. Sqoop is warning you that you
are using —driver option that will force Sqoop to use Generic JDBC connector. That is fine
in your case as Sqoop don’t have specialized connector for Vertica. You can always use --connect-manager
parameter to get rid of the message as is the warning suggesting:

	sqoop export --connect-manager org.apache.sqoop.manager.GenericJdbcManager ...

You are however hitting a different problem, my best guess that Vertica is not supporting
multi value insert statements that Sqoop is generating by default. Try to disable this feature
using property sqoop.export.records.per.statement:

	sqoop export -Dsqoop.export.records.per.statement=1 —connect ...

Jarcec

On Aug 25, 2014, at 6:55 AM, Muthu Pandi <muthu1086@gmail.com> wrote:

> Hi every one,
>                Is it possible to export data from HDFS to vetica.I successfully imported
data from vertica to HDFS using sqoop.When i Export i get the following error.Please give
me solution as soon as possible.this is the error which am getting,
> 
> WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate
connection manager is not being set (via --connection-manager). Sqoop is going to fall back
to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection
manager should be used next time.
> 
> will above will have impact.How do i create connection manager for vertica.
> 
> My sqoop query is
> 
> sqoop export  --driver com.vertica.jdbc.Driver --connect "jdbc:vertica://X.X.X.X:5433/dbname"
--username username -P --table tablename  -m 1 --export-dir /ver.txt  --input-fields-terminated-by
',' --lines-terminated-by '\n' 
> 
> 
> and am using hadoop2.4.1 and sqoop1.4.4 version
> 
> 
> 
> 14/08/25 10:12:57 WARN tool.BaseSqoopTool: Setting your password on the command-line
is insecure. Consider using -P instead.
> 14/08/25 10:12:57 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver
however appropriate connection manager is not being set (via --connection-manager). Sqoop
is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
which connection manager should be used next time.
> 14/08/25 10:12:57 INFO manager.SqlManager: Using default fetchSize of 1000
> 14/08/25 10:12:57 INFO tool.CodeGenTool: Beginning code generation
> 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM AGE
AS t WHERE 1=0
> 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM AGE
AS t WHERE 1=0
> 14/08/25 10:12:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.4.1
> Note: /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.java uses or overrides
a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 14/08/25 10:13:02 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.jar
> 14/08/25 10:13:03 INFO mapreduce.ExportJobBase: Beginning export of AGE
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/08/25 10:13:03 INFO Configuration.deprecation: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
> 14/08/25 10:13:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM AGE
AS t WHERE 1=0
> 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution
is deprecated. Instead, use mapreduce.reduce.speculative
> 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution
is deprecated. Instead, use mapreduce.map.speculative
> 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
> 14/08/25 10:13:06 INFO client.RMProxy: Connecting to ResourceManager at nn01/10.10.200.74:8032
> 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/08/25 10:13:27 INFO mapreduce.JobSubmitter: number of splits:1
> 14/08/25 10:13:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1408941947869_0001
> 14/08/25 10:13:30 INFO impl.YarnClientImpl: Submitted application application_1408941947869_0001
> 14/08/25 10:13:30 INFO mapreduce.Job: The url to track the job: http://nn01:8088/proxy/application_1408941947869_0001/
> 14/08/25 10:13:30 INFO mapreduce.Job: Running job: job_1408941947869_0001
> 14/08/25 10:13:44 INFO mapreduce.Job: Job job_1408941947869_0001 running in uber mode
: false
> 14/08/25 10:13:44 INFO mapreduce.Job:  map 0% reduce 0%
> 14/08/25 10:14:11 INFO mapreduce.Job:  map 100% reduce 0%
> 14/08/25 10:14:11 INFO mapreduce.Job: Task Id : attempt_1408941947869_0001_m_000000_0,
Status : FAILED
> Error: java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax
error at or near ","
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax error at or near
","
> 	at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> 	at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> 	at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown Source)
> 	at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> 	at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> 	at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> 	... 8 more
> Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) ERROR: Syntax error
at or near ","
> 	... 20 more
> 
> 14/08/25 10:14:12 INFO mapreduce.Job:  map 0% reduce 0%
> 14/08/25 10:14:25 INFO mapreduce.Job: Task Id : attempt_1408941947869_0001_m_000000_1,
Status : FAILED
> Error: java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax
error at or near ","
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax error at or near
","
> 	at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> 	at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> 	at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown Source)
> 	at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> 	at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> 	at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> 	... 8 more
> Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) ERROR: Syntax error
at or near ","
> 	... 20 more
> 
> 14/08/25 10:14:34 INFO mapreduce.Job: Task Id : attempt_1408941947869_0001_m_000000_2,
Status : FAILED
> Error: java.io.IOException: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856)
ERROR: Syntax error at or near ","
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856) ERROR: Syntax error
at or near ","
> 	at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> 	at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> 	at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown Source)
> 	at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown Source)
> 	at com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown Source)
> 	at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
> 	at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown Source)
> 	at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> 	... 8 more
> Caused by: com.vertica.support.exceptions.SyntaxErrorException: [Vertica][VJDBC](4856)
ERROR: Syntax error at or near ","
> 	... 21 more
> 
> 14/08/25 10:14:43 INFO mapreduce.Job:  map 100% reduce 0%
> 14/08/25 10:14:44 INFO mapreduce.Job: Job job_1408941947869_0001 failed with state FAILED
due to: Task failed task_1408941947869_0001_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 
> 14/08/25 10:14:45 INFO mapreduce.Job: Counters: 9
> 	Job Counters 
> 		Failed map tasks=4
> 		Launched map tasks=4
> 		Other local map tasks=3
> 		Rack-local map tasks=1
> 		Total time spent by all maps in occupied slots (ms)=53177
> 		Total time spent by all reduces in occupied slots (ms)=0
> 		Total time spent by all map tasks (ms)=53177
> 		Total vcore-seconds taken by all map tasks=53177
> 		Total megabyte-seconds taken by all map tasks=54453248
> 14/08/25 10:14:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use
org.apache.hadoop.mapreduce.FileSystemCounter instead
> 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 99.0548 seconds
(0 bytes/sec)
> 14/08/25 10:14:45 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter
is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
> 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Exported 0 records.
> 14/08/25 10:14:45 ERROR tool.ExportTool: Error during export: Export job failed!
> Regards
> Muthupandi.K
> 
> 
> 
> <image002.png>
> 


Mime
View raw message