sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mengwei Ding <mengwei.d...@gmail.com>
Subject Re: Import data to HDFS using Sqoop2
Date Thu, 05 Sep 2013 02:32:11 GMT
Hmm... would you mind showing us your most updated job configuration by
typing "show job --jid 3"? I just want to make sure that you provide the
partition column correctly.

Also, I notice that the primary key for this table is "VARCHAR(23)" type,
this might be the problem.

Mengwei


On Wed, Sep 4, 2013 at 10:23 PM, Yanting Chen <mx.alexender@gmail.com>wrote:

> Hi Mengwei
>
> I try to set primary key into partition column. But still the same error!
>
>
> On Thu, Sep 5, 2013 at 10:17 AM, Mengwei Ding <mengwei.ding@gmail.com>wrote:
>
>> Hi Yanting,
>>
>> It seems like you did not specify the 'partition column' for the job.
>> Generally, the primary key of  the table is a good choice for 'parition
>> column'.
>>
>> You could use 'update job -jid 3' to update the job configuration.
>>
>> Mengwei
>>
>>
>> On Wed, Sep 4, 2013 at 9:37 PM, Yanting Chen <mx.alexender@gmail.com>wrote:
>>
>>> Abraham,
>>>
>>> Thanks for you answer. I review my database.
>>> I think the database name is invoice and the schema name is public, just
>>> like the picture below.
>>> http://imgur.com/ns0iNLi
>>>
>>> So, I change the schema name with a new value, "public"
>>> Then, I run this job and get a different error.
>>>
>>> Status: FAILURE_ON_SUBMIT
>>> Creation date: 2013-09-05 09:30:44 CST
>>> Last update date: 2013-09-05 09:30:44 CST
>>> Exception: org.apache.sqoop.common.SqoopException:
>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>> Stack trace: org.apache.sqoop.common.SqoopException:
>>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>>> at
>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:87)
>>>  at
>>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:32)
>>> at
>>> org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:71)
>>>  at
>>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:452)
>>> at
>>> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:469)
>>>  at
>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366)
>>> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
>>>  at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
>>> at
>>> org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:265)
>>>  at
>>> org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:480)
>>> at
>>> org.apache.sqoop.handler.SubmissionRequestHandler.submissionSubmit(SubmissionRequestHandler.java:112)
>>>  at
>>> org.apache.sqoop.handler.SubmissionRequestHandler.handleActionEvent(SubmissionRequestHandler.java:98)
>>> at
>>> org.apache.sqoop.handler.SubmissionRequestHandler.handleEvent(SubmissionRequestHandler.java:68)
>>>  at
>>> org.apache.sqoop.server.v1.SubmissionServlet.handlePostRequest(SubmissionServlet.java:44)
>>> at
>>> org.apache.sqoop.server.SqoopProtocolServlet.doPost(SqoopProtocolServlet.java:63)
>>>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>  at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>  at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>  at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>  at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>>  at
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>>> at
>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>>>  at
>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>> Also, I try to remove the schema name, but I get the same error as above.
>>>
>>> On Thu, Sep 5, 2013 at 6:11 AM, Abraham Elmahrek <abe@cloudera.com>
>>>  wrote:
>>>
>>> Yanting,
>>>>
>>>> I'm sorry, I'm a bit confused. The database you are using here is
>>>> called "invoice". Do you also have a schema called "invoice"? Try removing
>>>> the "Schema name" from your job and see if that works for you.
>>>>
>>>> http://www.postgresql.org/docs/9.1/static/ddl-schemas.html
>>>>
>>>> -Abe
>>>>
>>>>
>>>> On Tue, Sep 3, 2013 at 6:34 PM, Yanting Chen <mx.alexender@gmail.com>
>>>>  wrote:
>>>>
>>>>> However, with Sqoop 1.4.4, all schema and table names are lower-case,
>>>>> and it is successful to make import.
>>>>>
>>>>> As a result, I am not sure if case sensitivity will be a problem in
>>>>> Sqoop 2.
>>>>>
>>>>> *This is the connection I created:*
>>>>> 1 connection(s) to show:
>>>>> Connection with id 1 and name ds_msg_log (Created 9/2/13 10:54 AM,
>>>>> Updated 9/3/13 6:35 PM)
>>>>> Using Connector id 1
>>>>>   Connection configuration
>>>>>     JDBC Driver Class: org.postgresql.Driver
>>>>>     JDBC Connection String: jdbc:postgresql://MY_IP:MY_PORT/invoice
>>>>>     Username: hadoop
>>>>>     Password:
>>>>>     JDBC Connection Properties:
>>>>>   Security related configuration options
>>>>>     Max connections: 123
>>>>>
>>>>> *This is the job I created:*
>>>>> Job with id 3 and name ds_msg_log (Created 9/2/13 11:37 AM, Updated
>>>>> 9/3/13 5:57 PM)
>>>>> Using Connection id 1 and Connector id 1
>>>>>   Database configuration
>>>>>     Schema name: invoice
>>>>>     Table name: ds_msg_log
>>>>>     Table SQL statement:
>>>>>     Table column names:
>>>>>     Partition column name:
>>>>>     Boundary query:
>>>>>   Output configuration
>>>>>     Storage type: HDFS
>>>>>     Output format: TEXT_FILE
>>>>>     Output directory: /user/root/ds_msg_log_sqoop2
>>>>>   Throttling resources
>>>>>     Extractors:
>>>>>     Loaders:
>>>>>
>>>>> *The command I use to start this job is:*
>>>>> submission start --jid 3
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2013 at 9:24 AM, Abraham Elmahrek <abe@cloudera.com>
>>>>>  wrote:
>>>>>
>>>>>> Sorry, I missed that you are using PostGreSQL. I think case
>>>>>> sensitivity will be a problem with that database.
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 3, 2013 at 6:23 PM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>  wrote:
>>>>>>
>>>>>>> What database are you using? Are table names or schema names
case
>>>>>>> sensitive in your database? Sqoop quotes all names, so case sensitivity
>>>>>>> makes a difference here.
>>>>>>>
>>>>>>> Also, could you provide the corresponding Sqoop 1.4.4 command
that
>>>>>>> worked for you?
>>>>>>>
>>>>>>> -Abe
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 3, 2013 at 6:06 PM, 陳彥廷(Yanting Chen) <
>>>>>>> mx.alexender@gmail.com> wrote:
>>>>>>>
>>>>>>>> I am pretty sure that the schema "invoice" exists because
I
>>>>>>>> successfully imported data to this schema using Sqoop 1.4.4
>>>>>>>>
>>>>>>>> Also, I already remove "*" when creating job.
>>>>>>>>
>>>>>>>> The following is the table in invoice schema:
>>>>>>>>     TABLE DS_MSG_LOG
>>>>>>>>     (
>>>>>>>>         MESSAGE_ID VARCHAR2(23) NOT NULL,
>>>>>>>>         LOGIN_ID VARCHAR2(30),
>>>>>>>>         ACPT_DTS TIMESTAMP(6),
>>>>>>>>         DLVR_DTS TIMESTAMP(6),
>>>>>>>>         SENDER_ID VARCHAR2(30),
>>>>>>>>         SENDER_VAC_ID VARCHAR2(39),
>>>>>>>>         RECEIVER_ID VARCHAR2(30),
>>>>>>>>         RECEIVER_VAC_ID VARCHAR2(39),
>>>>>>>>         STATUS VARCHAR2(1),
>>>>>>>>         MESSAGE_TYPE VARCHAR2(8),
>>>>>>>>         FLOW_TYPE VARCHAR2(5),
>>>>>>>>         SERVICE_TYPE VARCHAR2(1),
>>>>>>>>         SOURCE_FILE_NAME VARCHAR2(150),
>>>>>>>>         ARCHIVE_FILE_NAME VARCHAR2(250),
>>>>>>>>         ARCHIVE_CHAR_COUNT NUMBER,
>>>>>>>>         DECRYPT_FILE_NAME VARCHAR2(250),
>>>>>>>>         DECRYPT_CHAR_COUNT NUMBER,
>>>>>>>>         RESP_FILE_NAME VARCHAR2(250),
>>>>>>>>         RESP_CHAR_COUNT NUMBER,
>>>>>>>>         RESP_FLAG VARCHAR2(1),
>>>>>>>>         RTG_SEQ VARCHAR2(8),
>>>>>>>>         RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
>>>>>>>>         TOTAL_INV_COUNT NUMBER,
>>>>>>>>         CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
>>>>>>>>     )
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2013 at 1:46 AM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>>> Hey User,
>>>>>>>>>
>>>>>>>>> It looks like the schema "invoice" does not exist in
your
>>>>>>>>> database. Could you please provide your database schema?
Also, "*" is
>>>>>>>>> unnecessary when specifying "Table column names". If
you leave it blank it
>>>>>>>>> will import all columns by default.
>>>>>>>>>
>>>>>>>>> -Abe
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 3, 2013 at 3:03 AM, 陳彥廷(Yanting Chen)
<
>>>>>>>>> mx.alexender@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> According to the official guide,
>>>>>>>>>> http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html
, I
>>>>>>>>>> successfully created a job.
>>>>>>>>>>
>>>>>>>>>> However, when I executed the command, submission
start --jid 1, I
>>>>>>>>>> got this error message: "Exception has occurred during
processing command
>>>>>>>>>> Server has returned exception: Exception: java.lang.Throwable
Message:
>>>>>>>>>> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the
SQL statement"
>>>>>>>>>>
>>>>>>>>>> This is the information of my job.
>>>>>>>>>>
>>>>>>>>>> Database configuration
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Schema name: invoice
>>>>>>>>>> Table name: ds_msg_log
>>>>>>>>>> Table SQL statement:
>>>>>>>>>> Table column names: *
>>>>>>>>>> Partition column name:
>>>>>>>>>> Boundary query:
>>>>>>>>>>
>>>>>>>>>> Output configuration
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Storage type: HDFS
>>>>>>>>>> Output format: TEXT_FILE
>>>>>>>>>> Output directory: /user/root/ds_msg_log
>>>>>>>>>>
>>>>>>>>>> Throttling resources Extractors: Loaders:
>>>>>>>>>>
>>>>>>>>>> Since there is no information in the official guide
talking about
>>>>>>>>>> how to set the values above, does any know anything
wrong in my job setting?
>>>>>>>>>>
>>>>>>>>>> This is the log: Stack trace: at
>>>>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor
>>>>>>>>>> (GenericJdbcExecutor.java:59)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:155)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:48)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:37)
>>>>>>>>>> at org.apache.sqoop.framework.FrameworkManager
>>>>>>>>>> (FrameworkManager.java:447)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:112)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:98)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:68)
>>>>>>>>>> at org.apache.sqoop.server.v1.SubmissionServlet
>>>>>>>>>> (SubmissionServlet.java:44)
>>>>>>>>>> at org.apache.sqoop.server.SqoopProtocolServlet
>>>>>>>>>> (SqoopProtocolServlet.java:63)
>>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:637)
>>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:717)
>>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>>> (ApplicationFilterChain.java:290)
>>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>>> (ApplicationFilterChain.java:206)
>>>>>>>>>> at org.apache.catalina.core.StandardWrapperValve
>>>>>>>>>> (StandardWrapperValve.java:233)
>>>>>>>>>> at org.apache.catalina.core.StandardContextValve
>>>>>>>>>> (StandardContextValve.java:191)
>>>>>>>>>> at org.apache.catalina.core.StandardHostValve
>>>>>>>>>> (StandardHostValve.java:127)
>>>>>>>>>> at org.apache.catalina.valves.ErrorReportValve
>>>>>>>>>> (ErrorReportValve.java:102)
>>>>>>>>>> at org.apache.catalina.core.StandardEngineValve
>>>>>>>>>> (StandardEngineValve.java:109)
>>>>>>>>>> at org.apache.catalina.connector.CoyoteAdapter
>>>>>>>>>> (CoyoteAdapter.java:293)
>>>>>>>>>> at org.apache.coyote.http11.Http11Processor
>>>>>>>>>> (Http11Processor.java:859)
>>>>>>>>>> at
>>>>>>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler
>>>>>>>>>> (Http11Protocol.java:602)
>>>>>>>>>> at org.apache.tomcat.util.net.JIoEndpoint$Worker
>>>>>>>>>> (JIoEndpoint.java:489)
>>>>>>>>>> at java.lang.Thread (Thread.java:724)
>>>>>>>>>> Caused by: Exception: java.lang.Throwable Message:
ERROR: schema
>>>>>>>>>> "invoice" does not exist Position: 46 Stack trace:
at
>>>>>>>>>> org.postgresql.core.v3.QueryExecutorImpl (QueryExecutorImpl.java:2102)
>>>>>>>>>> at org.postgresql.core.v3.QueryExecutorImpl
>>>>>>>>>> (QueryExecutorImpl.java:1835)
>>>>>>>>>> at org.postgresql.core.v3.QueryExecutorImpl
>>>>>>>>>> (QueryExecutorImpl.java:257)
>>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>>> (AbstractJdbc2Statement.java:500)
>>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>>> (AbstractJdbc2Statement.java:374)
>>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>>> (AbstractJdbc2Statement.java:254)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor
>>>>>>>>>> (GenericJdbcExecutor.java:56)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:155)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:48)
>>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>>> (GenericJdbcImportInitializer.java:37)
>>>>>>>>>> at org.apache.sqoop.framework.FrameworkManager
>>>>>>>>>> (FrameworkManager.java:447)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:112)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:98)
>>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>>> (SubmissionRequestHandler.java:68)
>>>>>>>>>> at org.apache.sqoop.server.v1.SubmissionServlet
>>>>>>>>>> (SubmissionServlet.java:44)
>>>>>>>>>> at org.apache.sqoop.server.SqoopProtocolServlet
>>>>>>>>>> (SqoopProtocolServlet.java:63)
>>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:637)
>>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:717)
>>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>>> (ApplicationFilterChain.java:290)
>>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>>> (ApplicationFilterChain.java:206)
>>>>>>>>>> at org.apache.catalina.core.StandardWrapperValve
>>>>>>>>>> (StandardWrapperValve.java:233)
>>>>>>>>>> at org.apache.catalina.core.StandardContextValve
>>>>>>>>>> (StandardContextValve.java:191)
>>>>>>>>>> at org.apache.catalina.core.StandardHostValve
>>>>>>>>>> (StandardHostValve.java:127)
>>>>>>>>>> at org.apache.catalina.valves.ErrorReportValve
>>>>>>>>>> (ErrorReportValve.java:102)
>>>>>>>>>> at org.apache.catalina.core.StandardEngineValve
>>>>>>>>>> (StandardEngineValve.java:109)
>>>>>>>>>> at org.apache.catalina.connector.CoyoteAdapter
>>>>>>>>>> (CoyoteAdapter.java:293)
>>>>>>>>>> at org.apache.coyote.http11.Http11Processor
>>>>>>>>>> (Http11Processor.java:859)
>>>>>>>>>> at
>>>>>>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler
>>>>>>>>>> (Http11Protocol.java:602)
>>>>>>>>>> at org.apache.tomcat.util.net.JIoEndpoint$Worker
>>>>>>>>>> (JIoEndpoint.java:489)
>>>>>>>>>> at java.lang.Thread (Thread.java:724)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>

Mime
View raw message