sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yanting Chen <mx.alexen...@gmail.com>
Subject Re: Import data to HDFS using Sqoop2
Date Thu, 05 Sep 2013 02:23:48 GMT
Hi Mengwei

I try to set primary key into partition column. But still the same error!


On Thu, Sep 5, 2013 at 10:17 AM, Mengwei Ding <mengwei.ding@gmail.com>wrote:

> Hi Yanting,
>
> It seems like you did not specify the 'partition column' for the job.
> Generally, the primary key of  the table is a good choice for 'parition
> column'.
>
> You could use 'update job -jid 3' to update the job configuration.
>
> Mengwei
>
>
> On Wed, Sep 4, 2013 at 9:37 PM, Yanting Chen <mx.alexender@gmail.com>wrote:
>
>> Abraham,
>>
>> Thanks for you answer. I review my database.
>> I think the database name is invoice and the schema name is public, just
>> like the picture below.
>> http://imgur.com/ns0iNLi
>>
>> So, I change the schema name with a new value, "public"
>> Then, I run this job and get a different error.
>>
>> Status: FAILURE_ON_SUBMIT
>> Creation date: 2013-09-05 09:30:44 CST
>> Last update date: 2013-09-05 09:30:44 CST
>> Exception: org.apache.sqoop.common.SqoopException:
>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>> Stack trace: org.apache.sqoop.common.SqoopException:
>> GENERIC_JDBC_CONNECTOR_0011:The type is not supported - 12
>> at
>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:87)
>>  at
>> org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner.getPartitions(GenericJdbcImportPartitioner.java:32)
>> at
>> org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:71)
>>  at
>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:452)
>> at
>> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:469)
>>  at
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366)
>> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
>>  at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
>> at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
>> at
>> org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:265)
>>  at
>> org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:480)
>> at
>> org.apache.sqoop.handler.SubmissionRequestHandler.submissionSubmit(SubmissionRequestHandler.java:112)
>>  at
>> org.apache.sqoop.handler.SubmissionRequestHandler.handleActionEvent(SubmissionRequestHandler.java:98)
>> at
>> org.apache.sqoop.handler.SubmissionRequestHandler.handleEvent(SubmissionRequestHandler.java:68)
>>  at
>> org.apache.sqoop.server.v1.SubmissionServlet.handlePostRequest(SubmissionServlet.java:44)
>> at
>> org.apache.sqoop.server.SqoopProtocolServlet.doPost(SqoopProtocolServlet.java:63)
>>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>  at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>  at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>  at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>  at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>  at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>>  at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> Also, I try to remove the schema name, but I get the same error as above.
>>
>> On Thu, Sep 5, 2013 at 6:11 AM, Abraham Elmahrek <abe@cloudera.com>
>>  wrote:
>>
>> Yanting,
>>>
>>> I'm sorry, I'm a bit confused. The database you are using here is called
>>> "invoice". Do you also have a schema called "invoice"? Try removing the
>>> "Schema name" from your job and see if that works for you.
>>>
>>> http://www.postgresql.org/docs/9.1/static/ddl-schemas.html
>>>
>>> -Abe
>>>
>>>
>>> On Tue, Sep 3, 2013 at 6:34 PM, Yanting Chen <mx.alexender@gmail.com>
>>>  wrote:
>>>
>>>> However, with Sqoop 1.4.4, all schema and table names are lower-case,
>>>> and it is successful to make import.
>>>>
>>>> As a result, I am not sure if case sensitivity will be a problem in
>>>> Sqoop 2.
>>>>
>>>> *This is the connection I created:*
>>>> 1 connection(s) to show:
>>>> Connection with id 1 and name ds_msg_log (Created 9/2/13 10:54 AM,
>>>> Updated 9/3/13 6:35 PM)
>>>> Using Connector id 1
>>>>   Connection configuration
>>>>     JDBC Driver Class: org.postgresql.Driver
>>>>     JDBC Connection String: jdbc:postgresql://MY_IP:MY_PORT/invoice
>>>>     Username: hadoop
>>>>     Password:
>>>>     JDBC Connection Properties:
>>>>   Security related configuration options
>>>>     Max connections: 123
>>>>
>>>> *This is the job I created:*
>>>> Job with id 3 and name ds_msg_log (Created 9/2/13 11:37 AM, Updated
>>>> 9/3/13 5:57 PM)
>>>> Using Connection id 1 and Connector id 1
>>>>   Database configuration
>>>>     Schema name: invoice
>>>>     Table name: ds_msg_log
>>>>     Table SQL statement:
>>>>     Table column names:
>>>>     Partition column name:
>>>>     Boundary query:
>>>>   Output configuration
>>>>     Storage type: HDFS
>>>>     Output format: TEXT_FILE
>>>>     Output directory: /user/root/ds_msg_log_sqoop2
>>>>   Throttling resources
>>>>     Extractors:
>>>>     Loaders:
>>>>
>>>> *The command I use to start this job is:*
>>>> submission start --jid 3
>>>>
>>>>
>>>> On Wed, Sep 4, 2013 at 9:24 AM, Abraham Elmahrek <abe@cloudera.com>
>>>>  wrote:
>>>>
>>>>> Sorry, I missed that you are using PostGreSQL. I think case
>>>>> sensitivity will be a problem with that database.
>>>>>
>>>>>
>>>>> On Tue, Sep 3, 2013 at 6:23 PM, Abraham Elmahrek <abe@cloudera.com>
>>>>>  wrote:
>>>>>
>>>>>> What database are you using? Are table names or schema names case
>>>>>> sensitive in your database? Sqoop quotes all names, so case sensitivity
>>>>>> makes a difference here.
>>>>>>
>>>>>> Also, could you provide the corresponding Sqoop 1.4.4 command that
>>>>>> worked for you?
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 3, 2013 at 6:06 PM, 陳彥廷(Yanting Chen) <
>>>>>> mx.alexender@gmail.com> wrote:
>>>>>>
>>>>>>> I am pretty sure that the schema "invoice" exists because I
>>>>>>> successfully imported data to this schema using Sqoop 1.4.4
>>>>>>>
>>>>>>> Also, I already remove "*" when creating job.
>>>>>>>
>>>>>>> The following is the table in invoice schema:
>>>>>>>     TABLE DS_MSG_LOG
>>>>>>>     (
>>>>>>>         MESSAGE_ID VARCHAR2(23) NOT NULL,
>>>>>>>         LOGIN_ID VARCHAR2(30),
>>>>>>>         ACPT_DTS TIMESTAMP(6),
>>>>>>>         DLVR_DTS TIMESTAMP(6),
>>>>>>>         SENDER_ID VARCHAR2(30),
>>>>>>>         SENDER_VAC_ID VARCHAR2(39),
>>>>>>>         RECEIVER_ID VARCHAR2(30),
>>>>>>>         RECEIVER_VAC_ID VARCHAR2(39),
>>>>>>>         STATUS VARCHAR2(1),
>>>>>>>         MESSAGE_TYPE VARCHAR2(8),
>>>>>>>         FLOW_TYPE VARCHAR2(5),
>>>>>>>         SERVICE_TYPE VARCHAR2(1),
>>>>>>>         SOURCE_FILE_NAME VARCHAR2(150),
>>>>>>>         ARCHIVE_FILE_NAME VARCHAR2(250),
>>>>>>>         ARCHIVE_CHAR_COUNT NUMBER,
>>>>>>>         DECRYPT_FILE_NAME VARCHAR2(250),
>>>>>>>         DECRYPT_CHAR_COUNT NUMBER,
>>>>>>>         RESP_FILE_NAME VARCHAR2(250),
>>>>>>>         RESP_CHAR_COUNT NUMBER,
>>>>>>>         RESP_FLAG VARCHAR2(1),
>>>>>>>         RTG_SEQ VARCHAR2(8),
>>>>>>>         RESENT_FLAG VARCHAR2(1) DEFAULT 'N',
>>>>>>>         TOTAL_INV_COUNT NUMBER,
>>>>>>>         CONSTRAINT PK_DS_MSG_LOG PRIMARY KEY (MESSAGE_ID)
>>>>>>>     )
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 4, 2013 at 1:46 AM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>>> Hey User,
>>>>>>>>
>>>>>>>> It looks like the schema "invoice" does not exist in your
database.
>>>>>>>> Could you please provide your database schema? Also, "*"
is unnecessary
>>>>>>>> when specifying "Table column names". If you leave it blank
it will import
>>>>>>>> all columns by default.
>>>>>>>>
>>>>>>>> -Abe
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 3, 2013 at 3:03 AM, 陳彥廷(Yanting Chen) <
>>>>>>>> mx.alexender@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> According to the official guide,
>>>>>>>>> http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html
, I
>>>>>>>>> successfully created a job.
>>>>>>>>>
>>>>>>>>> However, when I executed the command, submission start
--jid 1, I
>>>>>>>>> got this error message: "Exception has occurred during
processing command
>>>>>>>>> Server has returned exception: Exception: java.lang.Throwable
Message:
>>>>>>>>> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL
statement"
>>>>>>>>>
>>>>>>>>> This is the information of my job.
>>>>>>>>>
>>>>>>>>> Database configuration
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Schema name: invoice
>>>>>>>>> Table name: ds_msg_log
>>>>>>>>> Table SQL statement:
>>>>>>>>> Table column names: *
>>>>>>>>> Partition column name:
>>>>>>>>> Boundary query:
>>>>>>>>>
>>>>>>>>> Output configuration
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Storage type: HDFS
>>>>>>>>> Output format: TEXT_FILE
>>>>>>>>> Output directory: /user/root/ds_msg_log
>>>>>>>>>
>>>>>>>>> Throttling resources Extractors: Loaders:
>>>>>>>>>
>>>>>>>>> Since there is no information in the official guide talking
about
>>>>>>>>> how to set the values above, does any know anything wrong
in my job setting?
>>>>>>>>>
>>>>>>>>> This is the log: Stack trace: at
>>>>>>>>> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor
>>>>>>>>> (GenericJdbcExecutor.java:59)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:155)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:48)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:37)
>>>>>>>>> at org.apache.sqoop.framework.FrameworkManager
>>>>>>>>> (FrameworkManager.java:447)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:112)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:98)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:68)
>>>>>>>>> at org.apache.sqoop.server.v1.SubmissionServlet
>>>>>>>>> (SubmissionServlet.java:44)
>>>>>>>>> at org.apache.sqoop.server.SqoopProtocolServlet
>>>>>>>>> (SqoopProtocolServlet.java:63)
>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:637)
>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:717)
>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>> (ApplicationFilterChain.java:290)
>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>> (ApplicationFilterChain.java:206)
>>>>>>>>> at org.apache.catalina.core.StandardWrapperValve
>>>>>>>>> (StandardWrapperValve.java:233)
>>>>>>>>> at org.apache.catalina.core.StandardContextValve
>>>>>>>>> (StandardContextValve.java:191)
>>>>>>>>> at org.apache.catalina.core.StandardHostValve
>>>>>>>>> (StandardHostValve.java:127)
>>>>>>>>> at org.apache.catalina.valves.ErrorReportValve
>>>>>>>>> (ErrorReportValve.java:102)
>>>>>>>>> at org.apache.catalina.core.StandardEngineValve
>>>>>>>>> (StandardEngineValve.java:109)
>>>>>>>>> at org.apache.catalina.connector.CoyoteAdapter
>>>>>>>>> (CoyoteAdapter.java:293)
>>>>>>>>> at org.apache.coyote.http11.Http11Processor
>>>>>>>>> (Http11Processor.java:859)
>>>>>>>>> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler
>>>>>>>>> (Http11Protocol.java:602)
>>>>>>>>> at org.apache.tomcat.util.net.JIoEndpoint$Worker
>>>>>>>>> (JIoEndpoint.java:489)
>>>>>>>>> at java.lang.Thread (Thread.java:724)
>>>>>>>>> Caused by: Exception: java.lang.Throwable Message: ERROR:
schema
>>>>>>>>> "invoice" does not exist Position: 46 Stack trace: at
>>>>>>>>> org.postgresql.core.v3.QueryExecutorImpl (QueryExecutorImpl.java:2102)
>>>>>>>>> at org.postgresql.core.v3.QueryExecutorImpl
>>>>>>>>> (QueryExecutorImpl.java:1835)
>>>>>>>>> at org.postgresql.core.v3.QueryExecutorImpl
>>>>>>>>> (QueryExecutorImpl.java:257)
>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>> (AbstractJdbc2Statement.java:500)
>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>> (AbstractJdbc2Statement.java:374)
>>>>>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement
>>>>>>>>> (AbstractJdbc2Statement.java:254)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor
>>>>>>>>> (GenericJdbcExecutor.java:56)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:155)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:48)
>>>>>>>>> at org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer
>>>>>>>>> (GenericJdbcImportInitializer.java:37)
>>>>>>>>> at org.apache.sqoop.framework.FrameworkManager
>>>>>>>>> (FrameworkManager.java:447)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:112)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:98)
>>>>>>>>> at org.apache.sqoop.handler.SubmissionRequestHandler
>>>>>>>>> (SubmissionRequestHandler.java:68)
>>>>>>>>> at org.apache.sqoop.server.v1.SubmissionServlet
>>>>>>>>> (SubmissionServlet.java:44)
>>>>>>>>> at org.apache.sqoop.server.SqoopProtocolServlet
>>>>>>>>> (SqoopProtocolServlet.java:63)
>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:637)
>>>>>>>>> at javax.servlet.http.HttpServlet (HttpServlet.java:717)
>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>> (ApplicationFilterChain.java:290)
>>>>>>>>> at org.apache.catalina.core.ApplicationFilterChain
>>>>>>>>> (ApplicationFilterChain.java:206)
>>>>>>>>> at org.apache.catalina.core.StandardWrapperValve
>>>>>>>>> (StandardWrapperValve.java:233)
>>>>>>>>> at org.apache.catalina.core.StandardContextValve
>>>>>>>>> (StandardContextValve.java:191)
>>>>>>>>> at org.apache.catalina.core.StandardHostValve
>>>>>>>>> (StandardHostValve.java:127)
>>>>>>>>> at org.apache.catalina.valves.ErrorReportValve
>>>>>>>>> (ErrorReportValve.java:102)
>>>>>>>>> at org.apache.catalina.core.StandardEngineValve
>>>>>>>>> (StandardEngineValve.java:109)
>>>>>>>>> at org.apache.catalina.connector.CoyoteAdapter
>>>>>>>>> (CoyoteAdapter.java:293)
>>>>>>>>> at org.apache.coyote.http11.Http11Processor
>>>>>>>>> (Http11Processor.java:859)
>>>>>>>>> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler
>>>>>>>>> (Http11Protocol.java:602)
>>>>>>>>> at org.apache.tomcat.util.net.JIoEndpoint$Worker
>>>>>>>>> (JIoEndpoint.java:489)
>>>>>>>>> at java.lang.Thread (Thread.java:724)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

Mime
View raw message