sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-1804) Repository Structure + API: Storing/Retrieving the From/To state of the incremental read/ write
Date Wed, 14 Jan 2015 20:15:35 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277592#comment-14277592
] 

Veena Basavaraj edited comment on SQOOP-1804 at 1/14/15 8:15 PM:
-----------------------------------------------------------------

State in an internal variable for connectors ( read only from users perspective), so they
cannot be edited, they of course should be shown in the submission if we decide to show configs
as well, but independent command can also help
{code}
show submission-state --jid 1 --type from
show submission-config--jid 1 --type from

{code}

If we want a input and state associated then the connector ( one of the discussion points
in SQOOP-1168 Question 3)  can say that as well in this config object


{code}
public class DeltaFetchConfig1 {
 
  @Input(size = 255) public String column;
  // validate supported operators, can provide a default value etc...
  @Input public String operator;
// if we edit this, override last_value
  @Input(overrides="last_value) public String value;
// relatedTo signifies to the connector that last_value relates to value
  @State(relatedTo="value") public String last_value; 

}

{code}

{code}


was (Author: vybs):
State in an internal variable for connectors ( read only from users perspective), so they
cannot be edited, they of course should be shown in the submission if we decide to show configs
as well, but independent command can also help
{code}
show submission-config --jid 1 --type from
{code}

If we want a input and state associated then the connector ( one of the discussion points
in SQOOP-1168 Question 3)  can say that as well in this config object


{code}
public class DeltaFetchConfig1 {
 
  @Input(size = 255) public String column;
  // validate supported operators, can provide a default value etc...
  @Input public String operator;
// if we edit this, override last_value
  @Input(overrides="last_value) public String value;
// relatedTo signifies to the connector that last_value relates to value
  @State(relatedTo="value") public String last_value; 

}

{code}

{code}

> Repository Structure + API: Storing/Retrieving the From/To state of the incremental read/
write
> -----------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-1804
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1804
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> Details of this proposal are in the wiki.
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Wheretostoretheoutputinsqoop?
> Update: The above highlights the pros and cons of each approach. 
> #4 is chosen, since it is less intrusive, more clean and allows U/Edit per value in the
output easily.
> Will use this ticket for more detailed discussion on storage options for the output from
connectors
> 1. 
> {code}
> // will have FK to submission
>  public static final String QUERY_CREATE_TABLE_SQ_JOB_OUTPUT_SUBMISSION =
>      "CREATE TABLE " + TABLE_SQ_JOB_OUTPUT + " ("
>      + COLUMN_SQ_JOB_OUT_ID + " BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT
BY 1), "
>      + COLUMN_SQ_JOB_OUT_KEY + " VARCHAR(32), "
>      + COLUMN_SQ_JOB_OUT_VALUE + " LONG VARCHAR,"
>      + COLUMN_SQ_JOB_OUT_TYPE + " VARCHAR(32),"
>      + COLUMN_SQD_ID + " VARCHAR(32)," // FK to the direction table, since this allows
to distinguish output from FROM/ TO part of the job
>    + COLUMN_SQRS_SUBMISSION + " BIGINT, "
>    + "CONSTRAINT " + CONSTRAINT_SQRS_SQS + " "
>      + "FOREIGN KEY (" + COLUMN_SQRS_SUBMISSION + ") "
>        + "REFERENCES " + TABLE_SQ_SUBMISSION + "(" + COLUMN_SQS_ID + ") ON DELETE CASCADE
"
> {code}
> 2.
> At the code level, we will define  MOutputType, one of the types can be BLOB as well,
if a connector decides to store the value as a BLOB
> {code}
> class JobOutput {
> String key;
> Object value;
> MOutputType type;
> }
> {code}
> 3. 
> At the repository API, add a new API to get job output for a particular submission Id
and allow updates on values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message