sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-2113) Modify REST API for the RU operation on job and link config
Date Tue, 24 Mar 2015 18:55:53 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378285#comment-14378285
] 

Veena Basavaraj edited comment on SQOOP-2113 at 3/24/15 6:55 PM:
-----------------------------------------------------------------

[~jarcec]
Not sure what point is missing, but here is more context.

If you get a chance to look at the updateConfig APIS that were added as part of 1516, it might
give you more context.

The update*Config api, take the link/job Id and the inputId and update a single row in the
LINK_INPUT or JOB_INPUT table, A config update does not need to update the JOB. it can update
just the config, similarly and INPUT update such as "lastValue" need not update the entire
job object.

Updating is usually done as a "UPDATE" statement in the SQL and this is what the code does.
So for this we assume that the record exists. The current code doe that creates LINKS and
JOBS insert a record into the LINK_INPUT and JOB_INPUT if the value is null ? Does this make
sense .

Now if the design is to make a change to update config to "INSERT" if value does not exist,
I can change that, I would have hoped that the review of SQOOP-1516 would have caught this
aspect.

But it is not what normally is done, and the optimization achieved by not storing values that
are null seems premature, considering how cheap disk is and also the number of such objects
created over time in a database such as a sqoop installation is never oing to be that huge.

Also I have highlight, the complexity in the code, to handle another scenario

If a update happens to convert a non-null to null, does this mean, the record needs to be
deleted?
All of this seems quite unnecessary if we can just remove the 2 lines of code in that skips
these values.

It might have been ok to have this in place when updating of config inputs was not supported
explicitly from the "user" or REST API/ shell. 
Having said that if we want to preserve this complexity I am happy to change the update*Config
API code to handle both the use cases.

{code}
private void createInputValues(String query, long id, List<MConfig> configs, Connection
conn)
      throws SQLException {
    PreparedStatement stmt = null;
    int result;

    try {
      stmt = conn.prepareStatement(query);

      for (MConfig config : configs) {
        for (MInput<?> input : config.getInputs()) {
          // Skip empty values as we're not interested in storing those in db
          if (input.isEmpty()) {
            continue;
          }
          stmt.setLong(1, id);
          stmt.setLong(2, input.getPersistenceId());
          stmt.setString(3, input.getUrlSafeValueString());

          result = stmt.executeUpdate();
          if (result != 1) {
            throw new SqoopException(CommonRepositoryError.COMMON_0017, Integer.toString(result));
          
{code}






was (Author: vybs):
[~jarcec]
Not sure what point is missing, but here is more context.

If you get a chance to look at the updateConfig APIS that were added as part of 1516, it might
give you more context.

The update*Config api, take the link/job Id and the inputId and update a single row in the
LINK_INPUT or JOB_INPUT table, A config update does not need to update the JOB. it can update
just the config, similarly and INPUT update such as "lastValue" need not update the entire
job object.

Updating is usually done as a "UPDATE" statement in the SQL and this is what the code does.
So for this we assume that the record exists. The current code doe that creates LINKS and
JOBS insert a record into the LINK_INPUT and JOB_INPUT if the value is null ? Does this make
sense .

Now if the design is to make a change to update config to "INSERT" if value does not exist,
I can change that, I would have hoped that the review of SQOOP-1516 would have caught this
aspect.

But it is not what normally is done, and the optimization achieved by not storing values that
are null seems premature, considering how cheap disk is and also the number of such objects
created over time in a database such as a sqoop installation is never oing to be that huge.





> Modify REST API for the RU operation on job and link config
> -----------------------------------------------------------
>
>                 Key: SQOOP-2113
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2113
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.6
>
>         Attachments: SQOOP-2113-1.patch, SQOOP-2113-1.patch, SQOOP-2113-2.patch, SQOOP-2113-3.patch
>
>
> based on the design doc for 1516, support the RU operation on configs for rest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message