sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Cecho" <jar...@apache.org>
Subject Re: Review Request 13181: SQOOP-773 Sqoop2: Batch execution support for client commands
Date Fri, 11 Oct 2013 20:02:52 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13181/#review26935
-----------------------------------------------------------


Hi Abe,
thank you very much for working on the patch, appreciated! I've tested it in my environment
and I do have couple of notes:

I think that we need to ensure that one failed command will end entire script execution. I'm
thinking about use case that would do the following:

1) create connection
2) create job for -^
3) run the job -^

In such situation we need to ensure that we are not creating job for non-existing connection
or running non-existing job.


shell/src/main/java/org/apache/sqoop/shell/CloneConnectionFunction.java
<https://reviews.apache.org/r/13181/#comment52475>

    Can we make this and similar code a bit more verbose?
    
    I've tried following script:
    
    create connection --cid 1 --name NameX
    
    And I got following output:
    
    [root@bousa-trunk sqoop]# ./bin/sqoop.sh client create.sqoop
    Sqoop home directory: /root/sqoop
    sqoop:000> create connection --cid 1 --name NameX
    Creating connection for connector with id 1
    [root@bousa-trunk sqoop]#
    
    Creating the connection has failed as I did not specified any arguments, but there is
no error message which is really confusing.



shell/src/main/java/org/apache/sqoop/shell/SqoopFunction.java
<https://reviews.apache.org/r/13181/#comment52464>

    It seems that some of the functions are just returning true and not doing any validations.
What about making such function default here and let the children override it only if some
validations will be required?



shell/src/main/java/org/apache/sqoop/shell/utils/FormOptions.java
<https://reviews.apache.org/r/13181/#comment52462>

    Nit: Unused import.


Jarcec

- Jarek Cecho


On Sept. 27, 2013, 6:30 p.m., Abraham Elmahrek wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13181/
> -----------------------------------------------------------
> 
> (Updated Sept. 27, 2013, 6:30 p.m.)
> 
> 
> Review request for Sqoop, Hari Shreedharan and Jarek Cecho.
> 
> 
> Bugs: SQOOP-773
>     https://issues.apache.org/jira/browse/SQOOP-773
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> commit 080ced16578c1d95015ce3e99b4335beb465861a
> Author: Abraham Elmahrek <abraham@elmahrek.com>
> Date:   Tue Aug 13 14:14:06 2013 -0700
> 
>     SQOOP-773 Sqoop2: Batch execution support for client commands
>     
>     Separated options into two groups: fixed and dynamic options.
>     Fixed options (IE: connector ID) come first and are used to select
>     what options should be used in dynamic options. Dynamic options
>     are automatically created based on forms selected from fixed options.
>     The keys for these options take on the form "<prefix>-<form name>-<input-name>".
> 
> :100644 100644 980a908... a7e7e7d... M  shell/src/main/java/org/apache/sqoop/shell/CloneCommand.java
> :100644 100644 856abaa... 2c2869c... M  shell/src/main/java/org/apache/sqoop/shell/CloneConnectionFunction.java
> :100644 100644 3e23025... dd2eb2b... M  shell/src/main/java/org/apache/sqoop/shell/CloneJobFunction.java
> :100644 100644 e62ce08... 9ad007b... M  shell/src/main/java/org/apache/sqoop/shell/CreateCommand.java
> :100644 100644 5fbf0a3... 973fd53... M  shell/src/main/java/org/apache/sqoop/shell/CreateConnectionFunction.java
> :100644 100644 6e4f04b... f0d4a6c... M  shell/src/main/java/org/apache/sqoop/shell/CreateJobFunction.java
> :100644 100644 c123732... d79516d... M  shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
> :100644 100644 d4095b7... df9a2cc... M  shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
> :100644 100644 f119660... b5a54b9... M  shell/src/main/java/org/apache/sqoop/shell/DisableConnectionFunction.java
> :100644 100644 a87e51f... 8cc4ea3... M  shell/src/main/java/org/apache/sqoop/shell/DisableJobFunction.java
> :100644 100644 f782b16... 0c9f184... M  shell/src/main/java/org/apache/sqoop/shell/EnableConnectionFunction.java
> :100644 100644 20c80dc... de5000b... M  shell/src/main/java/org/apache/sqoop/shell/EnableJobFunction.java
> :100644 100644 1c43dce... f0153b8... M  shell/src/main/java/org/apache/sqoop/shell/SetOptionFunction.java
> :100644 100644 abd9cea... 1aacd7c... M  shell/src/main/java/org/apache/sqoop/shell/SetServerFunction.java
> :100644 100644 b55d5d1... f0720a0... M  shell/src/main/java/org/apache/sqoop/shell/ShowConnectionFunction.java
> :100644 100644 97a4ab2... a9aba71... M  shell/src/main/java/org/apache/sqoop/shell/ShowConnectorFunction.java
> :100644 100644 28497db... c71696c... M  shell/src/main/java/org/apache/sqoop/shell/ShowFrameworkFunction.java
> :100644 100644 da4a4ff... 68bd3b3... M  shell/src/main/java/org/apache/sqoop/shell/ShowJobFunction.java
> :100644 100644 5e3c3ff... d0b05e2... M  shell/src/main/java/org/apache/sqoop/shell/ShowOptionFunction.java
> :100644 100644 ec97e63... 11bff86... M  shell/src/main/java/org/apache/sqoop/shell/ShowServerFunction.java
> :100644 100644 a592a98... a571269... M  shell/src/main/java/org/apache/sqoop/shell/ShowSubmissionFunction.java
> :100644 100644 764b754... 5d6f97f... M  shell/src/main/java/org/apache/sqoop/shell/ShowVersionFunction.java
> :100644 100644 675a796... 371c867... M  shell/src/main/java/org/apache/sqoop/shell/SqoopFunction.java
> :100644 100644 f03e08f... 914454f... M  shell/src/main/java/org/apache/sqoop/shell/StartCommand.java
> :100644 100644 02148de... b8771c4... M  shell/src/main/java/org/apache/sqoop/shell/StartJobFunction.java
> :100644 100644 184892a... ebd4548... M  shell/src/main/java/org/apache/sqoop/shell/StatusCommand.java
> :100644 100644 be0de8c... c61e9b9... M  shell/src/main/java/org/apache/sqoop/shell/StatusJobFunction.java
> :100644 100644 698bca7... 65a454b... M  shell/src/main/java/org/apache/sqoop/shell/StopCommand.java
> :100644 100644 6c0e3c2... ae559cc... M  shell/src/main/java/org/apache/sqoop/shell/StopJobFunction.java
> :100644 100644 9262ccd... 24f31ea... M  shell/src/main/java/org/apache/sqoop/shell/UpdateCommand.java
> :100644 100644 c062fe6... 31d9af9... M  shell/src/main/java/org/apache/sqoop/shell/UpdateConnectionFunction.java
> :100644 100644 da1e0c5... 6f83b91... M  shell/src/main/java/org/apache/sqoop/shell/UpdateJobFunction.java
> :000000 100644 0000000... 6b6e858... A  shell/src/main/java/org/apache/sqoop/shell/utils/ConnectionDynamicFormOptions.java
> :000000 100644 0000000... cc63610... A  shell/src/main/java/org/apache/sqoop/shell/utils/DynamicFormOptions.java
> :100644 100644 9bc0b93... bf0615e... M  shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
> :000000 100644 0000000... 516790a... A  shell/src/main/java/org/apache/sqoop/shell/utils/FormOptions.java
> :000000 100644 0000000... aa118e1... A  shell/src/main/java/org/apache/sqoop/shell/utils/JobDynamicFormOptions.java
> 
> 
> Diffs
> -----
> 
>   shell/src/main/java/org/apache/sqoop/shell/CloneCommand.java 980a9086f433fd3a2228122b6a253a9a1c562315

>   shell/src/main/java/org/apache/sqoop/shell/CloneConnectionFunction.java 856abaae5a7745b38b1db42cd9283732d1bc0b64

>   shell/src/main/java/org/apache/sqoop/shell/CloneJobFunction.java 3e2302514d9350affbc9ab199fb4e91355201290

>   shell/src/main/java/org/apache/sqoop/shell/CreateCommand.java e62ce08ca1ee9168a77b619d21db93b04809b7ec

>   shell/src/main/java/org/apache/sqoop/shell/CreateConnectionFunction.java 5fbf0a3d8ec13c8e3628525b7d6432d56ddbb0c3

>   shell/src/main/java/org/apache/sqoop/shell/CreateJobFunction.java 6e4f04bf392f0c062e211fd6709f362ec6e584c3

>   shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java c1237320b3ceec48569c34f4b2a3fa78b4d2c26e

>   shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java d4095b736d3dcc97487c20cb71c7c460139c2afd

>   shell/src/main/java/org/apache/sqoop/shell/DisableConnectionFunction.java f1196601857b90442b1c9394902b71d3bd140915

>   shell/src/main/java/org/apache/sqoop/shell/DisableJobFunction.java a87e51f32226ad6e65b8970d159861e013c6fe9d

>   shell/src/main/java/org/apache/sqoop/shell/EnableConnectionFunction.java f782b1654203c450b4d22b762dc0f406b560aa29

>   shell/src/main/java/org/apache/sqoop/shell/EnableJobFunction.java 20c80dca160ddbfa9edb6c5ea5c19fdc3d0bb7c9

>   shell/src/main/java/org/apache/sqoop/shell/SetOptionFunction.java 1c43dceea2581cea70d0324a63292a0909a402e1

>   shell/src/main/java/org/apache/sqoop/shell/SetServerFunction.java abd9ceac5a8338dcb1273b0f4732b6bd79962d65

>   shell/src/main/java/org/apache/sqoop/shell/ShowConnectionFunction.java b55d5d1126b1bd33f4517173257188504557b3f2

>   shell/src/main/java/org/apache/sqoop/shell/ShowConnectorFunction.java 97a4ab2a2377a2aff564f8ffc18884aa11d8a5e3

>   shell/src/main/java/org/apache/sqoop/shell/ShowFrameworkFunction.java 28497db27295c4ca89ef4b56986818803296cb10

>   shell/src/main/java/org/apache/sqoop/shell/ShowJobFunction.java da4a4ff324c32deceadd45d336646eb5bc5a7882

>   shell/src/main/java/org/apache/sqoop/shell/ShowOptionFunction.java 5e3c3ff9469ee75d6b9d7635de66b033e93c37cd

>   shell/src/main/java/org/apache/sqoop/shell/ShowServerFunction.java ec97e637937129ff0e2151612f98eceb284ec126

>   shell/src/main/java/org/apache/sqoop/shell/ShowSubmissionFunction.java a592a9861f01d333ef72f1f96bdf038ab334ea9e

>   shell/src/main/java/org/apache/sqoop/shell/ShowVersionFunction.java 764b754180500a86501450c90b119813de6cf1f7

>   shell/src/main/java/org/apache/sqoop/shell/SqoopFunction.java 675a796d7919a3afadea2e14a88f7823d5dced47

>   shell/src/main/java/org/apache/sqoop/shell/StartCommand.java f03e08fe2178c2b044eef1bfbfa82c9235d8914d

>   shell/src/main/java/org/apache/sqoop/shell/StartJobFunction.java 02148de497c15caa363c17a54202690a9026a68b

>   shell/src/main/java/org/apache/sqoop/shell/StatusCommand.java 184892a33b1b26391c7a7eaa8c79cd27f0178c39

>   shell/src/main/java/org/apache/sqoop/shell/StatusJobFunction.java be0de8c29289a904cfc00361da7fbf9a6c8e7d1c

>   shell/src/main/java/org/apache/sqoop/shell/StopCommand.java 698bca778d3aa09fa24372a6367986391f9e61d9

>   shell/src/main/java/org/apache/sqoop/shell/StopJobFunction.java 6c0e3c284b997561fc351a57e35ce2d6c77f6121

>   shell/src/main/java/org/apache/sqoop/shell/UpdateCommand.java 9262ccd085730c7101179157db9828ac709c267d

>   shell/src/main/java/org/apache/sqoop/shell/UpdateConnectionFunction.java c062fe6781aebac15c4d59b5690772695d8a10a9

>   shell/src/main/java/org/apache/sqoop/shell/UpdateJobFunction.java da1e0c570fb43cea439549c0ab5268f67d55996d

>   shell/src/main/java/org/apache/sqoop/shell/core/ShellError.java e5a99f1cdef7afc4df12e7eed36e1096e2ccd3ea

>   shell/src/main/java/org/apache/sqoop/shell/utils/ConnectionDynamicFormOptions.java
PRE-CREATION 
>   shell/src/main/java/org/apache/sqoop/shell/utils/DynamicFormOptions.java PRE-CREATION

>   shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java 9bc0b93649cb2ae6c84228b52c04b60a36c13b18

>   shell/src/main/java/org/apache/sqoop/shell/utils/FormOptions.java PRE-CREATION 
>   shell/src/main/java/org/apache/sqoop/shell/utils/JobDynamicFormOptions.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/13181/diff/
> 
> 
> Testing
> -------
> 
> Executed the following commands in a file:
> 
> create connection --cid 1 --name mysql-test --connector-connection-jdbcDriver com.mysql.jdbc.Driver
--connector-connection-connectionString jdbc:mysql://hue.ent.cloudera.com/test --connector-connection-username
root --connector-connection-password root
> show connection
> create job --xid 1 --type import --name mysql-import-job1 --connector-table-tableName
test --connector-table-partitionColumn a --framework-output-storageType 0 --framework-output-outputFormat
0 --framework-output-outputDirectory /tmp/output
> show job
> start job --jid 1
> 
> 
> Thanks,
> 
> Abraham Elmahrek
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message