sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1496) Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
Date Fri, 10 Oct 2014 03:56:48 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166337#comment-14166337
] 

Hudson commented on SQOOP-1496:
-------------------------------

FAILURE: Integrated in Sqoop2-hadoop100 #613 (See [https://builds.apache.org/job/Sqoop2-hadoop100/613/])
SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs (abraham: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3d539dd4d7477324dfe62a4e57f684351769b000)
* connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestCSVIntermediateDataFormat.java
* connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
* common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
* connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormatTest.java
* connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java
* shell/src/main/java/org/apache/sqoop/shell/utils/SubmissionDisplayer.java
* core/src/main/java/org/apache/sqoop/framework/JobRequest.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java
* core/src/test/java/org/apache/sqoop/framework/TestFrameworkValidator.java
* core/src/main/java/org/apache/sqoop/framework/ExecutionEngine.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
* core/src/test/java/org/apache/sqoop/framework/TestSubmissionRequest.java
* submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
* common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
* core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
* spi/src/main/java/org/apache/sqoop/job/etl/Transferable.java
* execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRSubmissionRequest.java
* spi/src/main/java/org/apache/sqoop/job/etl/From.java
* core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
* core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
* core/src/test/java/org/apache/sqoop/framework/TestJobRequest.java
* execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
* core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java
* spi/src/main/java/org/apache/sqoop/validation/Validator.java
* spi/src/main/java/org/apache/sqoop/job/etl/CallbackBase.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java
* core/src/main/java/org/apache/sqoop/framework/JobManager.java
* core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
* common/src/main/java/org/apache/sqoop/model/MSubmission.java
* execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRJobRequest.java
* spi/src/main/java/org/apache/sqoop/job/etl/To.java
* connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestToInitializer.java


> Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
> ------------------------------------------------------------------
>
>                 Key: SQOOP-1496
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1496
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: from/to
>
>         Attachments: SQOOP-1496.patch, Screen Shot 2014-09-08 at 2.35.06 PM.png
>
>
> Here are the things this ticket should address
> 1. Refactor the submit() in JobManager ( it is way too long for any good unit testing
to happen)
> 2. Revisit the Execution Engine and Submission Engine relationship in the code.
> ExecutionEngine api should probably not create a submissionRequest. It should be the
inverse. The Submission Engine is an abstraction of a JobTracker/ YARN/ Mesos/ OOzie. It will
create a job request for the execution engine while it submits the job into the execution
engine
> a) Suggest renaming SubmissionEngine to a JobSubmissionManager.
>  
> b) SubmissionRequest is a JobRunRequest. See the java doc on the current code SubmissionRequest,
it should already hint us that this object holds the job context to aid the execution engine
> c) Change the apis in Execution Engine/ SubmissionManager to reflect the same, i.e  rename
prepareSubmissionRequest to prepareJobRequest 
> d) The Submission term is overloaded in the code base. In one case it is used to represent
the JobRuns( history of the jobs) and in another place it is the submission engine such as
JobTracker/YARN/ OOzie. The act of submitting creates a submission record. So submission is
an artifact of completing//failing a submit request. Prior to that it is still a job request.
> YARN is an example of a submission engine and it can do much more than submissions, so
at some point we should revisit and outline what the responsibilities of the submission engine
API should be,
> e) define other responsibilities of the submission engine. For instance, if we are to
add more monitoring/ tracking into how the job execution is progressing, we should be able
to add those apis to this entity. There is more room for improvement for this api
> 3. Lastly add unit test for the  JobManager!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message