tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-385) Refactoring TaskScheduler to assign multiple fragments
Date Thu, 02 Jan 2014 04:42:50 GMT

    [ https://issues.apache.org/jira/browse/TAJO-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860014#comment-13860014
] 

Hudson commented on TAJO-385:
-----------------------------

FAILURE: Integrated in Tajo-trunk-postcommit #660 (See [https://builds.apache.org/job/Tajo-trunk-postcommit/660/])
TAJO-385: Refactoring TaskScheduler to assign multiple fragments. (jihoon) (jihoonson: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=df5727c49edd2e726b99dfa49dcc6b3d5b6f252b)
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultFragmentScheduleAlgorithm.java
* tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/MergeScanner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryUnitAttemptScheduleEvent.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskSchedulerEventFactory.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/FragmentScheduleAlgorithmFactory.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/FragmentScheduleAlgorithm.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskSchedulerEvent.java
* tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/FetchScheduleEvent.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/DefaultTaskSchedulerEvent.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/ScheduledFetches.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GreedyFragmentScheduleAlgorithm.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/FragmentScheduleEvent.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
* tajo-core/tajo-core-backend/src/main/resources/tajo-default.xml
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TaskSchedulerContext.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/query/QueryUnitRequestImpl.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/FragmentPair.java
* CHANGES.txt
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TaskSchedulerFactory.java


> Refactoring TaskScheduler to assign multiple fragments
> ------------------------------------------------------
>
>                 Key: TAJO-385
>                 URL: https://issues.apache.org/jira/browse/TAJO-385
>             Project: Tajo
>          Issue Type: Improvement
>          Components: query master
>    Affects Versions: 0.8-incubating
>            Reporter: Jihoon Son
>            Assignee: Jihoon Son
>         Attachments: TAJO-385.patch, TAJO-385_2.patch, TAJO-385_3.patch, TAJO-385_4.patch,
TAJO-385_5.patch
>
>
> In the current implementation, each task processes only one fragment.
> However, processing multiple fragments in a task will increase the query processing performance
according to the storage layout and the user queries.
> In this issue, TaskScheduler is refactored to enable assigning multiple fragments to
each task.
> Followings should be contained.
> * Schedule Fragments instead of QueryUnits in TaskScheduler
> ** The QueryUnit creation is postponed until TaskScheduler receives task requests from
workers.
> ** When TaskScheduler receives task requests from workers, it dynamically creates an
QueryUnit and assigns one or more fragments.
> ** The fragment scheduling should take into account the disk load balancing.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message