falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikhil Mulley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-25) Resource Manager
Date Wed, 05 Jun 2013 12:45:21 GMT

    [ https://issues.apache.org/jira/browse/FALCON-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675875#comment-13675875

Nikhil Mulley commented on FALCON-25:

Hi Suhas,

Nice initiative. 

Just a couple of questions:
a) is the amount of bandwidth as the only resource that will be controlled via tokens or even
the number of maps in execution?
b) when we say effective use of bandwidth available, are the tokens issued only after the
completion of the existing jobs or also while they are in progress and perhaps relinquishing
some of the bandwidth already in the midst(or at end) of their transferrs.
 lets say, after acquiring the token, a data transferr (distcp) job has run for a while and
only couple of maps are in execution meaning to say only small amount of data transferr needs
to be complete, which means job may no longer requires the total bandwidth that has been allocated
to it during its initial run, would be left unused while the other transferr jobs are still
waiting for the token to be issued.
c) how will you know the bandwidth totally available for the transferrs between different
cluster(s) or data center(s), will this be a static property or dynamically detected from
the network management resource(s)?
d) while distcp (now) has got options for throttling bandwidth at MB/KB level after recent
changes from Srikanth, would this be inline with resource(bandwidth) management?


> Resource Manager
> ----------------
>                 Key: FALCON-25
>                 URL: https://issues.apache.org/jira/browse/FALCON-25
>             Project: Falcon
>          Issue Type: Improvement
>          Components: replication
>            Reporter: Suhas Vasu
>            Priority: Minor
> We need a way where we can control replication so that we can make effective use of the
bandwidth available. We can introduce a resource manager like zoo-keeper and have a counter/semaphore
through which we can allocate resource to a replication job. The job acquires maps only if
it is allocated resource from the resource manager. 
> This way we can ensure good transfer rates as well as avoid jobs starving for bandwidth
& efficient use of mappers.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message