giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-508) Increase the limit on the number of partitions
Date Fri, 08 Feb 2013 04:31:13 GMT


Hudson commented on GIRAPH-508:

Integrated in Giraph-trunk-Commit #684 (See [])
    GIRAPH-508: Increase the limit on the number of partitions (majakabiljo) (Revision 8c86fa656d1dafeaba3fe55ac184ff3cbaa3e324)

     Result = SUCCESS
majakabiljo :
Files : 
* giraph-core/src/main/java/org/apache/giraph/partition/
* giraph-core/src/main/java/org/apache/giraph/graph/
* giraph-core/src/main/java/org/apache/giraph/partition/
* giraph-core/src/main/java/org/apache/giraph/partition/
* giraph-core/src/main/java/org/apache/giraph/partition/
* giraph-core/src/main/java/org/apache/giraph/conf/

> Increase the limit on the number of partitions
> ----------------------------------------------
>                 Key: GIRAPH-508
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Maja Kabiljo
>            Assignee: Maja Kabiljo
>         Attachments: GIRAPH-508.diff
> We have the limit for total number of partitions of 2995. This is because of Zookeeper
znode limit of 1MB, and from the assumption that partition owner description can take 300
> In the simplest case, when checkpointing is not used and partitions don't move around,
we have 5 ints and hostname written per partition. If partitions move around we have one more
hostname and 2 ints. And when checkpointing is used we also have the path to checkpoint file
> For now, we can get rid of whole WorkerInfo description per partition, and just use taskIds,
since all WorkerInfos are written in the beginning. This will lead to having just 4 ints per
partition in the case when checkpointing is not used, and allow us to have much more partitions.
> When checkpointing is used, we can keep the limit (still up it a bit), or have all workers
read partition metadata when restarting from checkpoint.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message