atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation
Date Mon, 18 May 2020 04:40:00 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109837#comment-17109837
] 

ASF subversion and git services commented on ATLAS-3797:
--------------------------------------------------------

Commit 0571fa42897e33a29baf4d1bd10ee04ae222dc70 in atlas's branch refs/heads/master from Ashutosh
Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=0571fa4 ]

ATLAS-3797: Refactoring: Improve Edge Creation.


> Entity Creation: Refactoring: Improve Edge Creation
> ---------------------------------------------------
>
>                 Key: ATLAS-3797
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3797
>             Project: Atlas
>          Issue Type: Improvement
>          Components:  atlas-core
>    Affects Versions: trunk
>            Reporter: Ashutosh Mestry
>            Assignee: Ashutosh Mestry
>            Priority: Major
>             Fix For: trunk
>
>         Attachments: ATLAS-3797-Refactoring-Improve-Edge-Creation.patch
>
>
> *Background*
> [ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge creation
improvement by improving on edge existence check.
> The same change could be applied to another area within the implementation.
> *Investigation*
> This was discovered as part of volume testing done for high-volume import. This included
importing over 100M entities. Various parts of the entity creation process had probes added
to check for performance.
> During this investigation the edge creation was doing the same existence check that was
addressed in earlier fixes. Re-using the same implementation showed marked improvement in
entity creation.
> These were the parameters used for during import.
> Node configuration: 128 GB RAM, 40 cores.
> {code:java}
> atlas.migration.mode.batch.size=1500
> atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message