atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Mestry (Jira)" <j...@apache.org>
Subject [jira] [Created] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation
Date Wed, 13 May 2020 18:04:00 GMT
Ashutosh Mestry created ATLAS-3797:
--------------------------------------

             Summary: Entity Creation: Refactoring: Improve Edge Creation
                 Key: ATLAS-3797
                 URL: https://issues.apache.org/jira/browse/ATLAS-3797
             Project: Atlas
          Issue Type: Improvement
          Components:  atlas-core
    Affects Versions: trunk
            Reporter: Ashutosh Mestry
            Assignee: Ashutosh Mestry
             Fix For: trunk


*Background*

[ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge creation improvement
by improving on edge existence check.

The same change could be applied to another area within the implementation.

*Investigation*

This was discovered as part of volume testing done for high-volume import. This included importing
over 100M entities. Various parts of the entity creation process had probes added to check
for performance.

During this investigation the edge creation was doing the same existence check that was addressed
in earlier fixes. Re-using the same implementation showed marked improvement in entity creation.

These were the parameters used for during import.

Node configuration: 128 GB RAM, 40 cores.
{code:java}
atlas.migration.mode.batch.size=1500
atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message