atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sidharth Mishra <sidharthkmis...@gmail.com>
Subject Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
Date Thu, 07 Nov 2019 23:54:08 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71737/
-----------------------------------------------------------

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.


Bugs: ATLAS-3477
    https://issues.apache.org/jira/browse/ATLAS-3477


Repository: atlas


Description
-------

As part of this solution users can purge the entities which are already soft deleted. This
will be useful as someone may be interested to purge some list of soft deleted entities from
Atlas. Due to governance compliances, only admin can perform purge of entities.

Purge will also create separate Audit entry at HBase similar to delete entity operation.


Diffs
-----

  authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 59c596d66 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 199d6bf58 
  intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 444167eae 
  intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java e9cc7cd3d 
  intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 7ace00dff

  intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java daf4ca88c 
  repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 43a9b842c

  repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
01e339d23 
  repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java b94590bf1

  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
d2544df4f 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
3389d2465 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
17a96493f 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
4d57d8bc8 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c 
  webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java 48f0cd3c2

  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 464d46f1e 
  webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 5d506bbc2



Diff: https://reviews.apache.org/r/71737/diff/1/


Testing
-------

IT has been added. 
Validated manually by running below mentioned queries and REST:


a) create table thursday_07_11_1 (id int);
   create table thursday_07_11_1_ctas as select * from thursday_07_11_1;

   create table thursday_07_11_2 (id int);
   create table thursday_07_11_2_ctas as select * from thursday_07_11_2;
b) The above will create two lineages. Go to Atlas UI and get the vertices for atleast any
two.


c) Using rest delete those two entities or use below query:
drop table thursday_07_11_1;
drop table thursday_07_11_1_ctas;

drop table thursday_07_11_2;
drop table thursday_07_11_2_ctas;

REST delete example:
DELETE http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c

d) Now use the same guids and call purge REST:
DELETE http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05&guid=98bee463-8810-402a-bcdc-3eb00bf50365

Check the response which will return all the successfully purged entities

e) For each guid check if the audit has separate new entry for purge. REST used:
GET http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit

Example Response:

[
    {
        "entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
        "timestamp": 1573161789065,
        "user": "admin",
        "action": "ENTITY_PURGE",
        "details": "Purged entity",
        "eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161789065:1",
        "entity": null,
        "type": null
    },
    {
        "entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
        "timestamp": 1573161729972,
        "user": "impala",
        "action": "ENTITY_DELETE",
        "details": "Deleted entity",
        "eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161729972:1",
        "entity": null,
        "type": null
    },
    {
        "entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
        "timestamp": 1573161727305,
        "user": "admin",
        "action": "ENTITY_UPDATE",
        "details": "Updated: {\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"replicatedTo\":null,\"replicatedFrom\":null,\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"description\":null,\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"status\":\"ACTIVE\",\"createdBy\":\"impala\",\"updatedBy\":\"admin\",\"createTime\":1573161723180,\"updateTime\":1573161727305,\"version\":0,\"labels\":[]}",
        "eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161727305:0",
        "entity": null,
        "type": null
    },
    {
        "entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
        "timestamp": 1573161723180,
        "user": "impala",
        "action": "ENTITY_CREATE",
        "details": "Created: {\"typeName\":\"hive_column\",\"attributes\":{\"owner\":\"admin\",\"qualifiedName\":\"default.thursday_07_11_5.id@cm\",\"name\":\"id\",\"comment\":null,\"position\":0,\"type\":\"int\"},\"guid\":\"7b6806c1-8daf-4e0f-8d10-672a192968f1\",\"isIncomplete\":false,\"provenanceType\":0,\"version\":0,\"relationshipAttributes\":{\"table\":{\"guid\":\"c5ae4503-b498-493c-86b5-d8d909c6b16a\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.thursday_07_11_5@cm\"},\"relationshipType\":\"hive_table_columns\"}},\"proxy\":false}",
        "eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161723180:2",
        "entity": null,
        "type": null
    }
]


Thanks,

Sidharth Mishra


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message