atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Umesh Padashetty (Jira)" <j...@apache.org>
Subject [jira] [Created] (ATLAS-4014) changeMarker in Atlas Export API response does not get updated
Date Wed, 04 Nov 2020 05:42:00 GMT
Umesh Padashetty created ATLAS-4014:
---------------------------------------

             Summary: changeMarker in Atlas Export API response does not get updated
                 Key: ATLAS-4014
                 URL: https://issues.apache.org/jira/browse/ATLAS-4014
             Project: Atlas
          Issue Type: Bug
          Components:  atlas-core
    Affects Versions: 2.1.0
            Reporter: Umesh Padashetty


As per [https://atlas.apache.org/index.html#/IncrementalExport,] changeMarker is used in
conjunction with "fetchType": "incremental" to incrementally export the metadata, instead
of exporting everything, thereby reducing the redundancy.

As per this logic, the Atlas Export API returns a new change changeMarker after every successful
Export API call (provided there is a change in Atlas metadata).

I ran the Atlas export on a DB. And then added few tables into the DB and then again ran the
Export. Expectation was changeMarker would be updated. But it is being observed that the changeMarker
never changes and is currently always set to 1604003211134 in the instance I am testing. 

The same changeMarker is set in the atlas-export-info.json too. 

The changeMarker is used by Hive Replication Manager for incremental Export. Since atlas
is always returning the same changeMarker, Hive is always passing back the same changeMarker
to atlas in the eventual export calls. This is leading to full export, instead of incremental
export as desired. 

Following is the result of grep i did in atlas server logs. As you can see, changeMarker returned
is always 1604003211134.
{code:java}
[root@quasar-ggealw-4 atlas]# grep "<== export" application.log | grep changeMarker 2020-10-30
05:48:54,451 INFO - [etp1261153343-492 - bd3b60a4-a2b3-4f5c-96c5-166b09157828:] ~ <== export(user=hrt_qa,
from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134 (ExportService:101) 2020-10-30
06:03:43,282 INFO - [etp1261153343-483 - 3114c858-8f89-4d8f-917c-2c7a3d695d2f:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:15:43,880 INFO - [etp1261153343-489 - ccb8da65-e433-44a4-a470-898843dccb17:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:23:10,810 INFO - [etp1261153343-51 - a702086a-984b-43d1-adc8-f9dd0d7950c7:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:26:20,227 INFO - [etp1261153343-493 - 1423275f-b645-42bd-9095-a135707f34ba:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:41:47,192 INFO - [etp1261153343-41 - 2bd5e05f-534f-4a67-b7ef-d8ccc70966ea:]
~ <== export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 06:43:39,920 INFO - [etp1261153343-27 - 812b6c59-24b4-4a4a-8737-008865af5384:]
~ <== export(user=hrt_qa, from=10.101.80.202): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 07:16:19,144 INFO - [etp1261153343-554 - e4678848-12b3-4bae-b339-b3766ec5bb9a:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 07:17:55,540 INFO - [etp1261153343-52 - 95551e0b-98f7-4366-8806-0452d7a49c4c:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 09:49:22,376 INFO - [etp1261153343-740 - f94e92cc-4d80-4d61-b19b-c6956b30b987:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 10:11:31,160 INFO - [etp1261153343-52 - 5e8152c8-d210-4688-914a-06f983fb3898:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 10:24:53,328 INFO - [etp1261153343-547 - 2cdc6326-4786-4a69-a116-f15c476dc6a0:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101) 2020-10-30 11:20:49,768 INFO - [etp1261153343-27 - 1c88be93-17e8-4530-a46d-8c28f8bcc4b7:hive:POST/api/atlas/admin/export]
~ <== export(user=hive, from=172.27.160.133): status SUCCESS: changeMarker: 1604003211134
(ExportService:101)  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message