hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nishant Bangarwa (JIRA)" <>
Subject [jira] [Commented] (HIVE-15571) Support Insert into for druid storage handler
Date Thu, 01 Jun 2017 19:10:05 GMT


Nishant Bangarwa commented on HIVE-15571:

[~jcamachorodriguez] handled review comments and updated JIRA desc. 

> Support Insert into for druid storage handler
> ---------------------------------------------
>                 Key: HIVE-15571
>                 URL:
>             Project: Hive
>          Issue Type: New Feature
>          Components: Druid integration
>            Reporter: slim bouguerra
>            Assignee: Nishant Bangarwa
>         Attachments: HIVE-15571.01.patch, HIVE-15571.02.patch
> Add support of inset into operator for druid storage handler.
> Implementation - 
> 1) Using DruidOutputFormat, multiple reducers write segments to a staging directory with
the new versions. 
> 2) When doing insert into, before publishing the segments to the metadata storage, versions
for the segments are updated based on the existing segments in druid metadata storage. 
> 3) After final versions are checked, the segments are moved from the staging directory
to the segment directory. 
> 4) New segment versions are published to the metadata storage. 
> Reading of existing segments from metadata storage and updating the versions of newly
created segments based on them is done in a single transaction. This will make sure that segments
generated by parallel Insert Into statements being executed will not overwrite each other.
In case of conflict in metadata storage transaction insert into will fail.   

This message was sent by Atlassian JIRA

View raw message