hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Updated] (HIVE-18153) refactor reopen in TezTask or WM
Date Tue, 05 Dec 2017 03:26:00 GMT


Sergey Shelukhin updated HIVE-18153:
    Attachment: HIVE-18153.patch

This patch basically cleans up and rewrites file management for TezSessionState to be less
It is not equivalent to the old code because the logic of the old code seems to have evolved
over time into something weird. 
reopen changes themselves are relatively small.

> refactor reopen in TezTask or WM
> --------------------------------
>                 Key: HIVE-18153
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-18153.patch
> TezTask reopen relies on getting the same session object in terms of setup; WM reopen
returns a new session from the pool. 
> The former has the advantage of not having to reupload files and stuff... but the object
reuse results in a lot of ugly code, and also reopen might be slower on average with the session
pool than just getting a session from the pool. Either WM needs to do the object-preserving
reopen, or TezTask needs to be refactored. It looks like DAG would have to be rebuilt to do
the latter because of some paths tied to a directory of the old session. Let me see if I can
get around that; if not we can do the former; and then if the former results in too much ugly
code in WM to account for object reuse for different Tez client I'd do the latter anyway since
it's a failure path :)

This message was sent by Atlassian JIRA

View raw message