hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19821) Distributed HiveServer2
Date Sun, 17 Jun 2018 15:45:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515113#comment-16515113
] 

Sahil Takiar commented on HIVE-19821:
-------------------------------------

Attached a WIP patch. The patch works for {{TestJdbcWithLocalClusterSpark}} and {{TestMultiSessionsHS2WithLocalClusterSpark}}.
The patch has several restrictions, but works for the basic queries I have tested with. There
are a number of follow up enhancements that will be necessary: security, operation log support,
HS2 Web UI integration, etc.

The high level design is to create a new {{IDriver}} called {{RemoteProcessDriver}} that runs
a {{IDriver}} in a remote process. For HoS, the remote process is the Spark driver process
that is launched as part of each Spark application (and by extension, each Hive session).
The patch modifies the client ({{AbstractSparkClient.ClientProtocol}}) and driver ({{RemoteDriver.DriverProtocol}})
RPC protocols to define new messages for supporting this feature. Several other modifications
to get HoS query compilation to work inside the Spark driver were necessary too. The patch
includes javadocs which explains the role of each new class.

Will post a high-level design doc soon.

> Distributed HiveServer2
> -----------------------
>
>                 Key: HIVE-19821
>                 URL: https://issues.apache.org/jira/browse/HIVE-19821
>             Project: Hive
>          Issue Type: New Feature
>          Components: HiveServer2
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>         Attachments: HIVE-19821.1.WIP.patch
>
>
> HS2 deployments often hit OOM issues due to a number of factors: (1) too many concurrent
connections, (2) query that scan a large number of partitions have to pull a lot of metadata
into memory (e.g. a query reading thousands of partitions requires loading thousands of partitions
into memory), (3) very large queries can take up a lot of heap space, especially during query
parsing. There are a number of other factors that cause HiveServer2 to run out of memory,
these are just some of the more commons ones.
> Distributed HS2 proposes to do all query parsing, compilation, planning, and execution
coordination inside a dedicated container. This should significantly decrease memory pressure
on HS2 and allow HS2 to scale to a larger number of concurrent users.
> For HoS (and I think Hive-on-Tez) this just requires moving all query compilation, planning,
etc. inside the application master for the corresponding Hive session.
> The main benefit here is isolation. A poorly written Hive query cannot bring down an
entire HiveServer2 instance and force all other queries to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message