hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-13232) Aggressively drop compression buffers in ORC OutStreams
Date Sun, 13 Mar 2016 00:04:33 GMT


Hive QA commented on HIVE-13232:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9774 tests executed
*Failed tests:*
TestMiniTezCliDriver-cte_4.q-orc_merge5.q-vectorization_limit.q-and-12-more - did not produce
a TEST-*.xml file
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a
TEST-*.xml file
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed

This message is automatically generated.

ATTACHMENT ID: 12792965 - PreCommit-HIVE-TRUNK-Build

> Aggressively drop compression buffers in ORC OutStreams
> -------------------------------------------------------
>                 Key: HIVE-13232
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.1, 1.3.0, 2.1.0
>         Attachments: HIVE-13232.patch, HIVE-13232.patch
> In Hive 0.11, when ORC's OutStream's were flushed they dropped all of the their buffers.
In the patch for HIVE-4324, we inadvertently changed that behavior so that one of the buffers
is held on to. For queries with a lot of writers and thus under significant memory pressure
this can have a significant impact on the memory usage. 
> Note that "hive.optimize.sort.dynamic.partition" avoids this problem by sorting on the
dynamic partition key and thus only a single ORC writer is open at once. This will use memory
more effectively and avoid creating ORC files with very small stripes, which will produce
better downstream performance.

This message was sent by Atlassian JIRA

View raw message