hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20720) Add partition column option to JDBC handler
Date Thu, 18 Oct 2018 23:06:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656006#comment-16656006
] 

Hive QA commented on HIVE-20720:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944603/HIVE-20720.10.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15106 tests executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=195)
	[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14553/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14553/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14553/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12944603 - PreCommit-HIVE-Build

> Add partition column option to JDBC handler
> -------------------------------------------
>
>                 Key: HIVE-20720
>                 URL: https://issues.apache.org/jira/browse/HIVE-20720
>             Project: Hive
>          Issue Type: New Feature
>          Components: StorageHandler
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>            Priority: Major
>         Attachments: HIVE-20720.1.patch, HIVE-20720.10.patch, HIVE-20720.2.patch, HIVE-20720.3.patch,
HIVE-20720.4.patch, HIVE-20720.5.patch, HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch,
HIVE-20720.9.patch
>
>
> Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits
can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param
if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split.
> We need to rely on this new feature if we want to support multi-splits. Here is my proposal:
> 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties
if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler
will run max/min query to get this in planner. We can currently limit partitionColumn to only
numeric/date/timestamp column for simplicity
> 2. If partitionColumn/numPartitions are not specified, don't split input
> 3. Splits are equal intervals without respect to data distribution
> 4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or
automatically by calcite)
> 5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset
logic (however, don't rely on numSplit).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message