hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vamsi Subhash Achanta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
Date Wed, 22 Aug 2018 08:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588536#comment-16588536
] 

Vamsi Subhash Achanta commented on HIVE-6147:
---------------------------------------------

[~swarnim]

Hi,

Is there any work currently going on to make the serialization work at a column level? If
I have 2 columns with different avro schema for each column, the current code is not working.
Ex:

{{CREATE EXTERNAL TABLE txn_store.transactions_single3_1}}
{{ROW FORMAT SERDE "org.apache.hadoop.hive.hbase.HBaseSerDe"}}
{{STORED BY "org.apache.hadoop.hive.hbase.HBaseStorageHandler"}}
{{WITH SERDEPROPERTIES (}}
{{"hbase.columns.mapping" = ":key,nsp:scr_p_1,nsp:scr_m_1,nsp:scr_a_1",}}{{"nsp.scr_p_1.serialization.type"
= "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.PaymentNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_p_1.avro.schema.url" = "hdfs://namenode:8020/user/admin/schemas/payment_namespace.avsc",}}{{"nsp.scr_m_1.serialization.type"
= "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.MerchantNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_m_1.avro.schema.url" = "hdfs://namenode.nm1:8020/user/admin/schemas/merchant_namespace.avsc",}}{{"nsp.scr_a_1.serialization.type"
= "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.AccountingNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_a_1.avro.schema.url" = "hdfs://namenode:8020/user/admin/schemas/accounting_namespace.avsc"}}
{{)}}
{{TBLPROPERTIES (}}
{{"hbase.table.name"="txn_store:transactions_single3_cf",}}
{{"hbase.table.default.storage.type"="binary",}}
{{"hbase.mapred.output.outputtable"="txn_store:transactions_single3_cf",}}
{{"hbase.struct.autogenerate"="true");}}

When executing the select query, it fails with below exception:
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
Error evaluating nsp_scra1
 

> Support avro data stored in HBase columns
> -----------------------------------------
>
>                 Key: HIVE-6147
>                 URL: https://issues.apache.org/jira/browse/HIVE-6147
>             Project: Hive
>          Issue Type: Improvement
>          Components: HBase Handler
>    Affects Versions: 0.12.0, 0.13.0
>            Reporter: Swarnim Kulkarni
>            Assignee: Swarnim Kulkarni
>            Priority: Major
>             Fix For: 0.14.0
>
>         Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt,
HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt
>
>
> Presently, the HBase Hive integration supports querying only primitive data types in
columns. It would be nice to be able to store and query Avro objects in HBase columns by making
them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase
data which can be deeply structured.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message