hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Gergely (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-21897) Setting serde / serde properties for partitions
Date Wed, 19 Jun 2019 23:41:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868104#comment-16868104
] 

Miklos Gergely edited comment on HIVE-21897 at 6/19/19 11:40 PM:
-----------------------------------------------------------------

[~mithun] after executing those commands:

SHOW EXTENDED foobar;

 
{code:java}
+-----------------------------+----------------------------------------------------+----------+

|          col_name           |                     data_type     
                | comment  |

+-----------------------------+----------------------------------------------------+----------+

| foo                         | string                          
                  |          |

| bar                         | string                          
                  |          |

| dt                          | string                          
                  |          |

|                             | NULL                          
                    | NULL     |

| # Partition Information     | NULL                                  
            | NULL     |

| # col_name                  | data_type                           
              | comment  |

| dt                          | string                          
                  |          |

|                             | NULL                          
                    | NULL     |

| Detailed Table Information  | Table(tableName:foobar, dbName:default, owner:hive, createTime:1560986681,
lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:foo, type:string,
comment:null), FieldSchema(name:bar, type:string, comment:null), FieldSchema(name:dt, type:string,
comment:null)], location:hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar,
inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde,
parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[],
skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:dt,
type:string, comment:null)], parameters:{last_modified_time=1560986885, totalSize=0, numRows=0,
rawDataSize=0, transactional_properties=insert_only, COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\"},
numFiles=0, numPartitions=2, transient_lastDdlTime=1560986885, bucketing_version=2, last_modified_by=hive,
transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
rewriteEnabled:false, catName:hive, ownerType:USER, writeId:0) |          |
{code}
SHOW CREATE TABLE foobar;

 
{code:java}
+----------------------------------------------------+

|                   createtab_stmt                   |

+----------------------------------------------------+

| CREATE TABLE `foobar`(                             |

|   `foo` string,                                    |

|   `bar` string)                                    |

| PARTITIONED BY (                                   |

|   `dt` string)                                     |

| ROW FORMAT SERDE                                   |

|   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'      |

| STORED AS INPUTFORMAT                              |

|   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'  |

| OUTPUTFORMAT                                       |

|   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |

| LOCATION                                           |

|   'hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar'
|

| TBLPROPERTIES (                                    |

|   'bucketing_version'='2',                         |

|   'last_modified_by'='hive',                       |

|   'last_modified_time'='1560986885',               |

|   'transactional'='true',                          |

|   'transactional_properties'='insert_only',        |

|   'transient_lastDdlTime'='1560986885')            |

+----------------------------------------------------+
{code}
So as it seems the table has only one SerDe, not per partition. Do we want to allow a different
SerDe per partition? Because if we do, it needs planning, and code changes. Or for now we
may stick to the one SerDe / table.


was (Author: mgergely):
[~mithun] after executing those commands:

SHOW EXTENDED foobar;

 
{code:java}
+-----------------------------+----------------------------------------------------+----------+

|          col_name           |                     data_type     
                | comment  |

+-----------------------------+----------------------------------------------------+----------+

| foo                         | string                          
                  |          |

| bar                         | string                          
                  |          |

| dt                          | string                          
                  |          |

|                             | NULL                          
                    | NULL     |

| # Partition Information     | NULL                                  
            | NULL     |

| # col_name                  | data_type                           
              | comment  |

| dt                          | string                          
                  |          |

|                             | NULL                          
                    | NULL     |

| Detailed Table Information  | Table(tableName:foobar, dbName:default, owner:hive, createTime:1560986681,
lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:foo, type:string,
comment:null), FieldSchema(name:bar, type:string, comment:null), FieldSchema(name:dt, type:string,
comment:null)], location:hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar,
inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde,
parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[],
skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:dt,
type:string, comment:null)], parameters:{last_modified_time=1560986885, totalSize=0, numRows=0,
rawDataSize=0, transactional_properties=insert_only, COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\"},
numFiles=0, numPartitions=2, transient_lastDdlTime=1560986885, bucketing_version=2, last_modified_by=hive,
transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
rewriteEnabled:false, catName:hive, ownerType:USER, writeId:0) |          |
{code}
SHOW CREATE TABLE foobar;

 
{code:java}
+----------------------------------------------------+

|                   createtab_stmt                   |

+----------------------------------------------------+

| CREATE TABLE `foobar`(                             |

|   `foo` string,                                    |

|   `bar` string)                                    |

| PARTITIONED BY (                                   |

|   `dt` string)                                     |

| ROW FORMAT SERDE                                   |

|   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'      |

| STORED AS INPUTFORMAT                              |

|   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'  |

| OUTPUTFORMAT                                       |

|   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |

| LOCATION                                           |

|   'hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar'
|

| TBLPROPERTIES (                                    |

|   'bucketing_version'='2',                         |

|   'last_modified_by'='hive',                       |

|   'last_modified_time'='1560986885',               |

|   'transactional'='true',                          |

|   'transactional_properties'='insert_only',        |

|   'transient_lastDdlTime'='1560986885')            |

+----------------------------------------------------+
{code}
So as it seems the table has only one SerDe, not per partition. Do we want to allow a different
SerDe per partition? Because if we do, it needs planning, and code changes. Or for now we
may stick to the one SerDe / table.

 

 

 

 

 

 

> Setting serde / serde properties for partitions
> -----------------------------------------------
>
>                 Key: HIVE-21897
>                 URL: https://issues.apache.org/jira/browse/HIVE-21897
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 3.1.1
>            Reporter: Miklos Gergely
>            Assignee: Ashutosh Chauhan
>            Priority: Major
>             Fix For: 4.0.0
>
>
> According to [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AddSerDeProperties] the
SerDe and the SerDe properties can be set for a partition too, so
>  
> {code:java}
> ALTERT TABLE table PARTITION (partition_col='partition_value') SET SERDE 'serde.class.name';{code}
> Is a valid statement. In fact it is not rejected, but it is not doing anything at all.
The execution is successful, everything remains the same. The same is true for setting the
serde properties:
> {code:java}
> ALTER TABLE table PARTITION (partition_col='partition_value') SET SERDEPROPERTIES ('property_name'='property_value');{code}
> is also a valid statement, and not doing anything.
> I suggest to modify the parser, and reject these statements. SerDe is for a table, and
not for a partition.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message