spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Omer (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-29595) Insertion with named_struct should match by name
Date Tue, 29 Oct 2019 08:18:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783
] 

Aman Omer edited comment on SPARK-29595 at 10/29/19 8:17 AM:
-------------------------------------------------------------

{color:#172b4d}Parameter required for named_struct is Seq(name1, val1, name2, val2, ...).
Validation step for named_struct only check for string at odd places. For example following
query will add a row in _str_ table.{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling
similar issue, changing fields of struct type according to names will introduce complexity.

So I think Spark should throw an exception when names does not match in named_struct.

cc [~srowen] [~maropu]


was (Author: aman_omer):
{color:#172b4d}Parameters required for named_struct is Seq(name1, val1, name2, val2, ...).
Validation step for named_struct only check for string at odd places. For example following
query will add a row in _str_ table.{color}{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling
similar issue, changing fields of struct type according to names will introduce complexity.

So I think Spark should throw an exception when names does not match in named_struct.

cc [~srowen] [~maropu]

> Insertion with named_struct should match by name
> ------------------------------------------------
>
>                 Key: SPARK-29595
>                 URL: https://issues.apache.org/jira/browse/SPARK-29595
>             Project: Spark
>          Issue Type: Task
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Gengliang Wang
>            Priority: Major
>
> {code:java}
> spark-sql> create table str using parquet as(select named_struct('a', 1, 'b', 2) as
data);
> spark-sql>  insert into str values named_struct("b", 3, "a", 1);
> spark-sql> select * from str;
> {"a":3,"b":1}
> {"a":1,"b":2}
> {code}
> The result should be 
> {code:java}
> {"a":1,"b":3}
> {"a":1,"b":2}
> {code}
> Spark should match the field names of named_struct on insertion



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message