[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783
]
Aman Omer edited comment on SPARK-29595 at 10/29/19 8:18 AM:
-------------------------------------------------------------
{color:#172b4d}named_struct takes Seq(name1, val1, name2, val2, ...) as parameter. Vmalidation
step for named_struct only check for string at odd places. For example following query will
add a row in _str_ table.{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling
similar issue, changing fields of struct type according to names will introduce complexity.
So I think Spark should throw an exception when names does not match in named_struct.
cc [~srowen] [~maropu]
was (Author: aman_omer):
{color:#172b4d}Parameter required for named_struct is Seq(name1, val1, name2, val2, ...).
Validation step for named_struct only check for string at odd places. For example following
query will add a row in _str_ table.{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling
similar issue, changing fields of struct type according to names will introduce complexity.
So I think Spark should throw an exception when names does not match in named_struct.
cc [~srowen] [~maropu]
> Insertion with named_struct should match by name
> ------------------------------------------------
>
> Key: SPARK-29595
> URL: https://issues.apache.org/jira/browse/SPARK-29595
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Gengliang Wang
> Priority: Major
>
> {code:java}
> spark-sql> create table str using parquet as(select named_struct('a', 1, 'b', 2) as
data);
> spark-sql> insert into str values named_struct("b", 3, "a", 1);
> spark-sql> select * from str;
> {"a":3,"b":1}
> {"a":1,"b":2}
> {code}
> The result should be
> {code:java}
> {"a":1,"b":3}
> {"a":1,"b":2}
> {code}
> Spark should match the field names of named_struct on insertion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|