spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dr. Kent Yao" <yaooq...@qq.com>
Subject Re: [DISCUSS] Support year-month and day-time Intervals
Date Fri, 17 Jan 2020 02:52:00 GMT
Following ANSI might be a good option but also a serious user behavior change
to introduce two different interval types, so I also agree with Reynold to
follow what we have done since version 1.5.0, just like Snowflake and
Redshift.

Perhaps, we can make some efforts for the current interval type to make it
more future-proofing. e.g.
1. add unstable annotation to the CalendarInterval class. People already use
it as UDF inputs so it’s better to make it clear it’s unstable.
2. Add a schema checker to prohibit create v2 custom catalog table with
intervals, as same as what we do for the builtin catalog
3. Add a schema checker for DataFrameWriterV2 too
4. Make the interval type incomparable as version 2.4 for disambiguation of
comparison between year-month and day-time fields
5. The 3.0 newly added to_csv should not support output intervals as same as
using CSV file format
6. The function to_json should not allow using interval as a key field as
same as the value field and JSON datasource, with a legacy config to
restore.
7. Revert interval ISO/ANSI SQL Standard output since we decide not to
follow ANSI, so there is no round trip.

Bests,

Kent




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message