spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Gekk (JIRA)" <>
Subject [jira] [Created] (SPARK-27008) Support java.time.LocalDate as an external type of DateType
Date Wed, 27 Feb 2019 21:29:00 GMT
Maxim Gekk created SPARK-27008:

             Summary: Support java.time.LocalDate as an external type of DateType
                 Key: SPARK-27008
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Maxim Gekk

Currently, Spark supports the java.sql.Date as external types for Catalyst's DateType. It
accepts and produces values of such types. Since Java 8, base classes for dates are java.time.Instant,
java.time.LocalDate/LocalDateTime, and java.time.ZonedDateTime. Need to add new converters
from/to LocalDate.

The LocalDate type holds epoch days, and directly reflects to Catalyst's DateType.

Main motivations for the changes:
- Smoothly support Java 8 time API
- Avoid inconsistency of calendars used inside Spark 3.0 (Proleptic Gregorian calendar) and
inside of java.sql.Date (hybrid calendar - Julian + Gregorian). 
- Make conversion independent from current system timezone.

In case of collecting values of DateType, the following SQL config can control types of returned
 - spark.sql.catalyst.dateType with supported values "Date" (by default, java.sql.Date) and
"Instant" (java.time.LocalDate)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message