sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qian Xu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-1964) Sqoop2: Encode the decimal type accordingly to AVRO-1402
Date Sun, 04 Jan 2015 08:17:34 GMT
Qian Xu created SQOOP-1964:
------------------------------

             Summary: Sqoop2: Encode the decimal type accordingly to AVRO-1402 
                 Key: SQOOP-1964
                 URL: https://issues.apache.org/jira/browse/SQOOP-1964
             Project: Sqoop
          Issue Type: Improvement
            Reporter: Qian Xu


Decimal has been added to Avro relatively recently via [AVRO-1402|https://issues.apache.org/jira/browse/AVRO-1402].
It was added only as a so called “logical type” that is annotating already existing physical
type. The user (in this case us, Sqoop developers) have to do the proper encoding to final
bytes - e.g. convert the logical decimal value and store as described in the spec via “Fixed”
Avro type.

Currently converting decimal to Avro String is to be consistent with Sqoop 1 that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1 (several years
back), there was no better way how to store decimal in Avro. At that time, there was no support
for decimal in the platform at all. As the platform evolved and added a proper decimal type.
 

We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe that it’s
better to encode the decimal type accordingly to AVRO-1402 that rest of the platform (Hive,
Impala, …) can easily process rather then keep backward compatibility/consistency with Sqoop
1.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message