sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qian Xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1964) Sqoop2: Encode the decimal type accordingly to AVRO-1402
Date Sun, 04 Jan 2015 08:19:34 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Qian Xu updated SQOOP-1964:
---------------------------
    Description: 
Decimal has been added to Avro relatively recently via AVRO-1402. It was added only as a so
called “logical type” that is annotating already existing physical type. The user (in
this case us, Sqoop developers) have to do the proper encoding to final bytes - e.g. convert
the logical decimal value and store as described in the spec via “Fixed” Avro type.

Currently converting decimal to Avro String is to be consistent with Sqoop 1 that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1 (several years
back), there was no better way how to store decimal in Avro. At that time, there was no support
for decimal in the platform at all. As the platform evolved and added a proper decimal type.
 

We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe that it’s
better to encode the decimal type accordingly to AVRO-1402 that rest of the platform (Hive,
Impala, …) can easily process rather then keep backward compatibility/consistency with Sqoop
1.


  was:
Decimal has been added to Avro relatively recently via [AVRO-1402|https://issues.apache.org/jira/browse/AVRO-1402].
It was added only as a so called “logical type” that is annotating already existing physical
type. The user (in this case us, Sqoop developers) have to do the proper encoding to final
bytes - e.g. convert the logical decimal value and store as described in the spec via “Fixed”
Avro type.

Currently converting decimal to Avro String is to be consistent with Sqoop 1 that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1 (several years
back), there was no better way how to store decimal in Avro. At that time, there was no support
for decimal in the platform at all. As the platform evolved and added a proper decimal type.
 

We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe that it’s
better to encode the decimal type accordingly to AVRO-1402 that rest of the platform (Hive,
Impala, …) can easily process rather then keep backward compatibility/consistency with Sqoop
1.



> Sqoop2: Encode the decimal type accordingly to AVRO-1402 
> ---------------------------------------------------------
>
>                 Key: SQOOP-1964
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1964
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Qian Xu
>
> Decimal has been added to Avro relatively recently via AVRO-1402. It was added only as
a so called “logical type” that is annotating already existing physical type. The user
(in this case us, Sqoop developers) have to do the proper encoding to final bytes - e.g. convert
the logical decimal value and store as described in the spec via “Fixed” Avro type.
> Currently converting decimal to Avro String is to be consistent with Sqoop 1 that [does
the same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1 (several years
back), there was no better way how to store decimal in Avro. At that time, there was no support
for decimal in the platform at all. As the platform evolved and added a proper decimal type.
 
> We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe that
it’s better to encode the decimal type accordingly to AVRO-1402 that rest of the platform
(Hive, Impala, …) can easily process rather then keep backward compatibility/consistency
with Sqoop 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message