sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
Date Mon, 03 Dec 2018 16:26:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707470#comment-16707470
] 

ASF GitHub Bot commented on SQOOP-3396:
---------------------------------------

Github user szvasas commented on a diff in the pull request:

    https://github.com/apache/sqoop/pull/60#discussion_r238318482
  
    --- Diff: src/java/org/apache/sqoop/hive/HiveTypes.java ---
    @@ -83,27 +89,58 @@ public static String toHiveType(int sqlType) {
           }
       }
     
    -  public static String toHiveType(Schema.Type avroType) {
    -      switch (avroType) {
    -        case BOOLEAN:
    -          return HIVE_TYPE_BOOLEAN;
    -        case INT:
    -          return HIVE_TYPE_INT;
    -        case LONG:
    -          return HIVE_TYPE_BIGINT;
    -        case FLOAT:
    -          return HIVE_TYPE_FLOAT;
    -        case DOUBLE:
    -          return HIVE_TYPE_DOUBLE;
    -        case STRING:
    -        case ENUM:
    -          return HIVE_TYPE_STRING;
    -        case BYTES:
    -        case FIXED:
    -          return HIVE_TYPE_BINARY;
    -        default:
    -          return null;
    +  public static String toHiveType(Schema schema, SqoopOptions options) {
    +    if (schema.getType() == Schema.Type.UNION) {
    +      for (Schema subSchema : schema.getTypes()) {
    +        if (subSchema.getType() != Schema.Type.NULL) {
    +          return toHiveType(subSchema, options);
    +        }
    +      }
    +    }
    +
    +    Schema.Type avroType = schema.getType();
    +    switch (avroType) {
    +      case BOOLEAN:
    +        return HIVE_TYPE_BOOLEAN;
    +      case INT:
    +        return HIVE_TYPE_INT;
    +      case LONG:
    +        return HIVE_TYPE_BIGINT;
    +      case FLOAT:
    +        return HIVE_TYPE_FLOAT;
    +      case DOUBLE:
    +        return HIVE_TYPE_DOUBLE;
    +      case STRING:
    +      case ENUM:
    +        return HIVE_TYPE_STRING;
    +      case BYTES:
    +        return mapToDecimalOrBinary(schema, options);
    +      case FIXED:
    +        return HIVE_TYPE_BINARY;
    +      default:
    +        throw new RuntimeException(String.format("There is no Hive type mapping defined
for the Avro type of: %s ", avroType.getName()));
    +    }
    +  }
    +
    +  private static String mapToDecimalOrBinary(Schema schema, SqoopOptions options) {
    +    boolean logicalTypesEnabled = options.getConf().getBoolean(ConfigurationConstants.PROP_ENABLE_PARQUET_LOGICAL_TYPE_DECIMAL,
false);
    +    if (logicalTypesEnabled && schema.getLogicalType() != null && schema.getLogicalType()
instanceof Decimal) {
    --- End diff --
    
    The null check is redundant here.


> Add parquet numeric support for Parquet in Hive import
> ------------------------------------------------------
>
>                 Key: SQOOP-3396
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3396
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Fero Szabo
>            Assignee: Fero Szabo
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message