hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kalman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13386) Upgrade Avro to 1.8.x
Date Thu, 25 Apr 2019 12:07:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826000#comment-16826000
] 

Kalman commented on HADOOP-13386:
---------------------------------

Hi [~arpitagarwal]



Regarding the dependency, it is not really transitive dependency but run time dependency:

Sqoop is using avro 1.8.1 and parquet 1.8.x for a while and Hadoop 3.1.1 contains avro 1.7.7.

During the execution avro 1.7.7 will be used and the following runtime error occured when
sqoop tried to import to hive in parquet format:
{code}
ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError:
org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
 at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
{code}

So the version update what is defined by this task would fix our problem.

As validation we executed the standard set of hdp-3.1.0 system tests with Hadoop, Hive, Sqoop,
HBase etc components and we haven't found any issue in connection with the version update.

> Upgrade Avro to 1.8.x
> ---------------------
>
>                 Key: HADOOP-13386
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13386
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Ben McCann
>            Assignee: Kalman
>            Priority: Major
>
> Avro 1.8.x makes generated classes serializable which makes them much easier to use with
Spark. It would be great to upgrade Avro to 1.8.x



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message