spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joaquin Tarraga <joaquintarr...@gmail.com>
Subject Typed datataset from Avro generated classes?
Date Wed, 27 Sep 2017 10:23:45 GMT
Hi all,

I have an  Avro generated class (e.g., AvroGenerateClass) and I am
using Encoders.bean to get a typed dataset (e.g.,
Dataset<AvroGeneratedClass>):

Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);

Dataset<AvroGeneratedClass> ds =
sparkSession.read().parquet(filename).as(encoder);

I am getting an exception from the Encoders.bean call:
"java.lang.UnsupportedOperationException: Cannot have circular references
in bean class, but got the circular reference of class class
org.apache.avro.Schema"

How can I get a typed dataset from Avro generated classes?

Thanks.
--
JoaquĆ­n

Mime
View raw message