spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elkhan Dadashov <elkhan8...@gmail.com>
Subject Re: Typed datataset from Avro generated classes?
Date Mon, 20 Apr 2020 08:47:48 GMT
Hi Spark users,

Did anyone resolve this issue?

Encoder<AvroGeneratedClass> encoder =
Encoders.bean(AvroGenereatedClass.class);
Dataset<AvroGeneratedClass> ds =
sparkSession.read().parquet(filename).as(encoder);

I'm also facing the same problem: "Cannot have circular references in bean
class, but got the circular reference of class class
org.apache.avro.Schema"

This happens due to getSchema() method in a generated Avro Java class.

How can I get a typed dataset from Avro generated classes?

Thanks.

On Wed, Sep 27, 2017 at 3:23 AM Joaquin Tarraga <joaquintarraga@gmail.com>
wrote:

> Hi all,
>
> I have an  Avro generated class (e.g., AvroGenerateClass) and I am using Encoders.bean
to get a typed dataset (e.g., Dataset<AvroGeneratedClass>):
>
> Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);
>
> Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);
>
> I am getting an exception from the Encoders.bean call:
> "java.lang.UnsupportedOperationException: Cannot have circular references
> in bean class, but got the circular reference of class class
> org.apache.avro.Schema"
>
> How can I get a typed dataset from Avro generated classes?
>
> Thanks.
> --
> JoaquĆ­n
>
>

-- 

Best regards,
Elkhan Dadashov

Mime
View raw message