spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sabarish Sasidharan <sabarish.sasidha...@manthan.com>
Subject Re: DataSet Evidence
Date Tue, 01 Mar 2016 08:02:35 GMT
BeanInfo?
On 01-Mar-2016 6:25 am, "Steve Lewis" <lordjoe2000@gmail.com> wrote:

>  I have a relatively complex Java object that I would like to use in a
> dataset
>
> if I say
>
> Encoder<MyType> evidence = Encoders.kryo(MyType.class);
>
> JavaRDD<MyType> rddMyType= generateRDD(); // some code
>
>  Dataset<MyType> datasetMyType= sqlCtx.createDataset( rddMyType.rdd(), evidence);
>
>
> I get one column - the whole object
>
> The object is a bean with all fields having getters and setters but some of the fields
are other complex java objects -
>
> It would be fine to serielize the objects in these fields with Kryo or Java serialization
but the Bean serializer treats all referenced objects as beans and some lack the required
getter and setter fields
>
> How can I get my columns with bean serializer even if some of the values in the columns
are not bean types
>
>

Mime
View raw message