spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: DataSet Evidence
Date Tue, 01 Mar 2016 23:46:12 GMT
Hey Steve,

This isn't possible today, but it would not be hard to allow.  You should
open a feature request JIRA.

Michael

On Mon, Feb 29, 2016 at 4:55 PM, Steve Lewis <lordjoe2000@gmail.com> wrote:

>  I have a relatively complex Java object that I would like to use in a
> dataset
>
> if I say
>
> Encoder<MyType> evidence = Encoders.kryo(MyType.class);
>
> JavaRDD<MyType> rddMyType= generateRDD(); // some code
>
>  Dataset<MyType> datasetMyType= sqlCtx.createDataset( rddMyType.rdd(), evidence);
>
>
> I get one column - the whole object
>
> The object is a bean with all fields having getters and setters but some of the fields
are other complex java objects -
>
> It would be fine to serielize the objects in these fields with Kryo or Java serialization
but the Bean serializer treats all referenced objects as beans and some lack the required
getter and setter fields
>
> How can I get my columns with bean serializer even if some of the values in the columns
are not bean types
>
>

Mime
View raw message