spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Chen <kc...@palantir.com>
Subject Re: Deserializing JSON into Scala objects in Java code
Date Tue, 08 Sep 2015 20:27:29 GMT
Hi Marcelo,

 Thanks for the quick response. I understand that I can just write my own
Java classes (I will use that as a fallback option), but in order to avoid
code duplication and further possible changes, I was hoping there would be
a way to use the Spark API classes directly, since it seems there should
be.

 I registered the Scala module in the same way (except in Java instead of
Scala),

mapper.registerModule(new DefaultScalaModule());

But I don’t think the module is being used/registered properly? Do you
happen to know whether the above line should work in Java?



On 9/8/15, 12:55 PM, "Marcelo Vanzin" <vanzin@cloudera.com> wrote:

>Hi Kevin,
>
>How did you try to use the Scala module? Spark has this code when
>setting up the ObjectMapper used to generate the output:
>
>  
>mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModul
>e)
>
>As for supporting direct serialization to Java objects, I don't think
>that was the goal of the API. The Scala API classes are public mostly
>so that API compatibility checks are performed against them. If you
>don't mind the duplication, you could write your own Java POJOs that
>mirror the Scala API, and use them to deserialize the JSON.
>
>
>On Tue, Sep 8, 2015 at 12:46 PM, Kevin Chen <kchen@palantir.com> wrote:
>> Hello Spark Devs,
>>
>>  I am trying to use the new Spark API json endpoints at /api/v1/[path]
>> (added in SPARK-3454).
>>
>>  In order to minimize maintenance on our end, I would like to use
>> Retrofit/Jackson to parse the json directly into the Scala classes in
>> org/apache/spark/status/api/v1/api.scala (ApplicationInfo,
>> ApplicationAttemptInfo, etc…). However, Jackson does not seem to know
>>how to
>> handle Scala Seqs, and will throw an error when trying to parse the
>> attempts: Seq[ApplicationAttemptInfo] field of ApplicationInfo. Our
>>codebase
>> is in Java.
>>
>>  My questions are:
>>
>> Do you have any recommendations on how to easily deserialize Scala
>>objects
>> from json? For example, do you have any current usage examples of
>>SPARK-3454
>> with Java?
>> Alternatively, are you committed to the json formats of /api/v1/path? I
>> would guess so, because of the ‘v1’, but wanted to confirm. If so, I
>>could
>> deserialize the json into instances of my own Java classes instead,
>>without
>> worrying about changing the class structure later due to changes in the
>> Spark API.
>>
>> Some further information:
>>
>> The error I am getting with Jackson when trying to deserialize the json
>>into
>> ApplicationInfo is Caused by:
>> com.fasterxml.jackson.databind.JsonMappingException: Can not construct
>> instance of scala.collection.Seq, problem: abstract types either need
>>to be
>> mapped to concrete types, have custom deserializer, or be instantiated
>>with
>> additional type information
>> I tried using Jackson’s DefaultScalaModule, which seems to have support
>>for
>> Scala Seqs, but got no luck.
>> Deserialization works if the Scala class does not have any Seq fields,
>>and
>> works if the fields are Java Lists instead of Seqs.
>>
>> Thanks very much for your help!
>> Kevin Chen
>>
>
>
>
>-- 
>Marcelo

Mime
View raw message