spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Converting a json struct to map
Date Wed, 19 Nov 2014 19:07:57 GMT
You can override the schema inference by passing a schema as the second
argument to jsonRDD, however thats not a super elegant solution.  We are
considering one option to make this easier here:
https://issues.apache.org/jira/browse/SPARK-4476

On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> Something like this?
>
>    val map_rdd = json_rdd.map(json => {
>       val mapper = new ObjectMapper() with ScalaObjectMapper
>       mapper.registerModule(DefaultScalaModule)
>
>       val myMap = mapper.readValue[Map[String,String]](json)
>
>       myMap
>     })
>
> Thanks
> Best Regards
>
> On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv <danielrulez@gmail.com>
> wrote:
>
>> Hi,
>> I'm loading a json file into a RDD and then save that RDD as parquet.
>> One of the fields is a map of keys and values but it is being translated
>> and stored as a struct.
>>
>> How can I convert the field into a map?
>>
>>
>> Thanks,
>> Daniel
>>
>
>

Mime
View raw message