spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yin Huai <huaiyin....@gmail.com>
Subject Re: Converting a json struct to map
Date Wed, 19 Nov 2014 20:23:10 GMT
Oh, actually, we do not support MapType provided by the schema given to
jsonRDD at the moment (my bad..). Daniel, you need to wait for the patch of
4476 (I should have one soon).

Thanks,

Yin

On Wed, Nov 19, 2014 at 2:32 PM, Daniel Haviv <danielrulez@gmail.com> wrote:

> Thank you Michael
> I will try it out tomorrow
>
> Daniel
>
> On 19 בנוב׳ 2014, at 21:07, Michael Armbrust <michael@databricks.com>
> wrote:
>
> You can override the schema inference by passing a schema as the second
> argument to jsonRDD, however thats not a super elegant solution.  We are
> considering one option to make this easier here:
> https://issues.apache.org/jira/browse/SPARK-4476
>
> On Tue, Nov 18, 2014 at 11:06 PM, Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> Something like this?
>>
>>    val map_rdd = json_rdd.map(json => {
>>       val mapper = new ObjectMapper() with ScalaObjectMapper
>>       mapper.registerModule(DefaultScalaModule)
>>
>>       val myMap = mapper.readValue[Map[String,String]](json)
>>
>>       myMap
>>     })
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Nov 19, 2014 at 11:01 AM, Daniel Haviv <danielrulez@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I'm loading a json file into a RDD and then save that RDD as parquet.
>>> One of the fields is a map of keys and values but it is being translated
>>> and stored as a struct.
>>>
>>> How can I convert the field into a map?
>>>
>>>
>>> Thanks,
>>> Daniel
>>>
>>
>>
>

Mime
View raw message