spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?
Date Mon, 09 Oct 2017 19:08:01 GMT
Tried the following.


dataset.map(new MapFunction<String, List<Map<String, Object>>>() {
    @Override
    public List<Map<String, Object>> call(String input) throws Exception {
        List<Map<String, Object>>  temp = new ArrayList<>();
        temp.add(new HashMap<String, Object>());
        return temp;
    }
}, Encoders.kryo(List.class));


This doesn't even compile.

error: no suitable method found for map(<anonymous
MapFunction<String,List<Map<String,Object>>>>,Encoder<List>)
        limDS.map(new MapFunction<String, List<Map<String, Object>>>() {
             ^
    method Dataset.<U#1>map(Function1<String,U#1>,Encoder<U#1>) is not
applicable
      (cannot infer type-variable(s) U#1
        (argument mismatch; <anonymous
MapFunction<String,List<Map<String,Object>>>> cannot be converted to
Function1<String,U#1>))
    method Dataset.<U#2>map(MapFunction<String,U#2>,Encoder<U#2>) is
not applicable
      (inferred type does not conform to equality constraint(s)
        inferred: List
        equality constraints(s): List,List<Map<String,Object>>)
  where U#1,T,U#2 are type-variables:
    U#1 extends Object declared in method
<U#1>map(Function1<T,U#1>,Encoder<U#1>)
    T extends Object declared in class Dataset
    U#2 extends Object declared in method
<U#2>map(MapFunction<T,U#2>,Encoder<U#2>)



On Mon, Oct 9, 2017 at 11:48 AM, Koert Kuipers <koert@tresata.com> wrote:

> if you are willing to use kryo encoder you can do your original Dataset<
> List<Map<String,Object>>>> i think
>
> for example in scala i create here an intermediate Dataset[Any]:
>
> scala> Seq(1,2,3).toDS.map(x => if (x % 2 == 0) x else
> x.toString)(org.apache.spark.sql.Encoders.kryo[Any]).map{ (x: Any) => x
> match { case i: Int => i.toString; case s: String => s }}.show
> +-----+
> |value|
> +-----+
> |    1|
> |    2|
> |    3|
> +-----+
>
>
>
>
> On Mon, Oct 9, 2017 at 2:38 PM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi Koert,
>>
>> Thanks! If I have this  Dataset<Seq<Map<String, X>>> what would
be the
>> Enconding?is it Encoding.kryo(Seq.class) ?
>>
>> Also shouldn't List be supported? Should I create a ticket for this?
>>
>>
>> On Mon, Oct 9, 2017 at 6:10 AM, Koert Kuipers <koert@tresata.com> wrote:
>>
>>> it supports Dataset<Seq<Map<String, X>>> where X must be a
supported
>>> type also. Object is not a supported type.
>>>
>>> On Mon, Oct 9, 2017 at 7:36 AM, kant kodali <kanth909@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am wondering if spark supports Dataset<List<Map<String,Object>>>
?
>>>>
>>>> when I do the following it says no map function available?
>>>>
>>>> Dataset<List<Map<String,Object>>> resultDs = ds.map(lambda,
>>>> Encoders.bean(List.class));
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message