spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Pavlyshyn <dimapavlys...@gmail.com>
Subject [No Subject]
Date Mon, 30 Mar 2020 20:51:57 GMT
Hello Apache Spark Support Team,
I am writing Spark on Java now. I use Dataset API and I face with an issue,
that  I am doing something like that

public <K, T> Dataset<Tuple2<K, List>> groupByKey(Dataset<Tuple2<K,
T>> consumers, Class<K> kClass) {

    consumers.groupBy("_1").agg(collect_list(col("_2"))).printSchema();
    return consumers.groupBy("_1").agg(collect_list(col("_2"))).as(Encoders.tuple(Encoders.bean(kClass),
            Encoders.bean(List.class)));
}

And I faced the issue that I can not deserialize collect_list part.
https://spark.apache.org/docs/latest/sql-reference.html#data-types  -
mapping ArrayType to java.util.List
Could you please give me any suggestions, wasted too much time trying to
fix it?
Best Regards,
Dmytro

Mime
View raw message