spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuming Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23178) Kryo Unsafe problems with count distinct from cache
Date Wed, 21 Nov 2018 02:21:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694120#comment-16694120
] 

Yuming Wang commented on SPARK-23178:
-------------------------------------

Cloud you try spark-2.4.0 or master branch? we have upgraded kryo to 4.0.2.

> Kryo Unsafe problems with count distinct from cache
> ---------------------------------------------------
>
>                 Key: SPARK-23178
>                 URL: https://issues.apache.org/jira/browse/SPARK-23178
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0, 2.2.1
>            Reporter: KIryl Sultanau
>            Priority: Minor
>         Attachments: Unsafe-issue.png, Unsafe-off.png
>
>
> Spark incorrectly process cached data with Kryo & Unsafe options.
> Distinct count from cache doesn't work correctly. Example available below:
> {quote}val spark = SparkSession
>          .builder
>          .appName("unsafe-issue")
>          .master("local[*]")
>          .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
>          .config("spark.kryo.unsafe", "true")
>          .config("spark.kryo.registrationRequired", "false")
>          .getOrCreate()
>     val devicesDF = spark.read.format("csv")
>      .option("header", "true")
>      .option("delimiter", "\t")
>      .load("/data/Devices.tsv").cache()
>     val gatewaysDF = spark.read.format("csv")
>      .option("header", "true")
>      .option("delimiter", "\t")
>      .load("/data/Gateways.tsv").cache()
>     val devJoinedDF = devicesDF.join(gatewaysDF, Seq("GatewayId"), "inner").cache()
>      devJoinedDF.printSchema()
>     println(devJoinedDF.count())
>     println(devJoinedDF.select("DeviceId").distinct().count())
>     println(devJoinedDF.groupBy("DeviceId").count().filter("count>1").count())
>     println(devJoinedDF.groupBy("DeviceId").count().filter("count=1").count())
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message