spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-12473) Reuse serializer instances for performance
Date Wed, 18 May 2016 11:33:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-12473:
------------------------------
    Target Version/s:   (was: 2.0.0)
            Priority: Minor  (was: Major)
          Issue Type: Improvement  (was: Bug)

> Reuse serializer instances for performance
> ------------------------------------------
>
>                 Key: SPARK-12473
>                 URL: https://issues.apache.org/jira/browse/SPARK-12473
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.6.0
>            Reporter: Andrew Or
>            Assignee: Andrew Or
>            Priority: Minor
>
> After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although currently
it's only 7%, we will likely register more classes in the future so this will only increase.
> The commit added 26 types to register every time we create a Kryo serializer instance.
I ran a small microbenchmark to prove that this is noticeably expensive:
> {code}
> import org.apache.spark.serializer._
> import org.apache.spark.SparkConf
> def makeMany(num: Int): Long = {
>   val start = System.currentTimeMillis
>   (1 to num).foreach { _ => new KryoSerializer(new SparkConf).newKryo() }
>   System.currentTimeMillis - start
> }
> // before commit de02782, averaged over multiple runs
> makeMany(5000) == 1500
> // after commit de02782, averaged over multiple runs
> makeMany(5000) == 2750
> {code}
> Since we create multiple serializer instances per partition, this means a 5000-partition
stage will unconditionally see an increase of > 1s for the stage. In page rank, we may
run many such stages.
> We should explore the alternative of reusing thread-local serializer instances, which
would lead to much fewer calls to `kryo.register`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message