spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aarondav <...@git.apache.org>
Subject [GitHub] incubator-spark pull request: Add subtractByKey to the JavaPairRDD...
Date Sun, 16 Feb 2014 20:02:56 GMT
Github user aarondav commented on the pull request:

    https://github.com/apache/incubator-spark/pull/600#issuecomment-35208921
  
    To answer my own question, the reason we do this is probably because we have no choice
when integrating with Java. Additionally, use of ClassTag as in our API rarely affects correctness,
just potentially safety and performance (e.g., creating a primitive-backed map versus an object
one).
    
    This unfortunately adds a burden we must keep in mind as we develop the Spark Scala APIs:
    
    - ClassTags should be used sparingly and never relied upon for correctness.
    - The Java API may be unavoidably slower than the Scala API where ClassTags are used for
perf.
    
    This also means it is very hard, in general, to verify the correctness of a Java API like
the one introduced in this patch, since we're tricking the compiler and introducing unusual
runtime behavior.
    
    In this particular case, I can verify that the API will work because subtractByKey never
actually uses W's ClassTag (I can even remove it from the signature without issue), but that's
not guaranteed to hold in the future.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.

Mime
View raw message