spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From markhamstra <>
Subject [GitHub] incubator-spark pull request: Clean up and added a few java api me...
Date Fri, 14 Feb 2014 07:03:37 GMT
Github user markhamstra commented on the pull request:
    It's what I'd be inclined to do.
    I think there is a need for something with a higher-level of abstraction than mapPartitions/mapPartitionsWithIndex
that avoids the boilerplate and potential pitfalls when the need is to create some per-partition
resource and then clean those up when they are no longer needed; but the *With functions neither
meet that entire need nor do they do what they do in a way that is significantly easier to
code than using mapPartitions/mapPartitionsWithIndex directly.
    Deprecate these and perhaps add a better attempt in 1.1+.0.

View raw message