spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: Take last k elements from RDD?
Date Thu, 24 Oct 2013 23:07:32 GMT
That's because a generic Iterator doesn't necessarily have a definite size
(cf. Iterator.hasDefiniteSize), and it is not possible to know where the
end of the Iterator is or how to take the last k elements from the
Iterator.  Indeed, a Iterator could be over an infinite stream and have no
end or last k elements.


On Thu, Oct 24, 2013 at 3:59 PM, Stoney Vintson <stoneyv@gmail.com> wrote:

> I don't see common iterable trait methods such as takeRight() or last() in
> the spark scala api documentation.  There are sampling and sorting methods.
>  sample, sortByKey
>
> spark scala api
>
> http://spark.incubator.apache.org/docs/latest/scala-programming-guide.html#rdd-operations
>
> org.apache.spark.rdd
>
> http://spark.incubator.apache.org/docs/latest/api/core/index.html#org.apache.spark.rdd.RDD
>
> spark-0.8.0-incubating/core/src/main/scala/org/apache/spark/rdd
>
> Also, remember that in scala tail() will return all of the elements except
> for head.
>
>
>
> On Thu, Oct 24, 2013 at 3:23 PM, Matt Cheah <mcheah@palantir.com> wrote:
>
>>  Hi everyone,
>>
>>  I see there is a take() function for RDDs, getting the first n
>> elements. Is there a way to get the last n elements?
>>
>>  Thanks,
>>
>>  -Matt Cheah
>>
>
>
>

Mime
View raw message