spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Testing if an RDD is empty?
Date Fri, 16 Jan 2015 04:22:07 GMT
How about checking whether take(1).length == 0? If I read the code
correctly, this will only examine the first partition, at least.

On Fri, Jan 16, 2015 at 4:12 AM, Tobias Pfeiffer <tgp@preferred.jp> wrote:
> Hi,
>
> On Fri, Jan 16, 2015 at 7:31 AM, freedafeng <freedafeng@yahoo.com> wrote:
>>
>> I myself saw many times that my app threw out exceptions because an empty
>> RDD cannot be saved. This is not big issue, but annoying. Having a cheap
>> solution testing if an RDD is empty would be nice if there is no such
>> thing
>> available now.
>
>
> I think the cheapest you can have is computing at least one element in the
> RDD, which in the case of, say,
>
>   val maybeEmptyRDD = veryExpensiveRDD.filter(false)
>
> will be just as expensive as .count().
>
> Tobias
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message