spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Testing if an RDD is empty?
Date Fri, 16 Jan 2015 04:12:45 GMT
Hi,

On Fri, Jan 16, 2015 at 7:31 AM, freedafeng <freedafeng@yahoo.com> wrote:
>
> I myself saw many times that my app threw out exceptions because an empty
> RDD cannot be saved. This is not big issue, but annoying. Having a cheap
> solution testing if an RDD is empty would be nice if there is no such thing
> available now.
>

I think the cheapest you can have is computing at least one element in the
RDD, which in the case of, say,

  val maybeEmptyRDD = veryExpensiveRDD.filter(false)

will be just as expensive as .count().

Tobias

Mime
View raw message