spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <>
Subject Re: Testing if an RDD is empty?
Date Fri, 16 Jan 2015 04:12:45 GMT

On Fri, Jan 16, 2015 at 7:31 AM, freedafeng <> wrote:
> I myself saw many times that my app threw out exceptions because an empty
> RDD cannot be saved. This is not big issue, but annoying. Having a cheap
> solution testing if an RDD is empty would be nice if there is no such thing
> available now.

I think the cheapest you can have is computing at least one element in the
RDD, which in the case of, say,

  val maybeEmptyRDD = veryExpensiveRDD.filter(false)

will be just as expensive as .count().


View raw message