spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Jin <karen...@gmail.com>
Subject a question about "take"
Date Fri, 03 Jan 2014 06:02:27 GMT
Hi, 

I just started to fiddle with Spark a week ago. I am trying to understand take function more.
I have a JavaRDD<T> variable, let’s say dataSet which is initialized by getSequenceFileAsRDD,
when I try dataSet.take(1), it throughs an error as follows:

java.lang.IllegalStateException: Must not use direct buffers with
InputStream

But when I add add the following lines before the take call: 

        dataSet.context().setCheckpointDir("/tmp/checkpoint/", true);
        dataSet.checkpoint();
        dataSet.saveAsTextFile("/tmp/backup.txt”);

	dataSet.take(1);

Then take() works. I am confused here by how RDD's take method. Can anybody help me explain
why this happens?

Thanks a lot,

-cj
Mime
View raw message