spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Spark Streaming not processing file with particular number of entries
Date Fri, 13 Jun 2014 20:55:46 GMT
This is very odd. If it is running fine on mesos, I dont see a obvious
reason why it wont work on Spark standalone cluster.
Is the .4 million file already present in the monitored directory when the
context is started? In that case, the file will not be picked up (unless
textFileStream is created with that option to process existing files set).
By default, only new files in a directory are picked up.

TD


On Tue, Jun 10, 2014 at 11:35 PM, praveshjain1991 <praveshjain1991@gmail.com
> wrote:

> Well i was able to get it to work by running spark over mesos. But it looks
> like a bug while running spark alone.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-not-processing-file-with-particular-number-of-entries-tp6694p7382.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message