spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Spark Streaming not processing file with particular number of entries
Date Fri, 13 Jun 2014 21:25:09 GMT
In the logs you posted (the 2nd set), i dont see the file being picked up.
The lines having "FileInputDStream: Finding new files ..." should show the
file name that has been picked up and i dont see any file in the second set
logs.
If the file is already present in the directory by the time streaming
context is started then it wont be picked up.

TD


On Fri, Jun 13, 2014 at 2:04 PM, praveshjain1991 <praveshjain1991@gmail.com>
wrote:

> There doesn't seem to be any obvious reason - that's why it looks like a
> bug.
>
> The .4 million file is present in the directory when the context is started
> - same as for all other files (which are processed just fine by the
> application). In the logs we can see that the file is being picked up by
> the
> app but it is not starting the tasks that are supposed to begin.
>
> --
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-not-processing-file-with-particular-number-of-entries-tp6694p7599.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message