spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Could not compute split, block not found
Date Tue, 01 Jul 2014 06:45:28 GMT
Are you by any change using only memory in the storage level of the input
streams?

TD


On Mon, Jun 30, 2014 at 5:53 PM, Tobias Pfeiffer <tgp@preferred.jp> wrote:

> Bill,
>
> let's say the processing time is t' and the window size t. Spark does not
> *require* t' < t. In fact, for *temporary* peaks in your streaming data, I
> think the way Spark handles it is very nice, in particular since 1) it does
> not mix up the order in which items arrived in the stream, so items from a
> later window will always be processed later, and 2) because an increase in
> data will not be punished with high load and unresponsive systems, but with
> disk space consumption instead.
>
> However, if all of your windows require t' > t processing time (and it's
> not because you are waiting, but because you actually do some computation),
> then you are in bad luck, because if you start processing the next window
> while the previous one is still processed, you have less resources for each
> and processing will take even longer. However, if you are only waiting
> (e.g., for network I/O), then maybe you can employ some asynchronous
> solution where your tasks return immediately and deliver their result via a
> callback later?
>
> Tobias
>
>
>
> On Tue, Jul 1, 2014 at 2:26 AM, Bill Jay <bill.jaypeterson@gmail.com>
> wrote:
>
>> Tobias,
>>
>> Your suggestion is very helpful. I will definitely investigate it.
>>
>> Just curious. Suppose the batch size is t seconds. In practice, does
>> Spark always require the program to finish processing the data of t seconds
>> within t seconds' processing time? Can Spark begin to consume the new batch
>> before finishing processing the next batch? If Spark can do them together,
>> it may save the processing time and solve the problem of data piling up.
>>
>> Thanks!
>>
>> Bill
>>
>>
>>
>>
>> On Mon, Jun 30, 2014 at 4:49 AM, Tobias Pfeiffer <tgp@preferred.jp>
>> wrote:
>>
>>> ​​If your batch size is one minute and it takes more than one minute to
>>> process, then I guess that's what causes your problem. The processing of
>>> the second batch will not start after the processing of the first is
>>> finished, which leads to more and more data being stored and waiting for
>>> processing; check the attached graph for a visualization of what I think
>>> may happen.
>>>
>>> Can you maybe do something hacky like throwing away a part of the data
>>> so that processing time gets below one minute, then check whether you still
>>> get that error?
>>>
>>> Tobias
>>>
>>>
>>> ​​
>>>
>>>
>>> On Mon, Jun 30, 2014 at 1:56 PM, Bill Jay <bill.jaypeterson@gmail.com>
>>> wrote:
>>>
>>>> Tobias,
>>>>
>>>> Thanks for your help. I think in my case, the batch size is 1 minute.
>>>> However, it takes my program more than 1 minute to process 1 minute's
>>>> data. I am not sure whether it is because the unprocessed data pile
>>>> up. Do you have an suggestion on how to check it and solve it? Thanks!
>>>>
>>>> Bill
>>>>
>>>>
>>>> On Sun, Jun 29, 2014 at 7:18 PM, Tobias Pfeiffer <tgp@preferred.jp>
>>>> wrote:
>>>>
>>>>> Bill,
>>>>>
>>>>> were you able to process all information in time, or did maybe some
>>>>> unprocessed data pile up? I think when I saw this once, the reason
>>>>> seemed to be that I had received more data than would fit in memory,
>>>>> while waiting for processing, so old data was deleted. When it was
>>>>> time to process that data, it didn't exist any more. Is that a
>>>>> possible reason in your case?
>>>>>
>>>>> Tobias
>>>>>
>>>>> On Sat, Jun 28, 2014 at 5:59 AM, Bill Jay <bill.jaypeterson@gmail.com>
>>>>> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I am running a spark streaming job with 1 minute as the batch size.
>>>>> It ran
>>>>> > around 84 minutes and was killed because of the exception with the
>>>>> following
>>>>> > information:
>>>>> >
>>>>> > java.lang.Exception: Could not compute split, block
>>>>> input-0-1403893740400
>>>>> > not found
>>>>> >
>>>>> >
>>>>> > Before it was killed, it was able to correctly generate output for
>>>>> each
>>>>> > batch.
>>>>> >
>>>>> > Any help on this will be greatly appreciated.
>>>>> >
>>>>> > Bill
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message