spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Checkpoint directory structure
Date Thu, 24 Sep 2015 01:44:50 GMT
Could you provide the logs on when and how you are seeing this error?

On Wed, Sep 23, 2015 at 6:32 PM, Bin Wang <wbin00@gmail.com> wrote:

> BTW, I just kill the application and restart it. Then the application
> cannot recover from checkpoint because of some lost of RDD. So I'm wonder,
> if there are some failure in the application, won't it possible not be able
> to recovery from checkpoint?
>
> Bin Wang <wbin00@gmail.com>于2015年9月23日周三 下午6:58写道:
>
>> I find the checkpoint directory structure is like this:
>>
>> -rw-r--r--   1 root root     134820 2015-09-23 16:55
>> /user/root/checkpoint/checkpoint-1442998500000
>> -rw-r--r--   1 root root     134768 2015-09-23 17:00
>> /user/root/checkpoint/checkpoint-1442998800000
>> -rw-r--r--   1 root root     134895 2015-09-23 17:05
>> /user/root/checkpoint/checkpoint-1442999100000
>> -rw-r--r--   1 root root     134899 2015-09-23 17:10
>> /user/root/checkpoint/checkpoint-1442999400000
>> -rw-r--r--   1 root root     134913 2015-09-23 17:15
>> /user/root/checkpoint/checkpoint-1442999700000
>> -rw-r--r--   1 root root     134928 2015-09-23 17:20
>> /user/root/checkpoint/checkpoint-1443000000000
>> -rw-r--r--   1 root root     134987 2015-09-23 17:25
>> /user/root/checkpoint/checkpoint-1443000300000
>> -rw-r--r--   1 root root     134944 2015-09-23 17:30
>> /user/root/checkpoint/checkpoint-1443000600000
>> -rw-r--r--   1 root root     134956 2015-09-23 17:35
>> /user/root/checkpoint/checkpoint-1443000900000
>> -rw-r--r--   1 root root     135244 2015-09-23 17:40
>> /user/root/checkpoint/checkpoint-1443001200000
>> drwxr-xr-x   - root root          0 2015-09-23 18:48
>> /user/root/checkpoint/d3714249-e03a-45c7-a0d5-1dc870b7d9f2
>> drwxr-xr-x   - root root          0 2015-09-23 17:44
>> /user/root/checkpoint/receivedBlockMetadata
>>
>>
>> I restart spark and it reads from
>> /user/root/checkpoint/d3714249-e03a-45c7-a0d5-1dc870b7d9f2. But it seems
>> that the data in it lost some rdds so it is not able to recovery. While I
>> find other directories in checkpoint/, like
>>  /user/root/checkpoint/checkpoint-1443001200000.  What does it used for?
>> Can I recovery my data from that?
>>
>

Mime
View raw message