spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Wang <wbi...@gmail.com>
Subject Re: Checkpoint directory structure
Date Thu, 24 Sep 2015 01:32:01 GMT
BTW, I just kill the application and restart it. Then the application
cannot recover from checkpoint because of some lost of RDD. So I'm wonder,
if there are some failure in the application, won't it possible not be able
to recovery from checkpoint?

Bin Wang <wbin00@gmail.com>于2015年9月23日周三 下午6:58写道:

> I find the checkpoint directory structure is like this:
>
> -rw-r--r--   1 root root     134820 2015-09-23 16:55
> /user/root/checkpoint/checkpoint-1442998500000
> -rw-r--r--   1 root root     134768 2015-09-23 17:00
> /user/root/checkpoint/checkpoint-1442998800000
> -rw-r--r--   1 root root     134895 2015-09-23 17:05
> /user/root/checkpoint/checkpoint-1442999100000
> -rw-r--r--   1 root root     134899 2015-09-23 17:10
> /user/root/checkpoint/checkpoint-1442999400000
> -rw-r--r--   1 root root     134913 2015-09-23 17:15
> /user/root/checkpoint/checkpoint-1442999700000
> -rw-r--r--   1 root root     134928 2015-09-23 17:20
> /user/root/checkpoint/checkpoint-1443000000000
> -rw-r--r--   1 root root     134987 2015-09-23 17:25
> /user/root/checkpoint/checkpoint-1443000300000
> -rw-r--r--   1 root root     134944 2015-09-23 17:30
> /user/root/checkpoint/checkpoint-1443000600000
> -rw-r--r--   1 root root     134956 2015-09-23 17:35
> /user/root/checkpoint/checkpoint-1443000900000
> -rw-r--r--   1 root root     135244 2015-09-23 17:40
> /user/root/checkpoint/checkpoint-1443001200000
> drwxr-xr-x   - root root          0 2015-09-23 18:48
> /user/root/checkpoint/d3714249-e03a-45c7-a0d5-1dc870b7d9f2
> drwxr-xr-x   - root root          0 2015-09-23 17:44
> /user/root/checkpoint/receivedBlockMetadata
>
>
> I restart spark and it reads from
> /user/root/checkpoint/d3714249-e03a-45c7-a0d5-1dc870b7d9f2. But it seems
> that the data in it lost some rdds so it is not able to recovery. While I
> find other directories in checkpoint/, like
>  /user/root/checkpoint/checkpoint-1443001200000.  What does it used for?
> Can I recovery my data from that?
>

Mime
View raw message