flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Pompéry...@pompery.net>
Subject Re: Ceph configuration for checkpoints?
Date Tue, 13 Feb 2018 10:13:39 GMT
Hello Julio,

how we deploy Ceph:

1) make sure Ceph is mounted through all the nodes that make up your Flink
cluster; ensure path(s) are the same for each node
2) simply reference the Ceph mountpoint(s) via file://<your path>


Martin


On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski <piotr@data-artisans.com>
wrote:

> Hi,
>
> Have you tried to refer to ceph documentation?
> http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be:
>
> > a drop-in replacement for the Hadoop File System (HDFS)
>
> So I would first try to configure ceph according to their documentation
> and try to use Flink’s built in Hadoop connector.
>
> Piotrek
>
> On 12 Feb 2018, at 20:11, Julio Biason <julio.biason@azion.com> wrote:
>
> Hello people,
>
> I'm looking for the configuration options required to use Ceph as a
> backend for checkpoints.
>
> So far, I can see that `state.backend.fs.checkpointdir` and
> `state.checkpoints.dir` should be something like `
> ceph:///my-checkpoint-bucket`. But where do I define server and acess and
> secret keys? I couldn't find anything remotely related to that in the
> docs...
>
> --
> *Julio Biason*, Sofware Engineer
> *AZION*  |  Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101 <callto:+555130838101>  |  Mobile: +55 51
> <callto:+5551996209291>*99907 0554*
>
>
>

Mime
View raw message