flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Flink and Prometheus setup in K8s
Date Thu, 16 May 2019 07:30:03 GMT
Thanks for sharing your solution Wouter!

Best, Fabian

Am Mi., 15. Mai 2019 um 15:28 Uhr schrieb Wouter Zorgdrager <
W.D.Zorgdrager@tudelft.nl>:

> Hi all,
>
> To answer my own questions I worked on the following solution:
>
> 1) Custom Docker image which pulls the Flink image and moves Prometheus
> jar to the correct folder [1, 2].
> 2) I wrote manifests for Kubernetes with service discovery configuration
> for Kubernetes [3]. Besides the 'official' Flink Kubernetes manifests, I
> added a TM service which exposes all TM metrics locations so that
> Prometheus can scrape it. This means that (re)-scaling Flink TM's are
> automatically picked up by Prometheus. The repository also includes a
> Grafana setup with a simple dashboard.
>
> I thought this might be useful for other users!
>
> Cheers,
> Wouter
>
> [1]:
> https://github.com/wzorgdrager/flink-k8s/blob/master/docker/Dockerfile
> [2]: https://hub.docker.com/r/wzorgdrager/flink-prometheus
> [3]: https://github.com/wzorgdrager/flink-k8s
>
> Op ma 13 mei 2019 om 14:16 schreef Wouter Zorgdrager <
> W.D.Zorgdrager@tudelft.nl>:
>
>> Hey all,
>>
>> I'm working on a deployment setup with Flink and Prometheus on
>> Kubernetes. I'm running into the following issues:
>>
>> 1) Is it possible to use the default Flink Docker image [1] and enable
>> the Prometheus reporter? Modifying the flink-config.yaml is easy, but
>> somehow the Prometheus reporter jar needs to be moved within the image.
>> This is easy if use my own Dockerfile (as done here [2]) , but I prefer
>> using the official one.
>> 2) I can define the jobmanager/taskmanager metric endpoints statically,
>> but w.r.t. scaling I prefer to have these resolved/discovered dynamically.
>> Did anyone get a working setup on this? I came across this resource for
>> YARN [3], is there something similar for Kubernetes? Or are there any other
>> ways of configuring Prometheus to pick this up automatically?
>>
>> Thanks a lot for your help!
>>
>> Kind regards,
>> Wouter
>>
>> [1]: https://hub.docker.com/_/flink/
>> [2]:
>> https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile
>> [3]: https://github.com/eastcirclek/flink-service-discovery
>>
>

Mime
View raw message