flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Running on a firewalled Yarn cluster?
Date Mon, 02 Nov 2015 15:05:52 GMT

Here at work our security guys have chosen (long time ago) to only allow
the firewalls to have the ports open that needed (I say: good call!).
For the Yarn cluster this includes things like the proxy to see the
application manager of an application.
For everything we've done so far (i.e. mr/pig/...) this has worked fine.

Now with Flink I run into problems:
When I run either the yarn-session or a job on Yarn the application manager
gets started and I can see the webinterface.
The problem is that the jobmanager.rpc.address is on one of the worker
nodes and the jobmanager.rpc.port is essentially a random value.
A random value which is not accessible because of the firewall rules.
So I cannot reach the jobmanager on the yarn cluster.

How do I tackle this assuming that opening the all ports on the firewall is
not an option?

Or is this something that should be handled by Flink? ( Perhaps the
application manager can proxy the RPC calls? )

Best regards / Met vriendelijke groeten,

Niels Basjes

View raw message