kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garrett Barton <garrett.bar...@gmail.com>
Subject Flink ignoring -yn
Date Tue, 10 Jul 2018 17:09:14 GMT
Hey all,
I am running flink in batch mode on yarn with independant jobs creating
their own clusters.

 I have a flow defined that scales parallelism based on input size (to keep
overall processing time somewhat constant).  Right now the flow initializes
with around ~22k tasks for a flow that sets parallelism to 1600 for a few
portions.  At the cli I launch flink with:

flink run -m yarn-cluster -yn 200 -ys 4 -yqu test -c bla.. jar

My expectation is that this would create a max cluster of 800 slots from
200 task managers.  When I run with the input that scales to parallel 1600
I end up with over 2832 slots and 708 task managers.  What looks like its
doing is flink is consuming the entirety of the available yarn queue its
assigned to (happened to get ~75% of the queue that last run).

Does -yn act as a suggestion rather than a limit?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message