flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Task and Operator Monitoring via JMX / naming
Date Sat, 15 Oct 2016 07:18:00 GMT
Hello Philipp,

there is certainly something very wrong here.

What you _should_ see is 6 entries, 1 for each operator; 2-3 more for 
the tasks the operators are executed in and the taskmanager stuff.

Usually, operator metrics use the name that you configured, like 
"TokenMapStream", whereas tasks use the concatenation of all operator 
names joined with =>, as in "KafkaSource => TokenMapStream".

I will look into this, I've never seen these issues before.

One more thing, which version of Flink are you currently using?


On 15.10.2016 00:07, Philipp Bussche wrote:
> Hi there,
> I am struggeling to understand what I am looking at after enabling JMX
> metric reporting on my taskmanager.
> The job I am trying this out with has 1 source, 2 map functions (where one
> is a RichMap) and 3 sinks.
> This is how I have written my Job:
> DataStream<Invitation> invitations = streaming
> 					.addSource(new FlinkKafkaConsumer09<>(
> 							dsp.getPropertyAsString("kafka.invitation.topic"),
> 							new InvitationSchema(),
> 							kafkaProps)).name("KafkaSource");
> invitations.addSink(new
> PostgresqlInvitationDetailsSink<>(psqlConfig)).name("InvitationDetailSink");
> DataStream<Tuple2&lt;String, String>> tokens = invitations
> 				.map(new TokenExtractor()).name("TokenMapStream");
> tokens.addSink(new
> PostgresqlInvitationTokenSink<>(psqlConfig)).name("InvitationTokenSink");
> DataStream<Tuple4&lt;String, String, String, String>> invitationResponses
> invitations
> 				.map(new InvitationDetailsExtractor(psqlConfig,
> tokenToSenderMapping)).name("InvitationDetailsRichMapStream");
> invitationResponses.addSink(new
> Neo4JInvitationSink<>(neo4jConfig)).name("InvitationRelationshipSink");
> streaming.execute("InvitationJob");
> Somehow I was expecting to have metrics representing the source, the sinks
> and the operators, however instead of 6 entries in my JMX tree I only have
> 4. Please see screenshot attached. Also I was somehow expecting the JMX
> objects to be named like my task / operator names but it has all sorts of
> prefix/suffix magic around the names. Finally I have one custom metric which
> obviously is attached to my RichMapFunction (InvitationDetailsExtractor).
> However the custom metric (invitationDetailsAdded) shows up under an object
> where one of the keys (which I would expect to be set to the operation name)
> is a combination of the prefix "Sink" plus the name of the first sink that I
> am using plus the name of the first map function (which is not the
> RichMapFunction actually: my custom metric "invitationDetailsAdded" shows up
> under "(Sink-_InvitationDetailSink-_TokenMapStream" which is very confusing
> because this metric is actually incremented as part of the
> InvitationDetailsRichMapStream).
> Can somebody please explain what I can expect from metrics exposed via JMX
> (should they really represent my tasks and operations) and why the naming is
> so strange ?
> Thanks
> Philipp
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n9560/Screen_Shot_2016-10-15_at_00.png>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Task-and-Operator-Monitoring-via-JMX-naming-tp9560.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message