spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Campbell <>
Subject Having trouble with streaming (updateStateByKey)
Date Wed, 11 Jun 2014 17:47:20 GMT
I'm having a little trouble getting an "updateStateByKey()" call to work;
was wondering if anyone could help.

In my chain of calls from getting Kafka messages out of the queue to
converting the message to a set of "things", then pulling out 2 attributes
of those things to a Tuple2, everything works.

So what I end up with is about a 1 second dump of things like this (this is
crufted up data, but it's basically 2 IPV6 addresses...)

Time: 1402507839000 ms

This works ok.

The problem is when I add a call to updateStateByKey - the streaming app
runs and runs and runs and never outputs anything.  When I debug, I can't
confirm that my state update passed-in function is ever actually getting

Indeed I have breakpoints and println statements in my updateFunc and it
LOOKS like it's never getting called.  I can confirm that the
updateStateByKey function IS getting called (via it stopping on a

Is there something obvious I'm missing?

View raw message