spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Jain <>
Subject Accessing the per-key state maintained by updateStateByKey for transformation of JavaPairDStream
Date Mon, 16 Jun 2014 18:01:27 GMT
Hello Spark Streaming Experts

I have a use-case, where I have a bunch of log-entries coming in, say every
10 seconds (Batch-interval). I create a JavaPairDStream[K,V] from these
log-entries. Now, there are two things I want to do with this

1. Use key-dependent state (updated by updateStateByKey) to apply a
transformation function on the JavaPairDStream[K, V]. I know that we get a
JavaPairDStream[K, S] as return value of updateStateByKey. However, I can't
possibly pass a JavaPairDStream to a transformation function, nor can I
convert JavaPairDStream[K,S]  to let's say a HashMap<K,S> (Or is there a way
to do this?). Even if I could convert it to a HashMap<K,S>, could I really
pass it to a transformation function, since this HashMap<K,S> changes after
every batch computation?

2. Update key-dependent state using Iterable<V>: This should be easily
doable using updateStateByKey.

In a nutshell, how do I access the very state updated by updateStateByKey
for applying let's say a map function on the JavaPairDStream[K,V]. Note that
I am not using any sliding windows at all. Just plain batches. 


View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message