On top of that you could make the topic part of the key (e.g. keyBy in .transform or manually emitting a tuple) and use one of the .xxxByKey operators for the processing.
If you have a stable, domain specific list of topics (e.g. 3-5 named topics) and the processing is
really different, I would also look at filtering by topic and saving as different Dstreams in your code.
Either way you need to start with Cody’s tip in order to extract the topic name.
From: Cody Koeninger
Date: Thursday, October 1, 2015 at 5:06 PM
To: Udit Mehta
Subject: Re: Kafka Direct Stream
You can get the topic for a given partition from the offset range. You can either filter using that; or just have a single rdd and match on topic when doing mapPartitions or foreachPartition (which I think is a better idea)