samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Hoover <roger.hoo...@gmail.com>
Subject Re: Keeping topic names out of the code?
Date Mon, 08 Sep 2014 21:51:36 GMT
"It might be a huge deal."  I mean "it might not* be huge deal.

On Mon, Sep 8, 2014 at 2:50 PM, Roger Hoover <roger.hoover@gmail.com> wrote:

> Hi,
>
> Wondering if people with more experience with Samza think it would be a
> good idea to keep topic names out of the code.  You might want to be able
> to change topics by editing the config instead of having to recompile the
> job.
>
> Maybe introduce an indirection so that output streams have names?
>
> Config:
> #Define an input named "raw" which maps to Kafka topic "wikipedia-raw"
> task.inputs.kafka.raw=wikipedia-raw
> #Use raw as an input
> task.inputs=kafka.raw
> #Define an output named "edits" which maps to Kafka topic "wikipedia-edits"
> task.outputs.kafka.edits=wikipedia-edits
>
> Task code:
>
> //Input stream would be called "raw" here instead of "wikipedia-raw"
> String stream =
> envelope.getSystemStreamPartition().getSystemStream().getStream();
> if (stream.equals("raw") {
>   processRawMsg(envelope, collector, coordinator);
> }
>
> //Send messages to locally named topic "edits"
> collector.send(new OutgoingMessageEnvelope(new SystemStream("kafka",
> "edits"), parsedJsonObject));
>
> Thoughts?  It might be a huge deal.  I just found myself copy and pasting
> names a lot across config and code files while writing some test jobs.
>
> Cheers,
>
> Roger
>
> Roger
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message