kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Noll <mich...@confluent.io>
Subject Re: Kafka without Storm/Spark
Date Fri, 01 Jul 2016 14:49:06 GMT
Numan,

you may also want to take a look at Kafka Streams, which is a new stream
processing library that's included in Apache Kafka since version 0.10.
Kafka Streams is definitely more convenient and quicker to implement than
the "normal" Kafka producer/consumer clients.  Also, Kafka Streams does not
require a separate processing cluster like Spark or Storm -- you only need
the Kafka cluster to read data from / to write data to.

> My question is:Is it possible to consume messages from a Kafka topic in
real-time
> and write directly into another topic without using any streaming
technology such
> as storm or spark? If yes, do you have any examples to do that in Java?

There's one specific examples that does exactly this:
https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/PassThroughIntegrationTest.java

Here are the key snippets that you would be using:

    Properties streamsConfiguration = new Properties();
    streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG,
"pass-through-integration-test");
    //...more configs here...

    KStreamBuilder builder = new KStreamBuilder();
    // Write the input data as-is to the output topic.
    builder.stream("my-input-topic").to("my-output-topic");
    KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
    streams.start();

Of course you can also transform the messages easily prior to writing them
to the output topic.  See the other examples/references down below.

Hope this helps,
Michael


Kafka Streams docs:
http://kafka.apache.org/documentation.html#streams
http://docs.confluent.io/3.0.0/streams/index.html

Some Kafka Streams examples:
https://github.com/confluentinc/examples/tree/master/kafka-streams

Some Kafka Streams articles:
http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple



On Fri, Jul 1, 2016 at 4:29 PM, numangoceri <numangoceri@yahoo.com.invalid>
wrote:

> Hi,
>
> Thanks for your answer. I meant actually if we can verify the data
> reliability without using Storm or Spark. Because i know that by using
> Storm, you can guarantee the messages (depending on the type of the
> Topology) such as exactly once, at least once.
> If i simply use kafka consumer and another producer to forward the
> messages, could the data tranfer completely be guaranteed as well?
>
>
> Numan Göceri
>
> ---
>
> Rakesh Vidyadharan <rvidyadharan@gracenote.com> wrote:
>
> >Definitely.  You can read off kafka using the samples shown in
> KafkaConsumer javadoc, transform if necessary and publish to the
> destination topic.
> >
> >
> >
> >
> >On 01/07/2016 03:24, "numan goceri" <numangoceri@yahoo.com.INVALID>
> wrote:
> >
> >>Hello everyone,
> >>I've a quick question:I'm using Apache Kafka producer to write the
> messages into a topic. My source at the moment a csv file but in the future
> i am supposed to read the messages from another kafka topic.My question
> is:Is it possible to consume messages from a Kafka topic in real-time and
> write directly into another topic without using any streaming technology
> such as storm or spark? If yes, do you have any examples to do that in Java?
> >>To sum up, it should be looking like this:Kafka reads from topic
> "kafkaSource" and writes into the topic "kafkaResult".
> >>
> >>Thanks in advance and Best Regards, Numan
>



-- 
Best regards,
Michael Noll



*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message