kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Torenvliet <natorenvl...@gmail.com>
Subject Re: Streaming Data
Date Wed, 10 Apr 2019 12:22:07 GMT
Hi All, thanks for the recommendations.  I really like staying close to the
Apache.org product line - so I will likely go the route that puts kafka
streams or spark into play.

On Wed, Apr 10, 2019 at 11:45 AM Robin Moffatt <robin@confluent.io> wrote:

> +1 for looking into Kafka Streams. You can build and maintain state within
> your app, and expose it on a REST endpoint for your node/react app to
> query.
> There's an example of this here:
>
> https://github.com/confluentinc/kafka-streams-examples/blob/5.2.1-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java
>
> Depending on the processing you want to do KSQL could also be useful,
> writing the results of queries (e.g. spotting exceptions in the data) to a
> Kafka topic that can be queried by your app.
>
>
> --
>
> Robin Moffatt | Developer Advocate | robin@confluent.io | @rmoff
>
>
> On Tue, 9 Apr 2019 at 21:26, Nick Torenvliet <natorenvliet@gmail.com>
> wrote:
>
> > Hi all,
> >
> > Just looking for some general guidance.
> >
> > We have a kafka -> druid pipeline we intend to use in an industrial
> setting
> > to monitor process data.
> >
> > Our kafka system recieves messages on a single topic.
> >
> > The messages are {"timestamp": yy:mm:ddThh:mm:ss.mmm,
> "plant_equipment_id":
> > "id_string", "sensorvalue": float}
> >
> > For our POC there are about 2000 unique plant_equipment ids, this will
> > quickly grow to 20,000.
> >
> > The kafka topic streams into druid
> >
> > We are building some node.js/react browser based apps for analytics and
> > real time stream monitoring.
> >
> > We are thinking that for visualizing historical data sets we will hit
> druid
> > for data.
> >
> > For real time streaming we are wondering what our best option is.
> >
> > One option is to just hit druid semi regularly and update the on screen
> > visualization as data arrives from there.
> >
> > Another option is to stream subset of the topics (somehow) from kafka
> using
> > some streams interface.
> >
> > With all the stock ticker apps out there, I have to imagine this is a
> > really common use case.
> >
> > Anyone have any thoughts as to what we are best to do?
> >
> > Nick
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message