kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avi Flax <avi.f...@parkassist.com>
Subject Re: Architecture recommendations for a tricky use case
Date Thu, 29 Sep 2016 20:09:53 GMT

> On Sep 29, 2016, at 09:54, Ali Akhtar <ali.rac200@gmail.com> wrote:
> 
> I'd appreciate some thoughts / suggestions on which of these alternatives I
> should go with (e.g, using raw Kafka consumers vs Spark for ETL, which
> persistent data store to use, and how to query that data store in the
> backend of the web UI, for displaying the reports).

Hi Ali, I’m no expert in any of this, but I’m working on a project that is broadly similar
to yours, and FWIW I’m evaluating Druid as the datastore which would host the queryable
data and, well, actually handle and fulfill queries.

Since Druid has built-in support for streaming ingestion from Kafka topics, I’m tentatively
thinking of doing my ETL in a stream processing topology (I’m using Kafka Streams, FWIW),
which would write the events destined for Druid into certain topics, from which Druid would
ingest those events.

HTH,
Avi

————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/

Mime
View raw message