kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Berryman, Eric" <berry...@frib.msu.edu>
Subject Re: Using Kafka to build a streaming platform for a research facility?
Date Mon, 12 Mar 2018 18:49:32 GMT

The European Spallation Source [1] seems to be using it for this case [2].


I am also using this code [2], but only for visualization in another "data center".



[1] https://europeanspallationsource.se/


[2] https://github.com/ess-dmsc/forward-epics-to-kafka


Thank you!

Eric




________________________________
From: Hannes Petri <hannespetri@gmail.com>
Sent: Monday, March 12, 2018 1:37:29 PM
To: users@kafka.apache.org
Subject: Using Kafka to build a streaming platform for a research facility?

Hi,

I work at a research facility where numerous hi-res detectors produce thousands of GB of data
every day. We want to build a highly flexible and performant streaming platform for storing,
transmitting and routing the data. For example, detector output needs to end up:

1. In permanent storage systems
2. In realtime or semi-realtime visualization software
3. In post-processing and analysis software
4. In metrics software

...and possibly more. Now I'm exploring Kafka as an option to back such a platform.

Would Kafka be a good fit? The reason I'm asking is because among the use cases, I've mostly
seen Kafka being used with more lightweight data, geared towards business events and high
frequency streams of text and scalars. In other words, *more* but *smaller* messages.

In my situation, we'd be looking at low frequency but huge files (typically these detectors
produce one large file at a time). In order not to flood the storage, raw data topics would
need to have a very short retention time (hours to days).

Does anybody have experience in using Kafka in a similar scenario? What are your thoughts
about the situation I describe? Would we benefit from using Kafka?

I highly appreciate any input on this. Many thanks in advance.

Best regards
Hannes

Sent from my iPhone

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message