kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <ao...@wikimedia.org>
Subject Re: Cross-Data-Center Mirroring, and Guaranteed Minimum Time Period on Data
Date Thu, 16 Oct 2014 13:49:29 GMT
Check out Camus.  It was built to do parallel loads from Kafka into time bucketed directories
in HDFS.

On Oct 16, 2014, at 9:32 AM, Gwen Shapira <gshapira@cloudera.com> wrote:

> I assume the messages themselves contain the timestamp?
> If you use Flume, you can configure a Kafka source to pull data from
> Kafka, use an interceptor to pull the date out of your message and
> place it in the event header and then the HDFS sink can write to a
> partition based on the timestamp.
> Gwen
> On Wed, Oct 15, 2014 at 8:47 PM, Jun Rao <junrao@gmail.com> wrote:
>> One way you can do that is to continually load data from Kafka to Hadoop.
>> During load, you put data into different HDFS directories based on the
>> timestamp. The Hadoop admin can decide when to open up those directories
>> for read based on whether data from all data centers have arrived.
>> Thanks,
>> Jun
>> On Tue, Oct 14, 2014 at 11:54 PM, Alex Melville <amelville@g.hmc.edu> wrote:
>>> Hi Apache Community,
>>> My company has the following use case. We have multiple geographically
>>> disparate data centers each with their own Kafka cluster, and we want to
>>> aggregate all of these center's data to one central Kafka cluster located
>>> in a data center distinct from the rest using MirrorMaker. Once in the
>>> central cluster, most of this data will be fed into Hadoop for analytics
>>> purposes. However, with how we have Hadoop working right now, it must wait
>>> until it has received data from all of the other data centers for a
>>> specific time period before it has the green light to load that data into
>>> HDFS and process it. For example, say we have 3 remote (as in not central)
>>> data centers, and DC1 has pushed to the central data center all of its data
>>> up to 4:00 PM, DC2 has pushed everything up to 3:30 PM, and DC2 is lagging
>>> behind and only pushed data up to the 2:00PM time period. Then Hadoop
>>> processes all data tagged with modification times before 2:00PM, and it
>>> must wait until DC3 catches up by pushing 2:15, 2:30, etc. data to the
>>> central cluster before it can process the 3:00 PM data.
>>> So our question is: What is the best way to handle this time-period-ordered
>>> requirement on our data using a distributed messaging log like Kafka? We
>>> originally started using Kafka to move away from a batch-oriented backend
>>> data pipeline transport system in favor of a more streaming-focused system,
>>> but we still need to keep track of the latest common time period of data
>>> streaming in from the remote clusters.
>>> Cheers,
>>> Alex M.

View raw message