From users-return-7580-apmail-kafka-users-archive=kafka.apache.org@kafka.apache.org Fri Dec 6 15:05:23 2013 Return-Path: X-Original-To: apmail-kafka-users-archive@www.apache.org Delivered-To: apmail-kafka-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DCD4910236 for ; Fri, 6 Dec 2013 15:05:23 +0000 (UTC) Received: (qmail 64611 invoked by uid 500); 6 Dec 2013 15:05:23 -0000 Delivered-To: apmail-kafka-users-archive@kafka.apache.org Received: (qmail 64565 invoked by uid 500); 6 Dec 2013 15:05:22 -0000 Mailing-List: contact users-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@kafka.apache.org Delivered-To: mailing list users@kafka.apache.org Received: (qmail 64551 invoked by uid 99); 6 Dec 2013 15:05:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Dec 2013 15:05:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of otis.gospodnetic@gmail.com designates 209.85.216.177 as permitted sender) Received: from [209.85.216.177] (HELO mail-qc0-f177.google.com) (209.85.216.177) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Dec 2013 15:05:15 +0000 Received: by mail-qc0-f177.google.com with SMTP id m20so528900qcx.8 for ; Fri, 06 Dec 2013 07:04:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wI0aRjLl5yZixZl2pf9KF7/BVruw1loTj5F/i4L1XgU=; b=MSiysOfk8dvRBYF1B3b8Vv1ubpYv+ArnBqUCkWwlS2S05E84B4qmYLh1DXgs9nSJCk E22AXQCdCwjmS621s6cxpYXEYao773a2IquopsnWjd2FFzSOFpTjQVu5I64yuE3jF+LG vjIR3Ox3r/SWqCb0SvZC9BRVAKPKbdsvuF5hJ/17J8yL09kstPpk7MGKfFBcpQPDAFvO sv81ckttfxXAVRTFXrMUJrsApojC82tFbHuYRrQNzUuhTdMA/uomN9UDLfnGEc6kWTqc 1xv4WxHze+8I04ORHaoMPA/ijjsvjd8K5B896OQ0VR7krpxNa5EnlJGexv9CAfboRndX mDFw== MIME-Version: 1.0 X-Received: by 10.49.26.129 with SMTP id l1mr56552947qeg.37.1386342294044; Fri, 06 Dec 2013 07:04:54 -0800 (PST) Received: by 10.224.103.72 with HTTP; Fri, 6 Dec 2013 07:04:54 -0800 (PST) In-Reply-To: References: Date: Fri, 6 Dec 2013 10:04:54 -0500 Message-ID: Subject: Re: Consuming "backwards"? From: Otis Gospodnetic To: users@kafka.apache.org Content-Type: multipart/alternative; boundary=047d7b676b54f9abcb04ecdef73c X-Virus-Checked: Checked by ClamAV on apache.org --047d7b676b54f9abcb04ecdef73c Content-Type: text/plain; charset=ISO-8859-1 Hi, On Fri, Dec 6, 2013 at 9:38 AM, Tom Brown wrote: > Do you mean you want to start from the most recent data and go backwards to > the oldest data, or that you want to start with old data and consume > forwards? > Forwards is the "normal way". I'm looking for the "abnormal way", of course ;) i.e. backwards. If the following are the messages that came in, oldest to newest: M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Then I'd love to be able to consume from the end, say in batches of 3, like this: get last 3: M10 M11 M12 get last 3: M7 M8 M9 get last 3: M4 M5 M6 get last 3: M1 M2 M3 Of course, if messages keep coming in, then the new ones that arrive would get picked up first and, eventually, assuming Consumer can consume faster than messages are produced, all messages will get consumed. But the important/key part is that any new ones that arrive will get picked up first. If the former, it would be difficult or impossible in 0.7.x, but I think > doable in 0.8.x. (They added some sort of message index). If the latter, > that is easily accomplished in both versions. > I'd love to know if that's really so and how to do it! We are looking to move to Kafka 0.8 in January and to add performance monitoring for Kafka 0.8 to SPM (see http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/) Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ > On Friday, December 6, 2013, Otis Gospodnetic wrote: > > > Hi, > > > > Does Kafka offer a way to consume messages in batches, but "from the > end"? > > > > This would be valuable to have in all systems where the most recent data > is > > a lot more important than older data, such as performance metrics, and > > maybe even logs....maybe also trading/financial data, and such. > > > > Any chance of this sort of functionality ever making it into Kafka, or is > > this simply not implementable due to some underlying assumptions, or data > > structures, or ... ? > > > > Thanks, > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > --047d7b676b54f9abcb04ecdef73c--