From users-return-2964-apmail-kafka-users-archive=kafka.apache.org@kafka.apache.org Wed Dec 19 16:48:59 2012 Return-Path: X-Original-To: apmail-kafka-users-archive@www.apache.org Delivered-To: apmail-kafka-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08D84E78A for ; Wed, 19 Dec 2012 16:48:59 +0000 (UTC) Received: (qmail 16441 invoked by uid 500); 19 Dec 2012 16:48:58 -0000 Delivered-To: apmail-kafka-users-archive@kafka.apache.org Received: (qmail 16341 invoked by uid 500); 19 Dec 2012 16:48:58 -0000 Mailing-List: contact users-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@kafka.apache.org Delivered-To: mailing list users@kafka.apache.org Received: (qmail 16332 invoked by uid 99); 19 Dec 2012 16:48:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 16:48:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of junrao@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 16:48:52 +0000 Received: by mail-ie0-f176.google.com with SMTP id 13so3076952iea.35 for ; Wed, 19 Dec 2012 08:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wJ238/aHbCo0eJjZK74J+f/lKrJ5MHH3wsVSHNodenA=; b=nKIqni1QF5qRobWJeeEXkiJk6zAq0gYzzAwitT5xYn1gxIaiAkWusC9E3vzBP25cgS DjKzM4qBJ9TCkgcJla1cEkajUrRmBT4dZei5iZqeMg6m6eLflatjP1pyNSn1DuN+zuUc 7EN8MIatI4o5K2K9Khtp97S3gk3IjMeYU+25wNDGpsrMH4yTe+87by2AevQCYz0OnIoE Z+bYCYYawPCtmUj0fFSBOFiyVMZspC5P4IldrsSSOaOFbQgNCUlLx0TM8UJmi5q6lecr 46nri53579rdOpPLRytDf3+ZVh9GExVjSciJFI9wwTfYI4BFcgfz6tmFKtvE0vRn0oMu GcvQ== MIME-Version: 1.0 Received: by 10.50.76.169 with SMTP id l9mr2767829igw.78.1355935711632; Wed, 19 Dec 2012 08:48:31 -0800 (PST) Received: by 10.43.124.129 with HTTP; Wed, 19 Dec 2012 08:48:31 -0800 (PST) In-Reply-To: References: Date: Wed, 19 Dec 2012 08:48:31 -0800 Message-ID: Subject: Re: partition and consumer stream number From: Jun Rao To: Yonghui Zhao Cc: "users@kafka.apache.org" Content-Type: multipart/alternative; boundary=e89a8f3ba0dd6e1cea04d13762ed X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3ba0dd6e1cea04d13762ed Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In your example, yes, all partitions go to one stream. In general, each stream maps 1 or more partitions. Thanks, Jun On Wed, Dec 19, 2012 at 8:32 AM, Yonghui Zhao wrote= : > Thanks Jun, I didn't get you previous answer. > I think I didn't state my question clearly. > > I know the number of the consumer in one consumer group should be less or > equal than partition number. > But my question is about the relation between the stream number in one > consumer and partition number subscribed by this consumer. > > For example this piece of code create only one stream in the consumer, so > all partitions data will go into one stream? > > Map topicCountMap =3D new HashMap()= ; > topicCountMap.put(topic, new Integer(1)); > Map>> consumerMap =3D consum= er.createMessageStreams(topicCountMap); > KafkaStream stream =3D consumerMap.get(topic).get(0)= ; > ConsumerIterator it =3D stream.iterator(); > > > > Sent from my iPad > > =E5=9C=A8 2012-12-19=EF=BC=8C23:49=EF=BC=8CJun Rao =E5= =86=99=E9=81=93=EF=BC=9A > > Did you get my previous answer? I am including it below. > > Currently, partition is the smallest unit that we distribute data among > consumers (in the same consumer group). So, if the # of consumers is larg= er > than the total number of partitions in a Kafka cluster (across all > brokers), some consumers will never get any data. > > Thanks, > > Jun > > On Wed, Dec 19, 2012 at 3:07 AM, =E6=B0=B8=E8=BE=89 =E8=B5=B5 wrote: > >> >> Hi all, >> >> In consumer we use a topic count map to set how many stream we want to >> split >> the topic. >> What's the relation between kafka partition number and consumer stream >> number? >> Seems stream number should be less or equal to partition number? >> Because if we set stream number bigger than partition number then some >> stream is empty or some partition is split. >> >> Thanks, >> Yonghui >> >> >> >> > --e89a8f3ba0dd6e1cea04d13762ed--