kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Create topic, then fetch offset
Date Wed, 14 Jan 2015 17:25:21 GMT
About the misleading error code, it is covered in the following ticket:
KAFKA-1852 <https://issues.apache.org/jira/browse/KAFKA-1852>

On Wed, Jan 14, 2015 at 12:11 AM, Henri Pihkala <henri.pihkala@streamr.com>
wrote:

> Thanks Harsha,
>
> Looks like ZkStringSerializer was the key to creating topics via
> AdminUtils. That’s quite an easy mistake to make. I now noticed that the
> issue is already being addressed at
> https://issues.apache.org/jira/browse/KAFKA-1737 <
> https://issues.apache.org/jira/browse/KAFKA-1737>, great job!
>
> Thanks everyone for your help!
>
> Best regards
> Henri
>
>
> > On 14.1.2015, at 4.26, Harsha <kafka@harsha.io> wrote:
> >
> > Internally producer sends a TopicMetadataRequest which creates the topic
> > if auto.create.topics.enable is true.
> >
> > On Tue, Jan 13, 2015, at 04:49 PM, Harsha wrote:
> >>
> >> You can import ZKStringSerializer from kafka.utils.ZkClient or write
> >> your own similar string serializer like this
> >> https://gist.github.com/harshach/7b5447c39168eb6062e0
> >>
> >> On Tue, Jan 13, 2015, at 04:31 PM, Harsha wrote:
> >>> It looks like you are not adding a ZkStringSerializer to your zkClient.
> >>>
> >>> ZkClient zkClient = new ZkClient(ZK_CONN_STRING);
> >>> in Kafka TopicCommand uses
> >>> "new ZkClient(opts.options.valueOf(opts.zkConnectOpt), 30000, 30000,
> >>> ZKStringSerializer)"
> >>>
> >>> because of this although your topic is getting created and a zookeeper
> >>> path created. Zk path contents are messed up
> >>> get /brokers/topics/test_d5ee7dfe-cf33-4135-9bbd-c1e7d9367af2
> >>> ��t${"version":1,"partitions":{"0":[0]}}
> >>> cZxid = 0x2ea0
> >>> ctime = Tue Jan 13 16:08:19 PST 2015
> >>> mZxid = 0x2ea0
> >>> mtime = Tue Jan 13 16:08:19 PST 2015
> >>>
> >>>
> >>>
> >>> On Tue, Jan 13, 2015, at 02:28 PM, Henri Pihkala wrote:
> >>>> Hi,
> >>>>
> >>>> Thanks for your reply, Guozhang, you are right.
> >>>>
> >>>> I eventually got it working, but it required TWO changes to what I
> >>>> initially tried:
> >>>>
> >>>> 1) Use OffsetRequest, not OffsetFetchRequest, like Guozhang pointed
> out.
> >>>>
> >>>> 2) Create topics by sending a TopicMetadataRequest and having
> >>>> auto.create.topics.enable=true. Initially I tried to create topics
> >>>> explicitly using AdminUtils.createTopic(), but this did not work for
> some
> >>>> reason!
> >>>>
> >>>> Do you think the problem I had in 2) is worth investigating or
> reporting
> >>>> further? The gist below is a test case that passes as is, but if I
> change
> >>>> the topic creation method to AdminUtils.createTopic() the test fails.
> Can
> >>>> anyone explain why?
> >>>>
> >>>> https://gist.github.com/hpihkala/f93ef3dc4c08382df69e
> >>>> <https://gist.github.com/hpihkala/f93ef3dc4c08382df69e>
> >>>>
> >>>> Best regards
> >>>> Henri
> >>>>
> >>>>
> >>>>> On 13.1.2015, at 19.34, Guozhang Wang <wangguoz@gmail.com>
wrote:
> >>>>>
> >>>>> Henri,
> >>>>>
> >>>>> FetchOffsetRequest is used for retrieving the last committed offset
> of a
> >>>>> certain consumer group; for your use case you would send
> OffsetRequest.
> >>>>>
> >>>>> That said the error code is misleading, it may be a bug in not
> indicating
> >>>>> the right error code.
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Fri, Jan 9, 2015 at 4:10 AM, Henri Pihkala <
> henri.pihkala@streamr.com>
> >>>>> wrote:
> >>>>>
> >>>>>> (1) Create topic
> >>>>>> (2) Send a FetchOffsetRequest for a partition in the new topic
> >>>>>>
> >>>>>> Results in error code 3, or UnknownTopicOrPartition.
> >>>>>>
> >>>>>> How can it be unknown, since I just created it? Shouldn’t
it return
> 0, as
> >>>>>> that’s the offset of the next (first) message?
> >>>>>>
> >>>>>> Is this a bug or a feature? Tried with both 0.8.1.1 and 0.8.2-beta
> >>>>>> (single-node cluster).
> >>>>>>
> >>>>>> Best regards
> >>>>>> Henri
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>
>
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message