samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash W Matheson <ash.mathe...@gmail.com>
Subject Re: New to Samza/Yarn and having Kafka issues
Date Sun, 22 Mar 2015 18:37:59 GMT
Sure - I'll do that in a bit and send it up to pastebin.

On Sun, Mar 22, 2015 at 11:35 AM, Chinmay Soman <chinmay.cerebro@gmail.com>
wrote:

> Can you please enable debug level logging and paste the log?
>
> On Sun, Mar 22, 2015 at 11:28 AM, Ash W Matheson <ash.matheson@gmail.com>
> wrote:
>
> > No, it's behind some corporate stuff - I just redacted it so I could
> share
> > it up.
> >
> > On Sun, Mar 22, 2015 at 11:17 AM, Chinmay Soman <
> chinmay.cerebro@gmail.com
> > >
> > wrote:
> >
> > > Just for sanity check, is the broker host 'redacted:9092' or '
> > > redactedec:9092'.
> > >
> > > Just wanted to rule out any typos. Are the 2 above hosts the same ?
> > >
> > > On Sun, Mar 22, 2015 at 11:08 AM, Ash W Matheson <
> ash.matheson@gmail.com
> > >
> > > wrote:
> > >
> > > > Also, here's the producer: http://pastebin.com/qMNJabTZ
> > > >
> > > >
> > > > On Sun, Mar 22, 2015 at 10:57 AM, Ash W Matheson <
> > ash.matheson@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Yep, first thing I checked (got bitten by that earlier in the week
> > with
> > > > no
> > > > > data actually in the topic).
> > > > >
> > > > > On Sun, Mar 22, 2015 at 10:56 AM, Chinmay Soman <
> > > > chinmay.cerebro@gmail.com
> > > > > > wrote:
> > > > >
> > > > >> Can you double check that you can read data from your Kafka
> broker ?
> > > > >>
> > > > >> > ./deploy/kafka/bin/kafka-topics.sh --describe --zookeeper
> > > > localhost:2181
> > > > >> --topic myTopic
> > > > >> > ./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper
> > > > localhost:2181
> > > > >> --topic myTopic --from-beginning
> > > > >>
> > > > >> I've seen cases where if the Kafka broker isn't shutdown properly,
> > > > >> something like this happens.
> > > > >>
> > > > >> On Sun, Mar 22, 2015 at 10:35 AM, Ash W Matheson <
> > > > ash.matheson@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hey all,
> > > > >> >
> > > > >> > Evaluating Samza currently and am running into some odd
issues.
> > > > >> >
> > > > >> > I'm currently working off the 'hello-samza' repo and trying
to
> > > parse a
> > > > >> > simple kafka topic that I've produced through an extenal
java
> app
> > > > >> (nothing
> > > > >> > other than a series of sentences) and it's failing pretty
hard
> for
> > > me.
> > > > >> The
> > > > >> > base 'hello-samza' set of apps works fine, but as soon as
I
> change
> > > the
> > > > >> > configuration to look at a different Kafka/zookeeper I get
the
> > > > >> following in
> > > > >> > the userlogs:
> > > > >> >
> > > > >> > 2015-03-22 17:07:09 KafkaSystemAdmin [WARN] Unable to fetch
last
> > > > offsets
> > > > >> > for streams [myTopic] due to kafka.common.KafkaException:
> fetching
> > > > topic
> > > > >> > metadata for topics [Set(myTopic)] from broker
> > > > >> > [ArrayBuffer(id:0,host:redacted,port:9092)] failed. Retrying.
> > > > >> >
> > > > >> >
> > > > >> > The modifications are pretty straightforward.  In the
> > > > >> > Wikipedia-parser.properties, I've changed the following:
> > > > >> > task.inputs=kafka.myTopic
> > > > >> > systems.kafka.consumer.zookeeper.connect=redacted:2181/
> > > > >> > systems.kafka.consumer.auto.offset.reset=smallest
> > > > >> > systems.kafka.producer.metadata.broker.list=redacted:9092
> > > > >> >
> > > > >> > and in the actual java file WikipediaParserStreamTask.java
> > > > >> >   public void process(IncomingMessageEnvelope envelope,
> > > > MessageCollector
> > > > >> > collector, TaskCoordinator coordinator) {
> > > > >> >     Map<String, Object> jsonObject = (Map<String,
Object>)
> > > > >> > envelope.getMessage();
> > > > >> >     WikipediaFeedEvent event = new
> WikipediaFeedEvent(jsonObject);
> > > > >> >
> > > > >> >     try {
> > > > >> >         System.out.println(event.getRawEvent());
> > > > >> >
> > > > >> > And then following the compile/extract/run process outlined
in
> the
> > > > >> > hello-samza website.
> > > > >> >
> > > > >> > Any thoughts?  I've looked online for any 'super simple'
> examples
> > of
> > > > >> > ingesting kafka in samza with very little success.
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Thanks and regards
> > > > >>
> > > > >> Chinmay Soman
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and regards
> > >
> > > Chinmay Soman
> > >
> >
>
>
>
> --
> Thanks and regards
>
> Chinmay Soman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message