samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Mangi <r...@chartbeat.com>
Subject Re: Best way to log from inside a Samza task?
Date Wed, 01 Jul 2015 19:03:20 GMT
Hi Jason,

It depends on how you’re deploying. I deploy on yarn via cloudera and my logging statements
wind up in the container log for each node.

You need to instantiate a logger for your class before writing to it. Look at the log4j docs
to see the right way to do that.

Rick


> On Jul 1, 2015, at 2:58 PM, jason@marketingscience.co wrote:
> 
> Rick, 
> 
> From your example, what is the correct way to call the logger in the task? Something
like:
> 
> import java.util.Map;
> import org.apache.samza.system.IncomingMessageEnvelope;
> import org.apache.samza.system.OutgoingMessageEnvelope;
> import org.apache.samza.system.SystemStream;
> import org.apache.samza.task.MessageCollector;
> import org.apache.samza.task.StreamTask;
> import org.apache.samza.task.TaskCoordinator;
> 
> public class uimodPageloadStreamTask implements StreamTask {
>   private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”, “kafka-output-test”);
> 
>   @Override
> public void process(IncomingMessageEnvelope envelope,
>       MessageCollector collector,
>       TaskCoordinator coordinator) {
>           logger.info(envelope.getMessage());
>          collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, text));
>       }
> }
> 
> Also, in what log file would the log messages end up? /logs/stdout for that application
container?
> 
> I got System.out.println() to work similarly but it seems to be doing some formatting
of the messages. 
> 
> I am trying to consume messages from one Kafka topic and produce to another. While I
can use zookeeper to see the messages in the originating topic they never make it to the destination
and I am trying to find out why. 
> 
> Thanks for your help,
> 
> Jason
> 
> 
> On Friday, Jun 26, 2558 at 12:03, Rick Mangi <rick@chartbeat.com <mailto:rick@chartbeat.com>>,
wrote:
> If you do something like this in your log4j.xml 
> 
> <root> 
> <priority value="INFO" /> 
> <appender-ref ref="RollingAppender" /> 
> </root> 
> 
> <logger name="cbsamza" additivity="false"> 
> <level value=“DEBUG" /> 
> <appender-ref ref="RollingAppender" /> 
> </logger> 
> 
> the root controls samza’s logging and the logger controls your own… I haven’t managed
to get the imx configuration working yet. 
> 
> 
> 
> 
> > On Jun 26, 2015, at 1:50 PM, jason@marketingscience.co wrote: 
> > 
> > I was almost there. Got it now. Thanks for your help Rick. 
> > 
> > 
> > 
> > 
> > Cheers, 
> > 
> > 
> > 
> > 
> > Jason 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > On Friday, Jun 26, 2558 at 11:43, Rick Mangi <rick@chartbeat.com>, wrote:

> > Hey Jason, 
> > 
> > 
> > If you configure log4j as described here: http://samza.apache.org/learn/documentation/0.9/jobs/logging.html
<http://samza.apache.org/learn/documentation/0.9/jobs/logging.html> 
> > 
> > 
> > Your log statements will wind up in the samza-container logs which you can get to
via the application master gui. 
> > 
> > 
> > hth, 
> > 
> > 
> > Rick 
> > 
> > 
> > 
> >> On Jun 26, 2015, at 1:39 PM, jason@marketingscience.co wrote: 
> > 
> >> 
> > 
> >> Hello, 
> > 
> >> 
> > 
> >> 
> > 
> >> I am working on a basic Samza task that pulls from one Kafka topic and writes
to another. This task runs in Yarn but the Output topic does not contain any data. 
> > 
> >> 
> > 
> >> 
> > 
> >> In order to troubleshoot this more effectively I would like to log the incoming
message as my example below. Ideally, I would like to be able to see the log messages in Yarn,
maybe in the .out files in the /logs directory. 
> > 
> >> 
> > 
> >> 
> > 
> >> Any advice is appreciated. 
> > 
> >> 
> > 
> >> 
> > 
> >> 
> > 
> >> Here is my task: 
> > 
> >> 
> > 
> >> 
> > 
> >> package com.project.samza.tasks; 
> > 
> >> 
> > 
> >> 
> > 
> >> import java.util.Map; 
> > 
> >> import org.apache.samza.system.IncomingMessageEnvelope; 
> > 
> >> import org.apache.samza.system.OutgoingMessageEnvelope; 
> > 
> >> import org.apache.samza.system.SystemStream; 
> > 
> >> import org.apache.samza.task.MessageCollector; 
> > 
> >> import org.apache.samza.task.StreamTask; 
> > 
> >> import org.apache.samza.task.TaskCoordinator; 
> > 
> >> 
> > 
> >> 
> > 
> >> public class exampleStreamTask implements StreamTask { 
> > 
> >> private static final SystemStream OUTPUT_STREAM = new SystemStream(“kafka”,
“new-topic-test”); 
> > 
> >> 
> > 
> >> 
> > 
> >> @Override 
> > 
> >> public void process(IncomingMessageEnvelope envelope, 
> > 
> >> MessageCollector collector, 
> > 
> >> TaskCoordinator coordinator) { 
> > 
> >> String msg = (String) envelope.getMessage(); 
> > 
> >> System.out.println(msg); 
> > 
> >> collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, msg)); 
> > 
> >> } 
> > 
> >> } 
> > 
> >> 
> > 
> >> 
> > 
> >> Thanks, 
> > 
> >> 
> > 
> >> 
> > 
> >> Jason 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message