chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Deadlock happening in ChukwaDailyRollingFileAppender
Date Mon, 25 Jun 2012 02:55:23 GMT
Hi Abhijit,

SocketAppender sends log entries as a full entry.  Chunk is created
per log entry (can be multiple lines).  There is no need to do
additional escaping.  SocketAdaptor should be configured in
initial_adaptors like:

add SocketAdaptor JobSummary 9098 0

This means to tag the log file collected through port 9098 as
JobSummary log file.
Hope this helps.

regards,
Eric

On Sun, Jun 24, 2012 at 6:27 PM, Abhijit Dhar <abhijit.dhar.in@gmail.com> wrote:
> Does SocketAppender ecsape newlines? Like, log an exception with a stack
> trace(multiple lines) as a single log statement? And can it register an
> adapter automatically?
>
> On Jun 24, 2012 12:20 PM, "Eric Yang" <eric818@gmail.com> wrote:
>>
>> Hi Abhijit,
>>
>> The dead lock happens when there are too many registrations with the
>> agent and agent has exceeded the open file descriptor limit.  I would
>> recommend to use Log4J SocketAppender with combination of
>> SocketAdaptor.  It reduces the overhead of file offset tracking and
>> reduces the number of file descriptors used by the agent.  It works
>> better than using ChukwaDailyFileAppender and file adaptor
>> combination.
>>
>> regards,
>> Eric
>>
>> On Fri, Jun 22, 2012 at 9:50 AM, Abhijit Dhar <abhijit.dhar.in@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am getting a deadlock in ChukwaDailyRollingFileAppender. The thread
>> > dump
>> > is:
>> >
>> > Found one Java-level deadlock:
>> > =============================
>> > "Thread-4":
>> >  waiting to lock monitor 0x00007f1970004818 (object 0x00000000cb6669e8,
>> > a
>> > java.lang.Class),
>> >  which is held by "Thread for syncLogs"
>> > "Thread for syncLogs":
>> >  waiting to lock monitor 0x00007f1970004ea8 (object 0x00000000d65c44b8,
>> > a
>> > org.apache.log4j.Logger),
>> >  which is held by "main"
>> > "main":
>> >  waiting to lock monitor 0x00007f1970003d98 (object 0x00000000d65c3e80,
>> > a
>> > org.apache.log4j.Logger),
>> >  which is held by "communication thread"
>> > "communication thread":
>> >  waiting to lock monitor 0x0000000041eba250 (object 0x00000000d6736ec0,
>> > a
>> > java.lang.Object),
>> >  which is held by "main"
>> >
>> > Java stack information for the threads listed above:
>> > ===================================================
>> > "Thread-4":
>> >        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:213)
>> >        - waiting to lock <0x00000000cb6669e8> (a java.lang.Class for
>> > org.apache.hadoop.mapred.TaskLog)
>> >        at org.apache.hadoop.mapred.Child$1.run(Child.java:75)
>> > "Thread for syncLogs":
>> >        at org.apache.log4j.Category.getAllAppenders(Category.java:409)
>> >        - waiting to lock <0x00000000d65c44b8> (a
>> > org.apache.log4j.Logger)
>> >        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:218)
>> >        - locked <0x00000000cb6669e8> (a java.lang.Class for
>> > org.apache.hadoop.mapred.TaskLog)
>> >        at org.apache.hadoop.mapred.Child$2.run(Child.java:89)
>> > "main":
>> >        at org.apache.log4j.Category.callAppenders(Category.java:202)
>> >        - waiting to lock <0x00000000d65c3e80> (a
>> > org.apache.log4j.Logger)
>> >        at org.apache.log4j.Category.forcedLog(Category.java:388)
>> >        at org.apache.log4j.Category.warn(Category.java:1008)
>> >        at
>> >
>> > org.apache.hadoop.chukwa.datacollection.controller.ChukwaAgentController.addByName(ChukwaAgentController.java:267)
>> >        at
>> >
>> > org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.subAppend(ChukwaDailyRollingFileAppender.java:565)
>> >        - locked <0x00000000d6736ec0> (a java.lang.Object)
>> >        at
>> > org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
>> >        at
>> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
>> >        - locked <0x00000000d65b38c0> (a
>> > com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
>> >        at
>> >
>> > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
>> >        at org.apache.log4j.Category.callAppenders(Category.java:203)
>> >        - locked <0x00000000d65c44b8> (a org.apache.log4j.Logger)
>> >        at org.apache.log4j.Category.forcedLog(Category.java:388)
>> >        at org.apache.log4j.Category.log(Category.java:853)
>> >        at
>> > org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
>> >        at
>> >
>> > com.glassdoor.util.netsuite.service.NetSuiteServiceImpl.init(NetSuiteServiceImpl.java:86)
>> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >        at
>> >
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >        at
>> >
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >        at java.lang.reflect.Method.invoke(Method.java:597)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1378)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1339)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1299)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:463)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:404)
>> >        at java.security.AccessController.doPrivileged(Native Method)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:375)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:263)
>> >        at
>> >
>> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:170)
>> >        - locked <0x00000000f77044f0> (a
>> > java.util.concurrent.ConcurrentHashMap)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:260)
>> >                at
>> >
>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:184)
>> >        at
>> >
>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:163)
>> >        at
>> >
>> > org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:430)
>> >        - locked <0x00000000f7704520> (a
>> > java.util.concurrent.ConcurrentHashMap)
>> >        at
>> >
>> > org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
>> >        at
>> >
>> > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
>> >        - locked <0x00000000f77045d8> (a java.lang.Object)
>> >        at
>> >
>> > org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
>> >        at
>> >
>> > org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
>> >        at
>> >
>> > com.glassdoor.util.hadoop.entity.GDHadoopConfiguration.getAppContext(GDHadoopConfiguration.java:74)
>> >        - locked <0x00000000d6736ef0> (a java.lang.Object)
>> >        at
>> >
>> > com.glassdoor.util.hadoop.entity.SpringHadoopMapperWrapper.run(SpringHadoopMapperWrapper.java:116)
>> >        at
>> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> > "communication thread":
>> >        at
>> >
>> > org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.checkEntryConditions(ChukwaDailyRollingFileAppender.java:489)
>> >        - waiting to lock <0x00000000d6736ec0> (a java.lang.Object)
>> >        at
>> > org.apache.log4j.WriterAppender.append(WriterAppender.java:156)
>> >        at
>> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
>> >        - locked <0x00000000d65b2c50> (a
>> > com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
>> >        at
>> >
>> > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
>> >        at org.apache.log4j.Category.callAppenders(Category.java:203)
>> >        - locked <0x00000000d65c3e80> (a org.apache.log4j.Logger)
>> >        at org.apache.log4j.Category.forcedLog(Category.java:388)
>> >        at org.apache.log4j.Category.log(Category.java:853)
>> >        at
>> > org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
>> >        at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:554)
>> >        at java.lang.Thread.run(Thread.java:662)
>> >
>> > Found 1 deadlock.
>> >
>> >
>> >
>> > I took a look at the code and surely, it is locking on chukwaLock object
>> > in
>> > checkEntryConditions() and subAppend() methods. I'm not too sure if we
>> > need
>> > to lock on chukwaLock for checkEntryConditions().
>> > Anybody has a clue of what is going on ? This deadlock has happened
>> > twice
>> > now so far.
>> >
>> > Thanks,
>> > Abhijit
>> >
>> > --
>> > View this message in context:
>> > http://apache-chukwa.679492.n3.nabble.com/Deadlock-happening-in-ChukwaDailyRollingFileAppender-tp4025029.html
>> > Sent from the Chukwa - Users mailing list archive at Nabble.com.

Mime
View raw message