sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SQOOP-2834) Sqoop2: Integration: Limit debug log to only classes that we're interested in
Date Mon, 07 Mar 2016 23:37:40 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jarek Jarcec Cecho resolved SQOOP-2834.
---------------------------------------
    Resolution: Fixed

Resolved as part of SQOOP-2832.

> Sqoop2: Integration: Limit debug log to only classes that we're interested in
> -----------------------------------------------------------------------------
>
>                 Key: SQOOP-2834
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2834
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.99.7
>
>
> To follow up on my recent patches that are helping with "debuggability" of pre-commit
hook and our integration tests in general, I've looked into what exactly are we logging that
size of our logs is more then 1GB per execution.
> Here is what I've done:
> # I've applied my patch from SQOOP-2832 to get log for one test only
> # I've run a magic that gives me classes that are responsible for logging:
> {code}
> cat test/target/surefire-reports/00000_org.apache.sqoop.integration.connector.hdfs.AppendModeTest.test.txt
| sed -re "s/^.*\] ([A-Z]+)[ ]+([A-Za-z.]+) .*$/\1 \2/" | sort | uniq -c | sort -r > report
> {code}
> With a top results being:
> {code}
> 6927 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction
> 5783 DEBUG org.apache.hadoop.ipc.Client
> 5752 DEBUG org.apache.sqoop.repository.common.CommonRepositoryHandler
> 5750 DEBUG org.apache.hadoop.hdfs.DFSClient
> 4784 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode
> 4715 DEBUG org.eclipse.jetty.io.SelectorManager
> 4660 DEBUG org.eclipse.jetty.server.HttpConnection
> 4306 DEBUG org.apache.hadoop.security.UserGroupInformation
> 3489 DEBUG org.eclipse.jetty.io.WriteFlusher
> 2927 DEBUG org.eclipse.jetty.io.ChannelEndPoint
> 2846 DEBUG org.apache.hadoop.conf.Configuration
> 2830 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine
> 2357 DEBUG org.eclipse.jetty.io.AbstractConnection
> 2350 DEBUG org.eclipse.jetty.io.SelectChannelEndPoint
> 2343 DEBUG org.eclipse.jetty.server.HttpChannel
> 2332 DEBUG org.eclipse.jetty.servlet.ServletHandler
> 2309 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction
> 16701 DEBUG org.apache.hadoop.security.SaslInputStream
> 14613 DEBUG org.eclipse.jetty.http.HttpParser
> 1426 
> 1175 DEBUG org.apache.sqoop.security.authorization.DefaultAuthorizationValidator
> 1168 DEBUG org.eclipse.jetty.server.handler.ContextHandler
> 1168 DEBUG org.eclipse.jetty.server.Server
> 1168 DEBUG org.eclipse.jetty.server.HttpChannelState
> 1034 DEBUG org.apache.hadoop.yarn.server.security.ApplicationACLsManager
> 10329 DEBUG org.apache.hadoop.ipc.Server
> {code}
> Based on that I would like to reconfigure certain classes to limit their logging to levels
higher then {{DEBUG}} - jetty seems as a no-brainer and Hadoop ipc might be another good candidate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message