sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2906) Optimization of AvroUtil.toAvroIdentifier
Date Thu, 19 May 2016 14:31:13 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291171#comment-15291171
] 

ASF subversion and git services commented on SQOOP-2906:
--------------------------------------------------------

Commit ac217a032c0755edac713ff53b3f55b7f2f46706 in sqoop's branch refs/heads/trunk from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=ac217a0 ]

Revert "SQOOP-2920: sqoop performance deteriorates significantly on wide datasets; sqoop 100%
on cpu"

I've mistakenly committed SQOOP-2920 and SQOOP-2906 inside this commit,
so I'll revert it and commit them separately.


> Optimization of AvroUtil.toAvroIdentifier
> -----------------------------------------
>
>                 Key: SQOOP-2906
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2906
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Joeri Hermans
>            Assignee: Joeri Hermans
>              Labels: avro, hadoop, optimization
>         Attachments: diff.txt
>
>
> Hi all
> Our distributed profiler indicated some inefficiencies in the AvroUtil.toAvroIdentifier
method, more specifically, the use of Regex patterns. This can be directly observed from the
FlameGraph generated by this profiler (https://jhermans.web.cern.ch/jhermans/sqoop_avro_flamegraph.svg).
We implemented an optimization, and compared this with the original method. On our testing
machine, the optimization by itself is about 500% (on average) more efficient compared to
the original implementation. We have yet to test how this optimization will influence the
performance of user jobs.
> Any suggestions or remarks are welcome.
> Kind regards,
> Joeri
> https://github.com/apache/sqoop/pull/18
> Writeup:
> https://db-blog.web.cern.ch/blog/joeri-hermans/2016-04-hadoop-performance-troubleshooting-stack-tracing-introduction



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message