sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2906) Optimization of AvroUtil.toAvroIdentifier
Date Thu, 19 May 2016 14:31:13 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291172#comment-15291172

ASF subversion and git services commented on SQOOP-2906:

Commit 5779aec031d4bb0bccd79329923e5dbad3786280 in sqoop's branch refs/heads/trunk from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5779aec ]

SQOOP-2906: Optimization of AvroUtil.toAvroIdentifier

(Joeri Hermans via Jarek Jarcec Cecho)

> Optimization of AvroUtil.toAvroIdentifier
> -----------------------------------------
>                 Key: SQOOP-2906
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2906
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Joeri Hermans
>            Assignee: Joeri Hermans
>              Labels: avro, hadoop, optimization
>         Attachments: diff.txt
> Hi all
> Our distributed profiler indicated some inefficiencies in the AvroUtil.toAvroIdentifier
method, more specifically, the use of Regex patterns. This can be directly observed from the
FlameGraph generated by this profiler (https://jhermans.web.cern.ch/jhermans/sqoop_avro_flamegraph.svg).
We implemented an optimization, and compared this with the original method. On our testing
machine, the optimization by itself is about 500% (on average) more efficient compared to
the original implementation. We have yet to test how this optimization will influence the
performance of user jobs.
> Any suggestions or remarks are welcome.
> Kind regards,
> Joeri
> https://github.com/apache/sqoop/pull/18
> Writeup:
> https://db-blog.web.cern.ch/blog/joeri-hermans/2016-04-hadoop-performance-troubleshooting-stack-tracing-introduction

This message was sent by Atlassian JIRA

View raw message