sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Szabo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2906) Optimization of AvroUtil.toAvroIdentifier
Date Thu, 12 May 2016 17:41:13 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281819#comment-15281819

Attila Szabo commented on SQOOP-2906:

Dear Joeri,

I'd like to run a set of measurements on my own ( to check if your solution + mine on this
issue are synergies or not and to check further improvements )

I found this hprofiler quite exciting however I had difficulties in using it ( my bad ).
Could you help with a the following questions:
- How did you ensure that the extra parameters are passed to the mappers and the Sqoop1 cmd
line tool?
- In case of -c which server/service IP is needed there and in what kind of format?
- Is it enough to use only with -h ?
- If I don't wanna use SSH keys is it a valid solution if I type my root password all the
time when it executes SSH?
- Should it work just out of box ( e.g. defining -h -j -f -t ) or is any postprocess needed?
- Did you use a hadoop distribution like CDH or just pure hdfs and sqoop1?

Many thanks in advance!

> Optimization of AvroUtil.toAvroIdentifier
> -----------------------------------------
>                 Key: SQOOP-2906
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2906
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Joeri Hermans
>            Assignee: Joeri Hermans
>              Labels: avro, hadoop, optimization
>         Attachments: diff.txt
> Hi all
> Our distributed profiler indicated some inefficiencies in the AvroUtil.toAvroIdentifier
method, more specifically, the use of Regex patterns. This can be directly observed from the
FlameGraph generated by this profiler (https://jhermans.web.cern.ch/jhermans/sqoop_avro_flamegraph.svg).
We implemented an optimization, and compared this with the original method. On our testing
machine, the optimization by itself is about 500% (on average) more efficient compared to
the original implementation. We have yet to test how this optimization will influence the
performance of user jobs.
> Any suggestions or remarks are welcome.
> Kind regards,
> Joeri
> https://github.com/apache/sqoop/pull/18
> Writeup:
> https://db-blog.web.cern.ch/blog/joeri-hermans/2016-04-hadoop-performance-troubleshooting-stack-tracing-introduction

This message was sent by Atlassian JIRA

View raw message