sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1532) Sqoop2: Support Sqoop on Spark Execution Engine
Date Mon, 14 Dec 2015 14:09:46 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056011#comment-15056011
] 

ASF GitHub Bot commented on SQOOP-1532:
---------------------------------------

GitHub user eruizgar opened a pull request:

    https://github.com/apache/sqoop/pull/11

    [SQOOP-1532] Support Sqoop on Spark Execution Engine

    We have implemented the Issue SQOOP-1532 to support sqoop jobs to run on spark environment.
You can run on the standalone spark cluster, using sqoop client.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Stratio/sqoop SQOOP-1532

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/sqoop/pull/11.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11
    
----
commit 6fe1428afd5d98ff97a4e26ad19f34467fe6ddfa
Author: aargomaniz@stratio.com <aargomaniz@stratio.com>
Date:   2015-11-17T16:50:33Z

    Added spark dependencies

commit eff93a0cc9428dcf401894649de0f679cf932fc6
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-17T16:56:16Z

    SQOOP-2684: Sqoop2: Upgrade groovy to 2.4.0
    
    (Dian Fu via Jarek Jarcec Cecho)

commit dc011e12b128853ea45a3c6f8a9704b7285ac16f
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-17T17:04:01Z

    SQOOP-2682: Sqoop2: Add test cases for the object name with special char
    
    (Colin Ma via Jarek Jarcec Cecho)

commit 370ea29c2aa1d1fb0814e56f0afd16d4f66e2e41
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-17T17:08:35Z

    SQOOP-2680: Sqoop2: Remove the id from public interface for connection
    
    (Colin Ma via Jarek Jarcec Cecho)

commit dfe984c14aef83338e1ac68972ba7503fcaa6d0f
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-18T16:06:49Z

    SQOOP-2396: Sqoop2: Race condition in purge/update threads on Server shutdown
    
    (Dian Fu via Jarek Jarcec Cecho)

commit 68ca8bc2e1f2ceb8ad0ab14763a146fb0d2682db
Author: Kate Ting <kathleen@apache.org>
Date:   2015-11-18T22:34:17Z

    SQOOP-2688: Sqoop2: Provide utility method to safely retrieve value from JSONObject
    (Jarek Jarcec  Cecho via Kate Ting)

commit cc3e77b89e653a5f33996d53d3fe4fb7839c16a3
Author: Kate Ting <kathleen@apache.org>
Date:   2015-11-18T22:58:57Z

    SQOOP-2694: Sqoop2: Doc: Register structure in sphinx for our docs
    (Jarek Jarcec Cecho via Kate Ting)

commit 4f6ea567ffd9b5f43614c2c2b632789e9c752422
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-19T15:07:07Z

    SQOOP-2700: Sqoop2: Tests in shell module are in infinite loop
    
    (Dian Fu via Jarek Jarcec Cecho)

commit ee64ec6e2fb856a48f18685daf8459b9ad1da083
Author: Jarek Jarcec Cecho <jarcec@apache.org>
Date:   2015-11-20T16:19:01Z

    SQOOP-2699: Sqoop2: Oraoop: Improve Oracle parameters
    
    (David Robson via Jarek Jarcec Cecho)

commit bad653c995df426189c67775748ed83321b6ad54
Author: Kate Ting <kathleen@apache.org>
Date:   2015-11-20T22:26:21Z

    SQOOP-2698: Sqoop2: RESTiliency: Split the InvalidRESTCallsTest into independent test
cases
     (Jarek Jarcec Cecho via Kate Ting)

commit 2c58a54ec871dc05198023ab6e8a3e2afa1d9343
Author: Enrique ruiz <eruiz@stratio.com>
Date:   2015-11-17T16:50:33Z

    [SQOOP-1532] Support Sqoop on Spark Execution Engine

----


> Sqoop2: Support Sqoop on Spark Execution Engine
> -----------------------------------------------
>
>                 Key: SQOOP-1532
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1532
>             Project: Sqoop
>          Issue Type: Improvement
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 2.0.0
>
>
> The current execution engine supported in sqoop is MR.
> The goal if this ticket is to support sqoop jobs ( map only and map+reduce ) to run on
spark environment.
> It should at the minimum support running on the standalone spark cluster and then subsequently
work with YARN/mesos.
> High level goals
> 1. Hook up with the connector apis to provide the basic load/ extract to the spark RDD.
> 2. Implementation of the Sqoop RDD to support extraction from different data sources
. The design proposal will discuss the alternatives on how this can be achieved.
> 3. Optimizing the loading/writing ( re-use/ refactor the consumer thread code to be agnostic
of the hadoop output format)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message