beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pei He (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version
Date Fri, 24 Jun 2016 21:20:17 GMT
Pei He created BEAM-375:
---------------------------

             Summary: HadoopIO and runners-spark conflict with hadoop.version
                 Key: BEAM-375
                 URL: https://issues.apache.org/jira/browse/BEAM-375
             Project: Beam
          Issue Type: Bug
            Reporter: Pei He
            Assignee: Pei He


HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, hadoop-common.

>From [~amitsela]
"Spark can be built against different hadoop versions, but the release in maven central is
a 2.2.0 build (latest). ''

For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check if it will work
with 2.2.0.

I am creating this issue, since I think it there is a general question.
In principle, HadoopIO and other sdks Sources should work with any runners. But, when one
set of runners require version A, but the other set of runners require version B. We will
need a solution for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message