mrunit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik K <oss....@gmail.com>
Subject Re: MRUNIT-56 - 0.8.0 release does not work with Hadoop 0.23
Date Tue, 14 Feb 2012 23:46:16 GMT
On Tue, Feb 14, 2012 at 2:30 PM, Brock Noland <brock@cloudera.com> wrote:

> Hi,
>
> First, thank you very much for you help!
>
> Second, with that change, when someone has a dependent project, they
> still have to specify the version of hadoop.
>
> mvn -Dhadoopversion=23 test
>
> Example project: http://people.apache.org/~brock/mrunit-test-verson.zip
>
> so that the mrunit pom does not include the wrong version of hadoop.
> I assume that this is related to Tom Whites JIRA:
>
> https://issues.apache.org/jira/browse/MRUNIT-55
>
> But I am not sure how we resolve that issue.
>


It depends on - how we define 'new versions'.   Are they api-compatible
versions with one of the branches supported ( hadoop20, hadoop100, hadoop23
) or completely brand-new altogether.


If they are api compatible, (say hadoop 1.0.1 ) then we can always override
the dependencies that comes with mrunit transitive dependencies.

<dependencies>
    <dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.8.0-incubating</version>
*        <classifier>hadoop100</version>*
*        <scope>test</scope>*
 </dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
* <version>1.0.1</version>  (or <version>1.1.0</version>, say)*
</dependency>

</dependencies>

For a completely new version tree though (say 0.24.x ) , we would need to
add a new platform classifier to mrunit though. ( add conditional source
directories etc. if necessary that is ).

--
  Karthik.






> Cheers,
> Brock
>
> On Tue, Feb 14, 2012 at 1:36 PM, Karthik K <oss.akk@gmail.com> wrote:
> > On Tue, Feb 14, 2012 at 11:07 AM, Brock Noland <brock@cloudera.com>
> wrote:
> >
> >> Hi,
> >>
> >> Hmm, I added the -D options earlier but clearly screwed something up.
> >> That works in the three jars get published:
> >>
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachemrunit-230/org/apache/mrunit/mrunit/0.8.0-incubating/
> >>
> >> Given that we have three jars with the same groupId and artifactId is
> >> there any risk of accidentally including more than one of the jars in
> >> the classpath?
> >>
> >>
> > There is also a classifier ( suffix name to the jar files ) to
> distinguish
> > between the same.
> >
> > So - this must be ok, for maven and non-maven people to pick the right
> jar.
> >
> > Maven people would use the following to import, what they want:
> >
> > <dependency>
> > <groupId>org.apache.mrunit</groupId>
> > <artifactId>mrunit</artifactId>
> > <version>0.8.0-incubating</version>
> > *        <classifier>hadoop20</classifier>*
> > </dependency>
> >
> > classifier: (other values: hadoop100 or hadoop23, as appropriate ).
> >
> > Non-maven people would just download the jar directly and incorporate in
> > their platforms.
> >
> > So - I guess we are good here.
> >
> >
> >
> >> Again, showing my maven noob status.
> >>
> >> Cheers!
> >> Brock
> >>
> >> On Tue, Feb 14, 2012 at 12:35 PM, Karthik K <oss.akk@gmail.com> wrote:
> >> > Thanks for sharing the command.
> >> >
> >> > That activates the default profile ( hadoop 0.20) and hence, deploys
> with
> >> > the default classifier ( hadoop20) .
> >> >
> >> > # mvn clean assembly:assembly verify
> >> > #mvn clean deploy -Pdeploy -Pjavadoc
> >> >  (deploys hadoop20 classifier artifacts, by default)
> >> >
> >> > To do for other hadoop platforms, though, we need to activate their
> >> > respective profiles
> >> >
> >> > # mvn  *-Dhadoopversion=100 *clean assembly:assembly verify
> >> > #mvn *-Dhadoopversion=100* clean deploy -Pdeploy -Pjavadoc
> >> >  (would deploy hadoop100 classifier, artifacts)
> >> >
> >> >
> >> > #mvn *-Dhadoopversion=23 *clean assembly:assembly verify
> >> > #mvn *-Dhadoopversion=23* clean deploy -Pdeploy -Pjavadoc
> >> >  (would deploy hadoop23 classifier, artifacts)
> >> >
> >> >
> >> >
> >> >
> >> > This would get mrunit with 3 different classifers for the different
> >> hadoop
> >> > platforms.
> >> >
> >> > Would that help ?
> >> >
> >> >
> >> >
> >> > On Tue, Feb 14, 2012 at 10:22 AM, Brock Noland <brock@cloudera.com>
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Yep!
> >> >>
> >> >> mvn clean assembly:assembly verify
> >> >> mvn clean deploy -Pdeploy -Pjavadoc
> >> >>
> >> >> From
> https://cwiki.apache.org/confluence/display/MRUNIT/How+to+Release
> >> >>
> >> >> Brock
> >> >>
> >> >> On Tue, Feb 14, 2012 at 12:12 PM, Karthik K <oss.akk@gmail.com>
> wrote:
> >> >> > Can you help with the commands used for the purpose ?
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Feb 14, 2012 at 7:02 AM, Brock Noland <brock@cloudera.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> On Sun, Feb 12, 2012 at 1:17 PM, Karthik K <oss.akk@gmail.com>
> >> wrote:
> >> >> >> > I have attached a patch to MRUNIT-57 with the configuration
of
> >> >> >> classifiers.
> >> >> >> >
> >> >> >> > May be, you can apply it locally and deploy to your apache
home
> >> >> directory
> >> >> >> > patch to see if that works ?
> >> >> >>
> >> >> >> I tried that for both 0.20 and 1.0, but the maven repo only
has
> the
> >> 0.20
> >> >> >> jar:
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> https://repository.apache.org/content/repositories/orgapachemrunit-227/org/apache/mrunit/mrunit/0.8.0-incubating/
> >> >> >>
> >> >> >> Brock
> >> >> >>
> >> >> >> --
> >> >> >> Apache MRUnit - Unit testing MapReduce -
> >> >> >> http://incubator.apache.org/mrunit/
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Apache MRUnit - Unit testing MapReduce -
> >> >> http://incubator.apache.org/mrunit/
> >> >>
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message