spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dhruve ashar <>
Subject Re: Where is yarn-shuffle.jar in maven?
Date Tue, 13 Dec 2016 18:05:20 GMT
Hi Neal,

>From my understanding, the reason I think the shuffle is not available in
maven central is that it is an external component and dependent on the
cluster manager version - atleast in case of yarn.

For example, it would require the appropriate hadoop profile based on your
underlying infra to build the right shuffle component. Publishing all of
the different combinations to maven doesn't make sense.

So you would have to store it externally - probably an internal central
repo in your company.


On Mon, Dec 12, 2016 at 11:56 PM, Neal Yin <> wrote:

> Hi,
> For dynamic allocation feature, I need spark-xxx-yarn-shuffle.jar. In my
> local spark build, I can see it.  But in maven central, I can’t find it. My
> build script pulls all jars from maven central. The only option is to check
> in this jar into git?
> Thanks,
> -Neal

-Dhruve Ashar

View raw message