spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Fan <>
Subject Re: How to configure alluxio cluster with spark in yarn
Date Fri, 17 May 2019 00:28:07 GMT
hi Andy

Assuming you are running Spark with YARN, then I would recommend deploying
Alluxio in the same YARN cluster if you are looking for best performance.
Alluxio can also be deployed separated as a standalone service, but in that
case, you may need to transfer data from Alluxio cluster to your Spark/YARN

Here is the documentation
deploying Alluxio with YARN.

- Bin

On Thu, May 9, 2019 at 4:19 AM u9g <> wrote:

> Hey,
> I want to speed up the Spark task running in the Yarn cluster through
> Alluxio. Is Alluxio recommended to run in the same yarn cluster on the yarn
> mode? Should I deploy Alluxio independently on the nodes of the yarn
> cluster? Or deploy a cluster separately?
> Best,
> Andy Li

View raw message