ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Franciscus, Naden" <Naden.Francis...@team.telstra.com>
Subject Spark/YARN
Date Mon, 02 May 2016 06:32:35 GMT

Hey guys,

So it looks like using Spark/Ignite on YARN together simply doesn't work.

Many of us have Hadoop appliances where we aren't allowed to install anything on the nodes.
So the only option is YARN which
barring a few bugs seems to work okay. But the IgniteContext within Spark doesn't allow you
to read configuration files from YARN.

So since you allow users to pass in an IgniteConfiguration we have tried to manually set configuration
on the POJOs:
https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala

But during any Spark distributed operation it will attempt to serialise this which is not
possible since most of the classes contained
within IgniteConfiguration e.g. TcpDiscoverySpi are not serializable.

I am going to go through and see how many classes will need to be marked serializable (could
be dozens) but a call will need to be made:

 1.  Mark everything within IgniteConfiguration as serializable.
 2.  Force ALL users of IgniteContext to either read config from HDFS or from a Local Filesystem.
Both will go through Spring layer.

What's the best way to get a decision on this ?

Cheers,
Naden

Mime
View raw message