ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Franciscus, Naden" <Naden.Francis...@team.telstra.com>
Subject Re: Spark/YARN
Date Tue, 03 May 2016 23:49:55 GMT

Hey Alexey,

Sorry I meant HDFS :)

It would be good for IgniteContext to have the option to read
configuration from HDFS.
Again many users of Hadoop aren't going to have the ability to modify data
nodes to install
Ignite. But I will raise that as a separate issue.

The big issue is that there is a lot of code like this in documentation
and elsewhere:
ignite.cache("myCache").put(personId, person);

Which we can't use in any RDD/DF map, filter etc because neither
Ignite/IgniteCache are serializable.


Cheers,
Naden





On 4/05/2016, 5:42 AM, "Alexey Goncharuk" <alexey.goncharuk@gmail.com>
wrote:

>Hi Naden,
>
> But the IgniteContext within Spark doesn't allow you to read
>configuration
>> files from YARN.
>>
>
>I am a little bit confused. Ignite can be configured via basic Spring XML,
>and you can definitely read those XML files off HDFS or any other source.
>Is there any reason why XML does not work for you?
>
>Now it is impossible to make IgniteConfiguration serializable because it
>contains such components as SPIs, which are non-serializable by it's
>nature.


Mime
View raw message