ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Pluggable SecondaryFileSystemProvider?
Date Thu, 17 Dec 2015 20:01:54 GMT
+ 1 for idea in general, but I have several concerns:

1) SecondaryFileSystemProvider is internal class and cannot be exposed. We
need some other interface here.
2) I would avoid setting properties through constructor because it is
error-prone when Spring is used. Instead, I would create setters as in
almost all other components.
3) Ideally we should not break existing interfaces.

To summarize, it could looks as follows:

class IgniteHadoopIgfsSecondaryFileSystem {
    /** These properties already exist. */
    void setUri(String fileSystemUri);
    void setConfig(String pathToConfig);
    void setUserName(String userName);

    /** This is a new property. */
    void setFileSystemFactory(HadoopFileSystemFactory factory);
}

interface HadoopFileSystemFactory {
    FileSystem create(String uri, String configPath, String userName);
}

Vladimir.

On Thu, Dec 17, 2015 at 10:34 PM, Ivan V. <iveselovskiy@gridgain.com> wrote:

> To summarize suggested change in short: we add constructor
>
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
> org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
> allowing to plug an arbitrary SecondaryFileSystemProvider bean when
> constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:
>
>                        <bean
> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>                             <constructor-arg name="userName" value="ivan"/>
>                             <constructor-arg name="provider">
>                                 <bean class="
> *com.foo.MySuperSecondaryFileSystemProvider*">
>                                         <constructor-arg name="secUri"
> value="hdfs://localhost:9000"/>
>                                         <constructor-arg
> name="secConfPath"><null/></constructor-arg>
>                                 </bean>
>                             </constructor-arg>
>                         </bean>
>
>
> On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <iveselovskiy@gridgain.com>
> wrote:
>
> > Hi, dev,
> > one of Ignite users discovered that IGFS does not correctly work with
> > secondary HDFS file system if that HDFS is "kerberised". The issue is
> > described in https://issues.apache.org/jira/browse/IGNITE-2195 (This may
> > also be close to this issue:
> >
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> > ). The user suggests a fix that creates SecondaryFileSystem using some
> > proxy UserGroupInformation, and that workaround works for him.
> > However, we're not sure this fix is generic enough and will work well in
> > all possible configurations.
> > So, there is an idea to make SecondaryFileSystemProvider pluggable to
> > allow users to write own code for specific configurations. An example can
> > be seen in this pull request:
> > https://github.com/apache/ignite/pull/350 .
> > Pros, cons, other thoughts?
> > Thanks in advance.
> > --
> > ivan
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message