ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Pluggable SecondaryFileSystemProvider?
Date Fri, 18 Dec 2015 02:30:01 GMT
+1 on Vladimir’s suggestion. Let’s remember to document it whenever we make
the change.

I also think that we should have an out-of-the-box implementation of the
file-system-factory with a fix for Kerberos.

D.

On Thu, Dec 17, 2015 at 12:01 PM, Vladimir Ozerov <vozerov@gridgain.com>
wrote:

> + 1 for idea in general, but I have several concerns:
>
> 1) SecondaryFileSystemProvider is internal class and cannot be exposed. We
> need some other interface here.
> 2) I would avoid setting properties through constructor because it is
> error-prone when Spring is used. Instead, I would create setters as in
> almost all other components.
> 3) Ideally we should not break existing interfaces.
>
> To summarize, it could looks as follows:
>
> class IgniteHadoopIgfsSecondaryFileSystem {
>     /** These properties already exist. */
>     void setUri(String fileSystemUri);
>     void setConfig(String pathToConfig);
>     void setUserName(String userName);
>
>     /** This is a new property. */
>     void setFileSystemFactory(HadoopFileSystemFactory factory);
> }
>
> interface HadoopFileSystemFactory {
>     FileSystem create(String uri, String configPath, String userName);
> }
>
> Vladimir.
>
> On Thu, Dec 17, 2015 at 10:34 PM, Ivan V. <iveselovskiy@gridgain.com>
> wrote:
>
> > To summarize suggested change in short: we add constructor
> >
> >
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
> >
> org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
> > allowing to plug an arbitrary SecondaryFileSystemProvider bean when
> > constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:
> >
> >                        <bean
> > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
> >                             <constructor-arg name="userName"
> value="ivan"/>
> >                             <constructor-arg name="provider">
> >                                 <bean class="
> > *com.foo.MySuperSecondaryFileSystemProvider*">
> >                                         <constructor-arg name="secUri"
> > value="hdfs://localhost:9000"/>
> >                                         <constructor-arg
> > name="secConfPath"><null/></constructor-arg>
> >                                 </bean>
> >                             </constructor-arg>
> >                         </bean>
> >
> >
> > On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <iveselovskiy@gridgain.com>
> > wrote:
> >
> > > Hi, dev,
> > > one of Ignite users discovered that IGFS does not correctly work with
> > > secondary HDFS file system if that HDFS is "kerberised". The issue is
> > > described in https://issues.apache.org/jira/browse/IGNITE-2195 (This
> may
> > > also be close to this issue:
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> > > ). The user suggests a fix that creates SecondaryFileSystem using some
> > > proxy UserGroupInformation, and that workaround works for him.
> > > However, we're not sure this fix is generic enough and will work well
> in
> > > all possible configurations.
> > > So, there is an idea to make SecondaryFileSystemProvider pluggable to
> > > allow users to write own code for specific configurations. An example
> can
> > > be seen in this pull request:
> > > https://github.com/apache/ignite/pull/350 .
> > > Pros, cons, other thoughts?
> > > Thanks in advance.
> > > --
> > > ivan
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message