ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Pavlov <dpavlov....@gmail.com>
Subject Re: work with files and directories
Date Wed, 16 May 2018 09:46:17 GMT
Andrey Gura expressed in words what I also thought.

I agree if we extract common code to some kind of file utils.

In the same time creating common file operation framework is not possible
because of different nature of WAL, Page Store and other components using
files.

ср, 16 мая 2018 г. в 11:27, Dmitriy Govorukhin <dmitriy.govorukhin@gmail.com
>:

> Andrey,
>
> My point is, it will be very cool if there is some component who will know
> about persistence folder and files,
> and we can move all generic code, for work with files,  to this component.
>
> On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jokserfn@gmail.com>
> wrote:
>
> > Andrey,
> >
> > I think the main idea of that not about how to unify writing content to
> > file. It's about creating peristence files and folders. For other
> > persistence components (WAL, Checkpoint, etc.) it should be like
> > FileFactory object with methods like "public File
> > getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So,
> all
> > logic about how files should look will be moved to centralized point and
> it
> > will be more easier to understand our persistence structure.
> >
> > 2018-05-16 0:42 GMT+03:00 Andrey Gura <agura@apache.org>:
> >
> > > Hi,
> > >
> > > I understand you idea but it just increases dependencies of different
> > > component from one that is in general bad practice.
> > >
> > > We have different components where each one can use different approach
> > for
> > > file management. For example page store and WAL have different file IO
> > > implementations due to performance reasons and we have to provide
> > different
> > > mechanics for work with files.
> > >
> > > Of course we can refactor mentioned components in more structured
> manner
> > > but we should not strongly link it with one implementation.
> > >
> > > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dpavlov.spb@gmail.com>:
> > >
> > > > Hi Maxim,
> > > >
> > > > FileWriteAheadLogManager  &  FsyncModeFileWriteAheadLogManager  were
> > > > intentionally copy-pasted in hope we will soon delete FsyncManager.
> > > >
> > > > But it is still shows this tool works good. Probably we could
> integrate
> > > > this tool to our processes.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > >
> > > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <maxmuzaf@gmail.com>:
> > > >
> > > > > +1 also for something like "resource manager".
> > > > >
> > > > > Recently, I've found for myself sonarcloud.io tool for code
> > analisys.
> > > > It's
> > > > > free for open source project and I've made Ignite project initial
> run
> > > > [1].
> > > > >
> > > > > I've prepeared analisys for mysefl and found a lot of duplicated
> code
> > > > > blocks [1]. Of course it's not the ideal tool but gave us direction
> > of
> > > > > thoughts. E.g. these classes [3]:
> > > > > 1) FileWriteAheadLogManager.java
> > > > > 2) FsyncModeFileWriteAheadLogManager.java
> > > > >
> > > > >
> > > > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> > > ignite%3Aapache-ignite
> > > > > [2]
> > > > >
> > > > >
> > > > https://sonarcloud.io/component_measures?id=org.
> > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > > > [3]
> > > > >
> > > > >
> > > > https://sonarcloud.io/component_measures?id=org.
> > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> > > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> > > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> > > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > > > >
> > > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jokserfn@gmail.com>:
> > > > >
> > > > > > +1 to this approach,
> > > > > >
> > > > > > It can be also very helpful in failover scenarios when something
> > > wrong
> > > > > > happened with disk. In this case we're reducing the number of
> > points
> > > of
> > > > > > failure.
> > > > > >
> > > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > > > dmitriy.govorukhin@gmail.com>
> > > > > > :
> > > > > >
> > > > > > > Hi Ignite,
> > > > > > >
> > > > > > > Do we have a general approach to work with a file and
> > directories?
> > > > > > > I see many duplication logic for write through .tmp file.
> > > > > > >
> > > > > > > For example,
> > > > > > >
> > > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > > > >
> > > > > > > All of these methods implement the same logic, write to
tmp
> file
> > > and
> > > > > > rename
> > > > > > > to normal name.
> > > > > > >
> > > > > > > I guess, will be better if we stopping write duplication
logic
> > code
> > > > and
> > > > > > > start to consolidate all in one place.
> > > > > > >
> > > > > > > Also, I think that current approach to creating files is
not
> > quite
> > > > > right
> > > > > > > faithful. Each internal component
> > > > > > > create/delete files inside himself, and nobody knows where
> which
> > > > files
> > > > > > > located.
> > > > > > >
> > > > > > > I suggest refactoring code and create something (maybe
new
> > manager)
> > > > > that
> > > > > > > will know about all files inside the node. All internal
> > components
> > > > must
> > > > > > > create files only through this one.
> > > > > > >
> > > > > > > It makes help to write tests for persistence easier and
reduce
> > > > > > duplication
> > > > > > > code in working with files.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message