ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Govorukhin <dmitriy.govoruk...@gmail.com>
Subject Re: work with files and directories
Date Wed, 16 May 2018 12:30:32 GMT
One more thing, If we implement this component and will aggregate
information about all files, we can
create failure handler which provided detail information about node
persistence structure. It will be helpful for debugging node crash.

On Wed, May 16, 2018 at 12:46 PM, Dmitry Pavlov <dpavlov.spb@gmail.com>
wrote:

> Andrey Gura expressed in words what I also thought.
>
> I agree if we extract common code to some kind of file utils.
>
> In the same time creating common file operation framework is not possible
> because of different nature of WAL, Page Store and other components using
> files.
>
> ср, 16 мая 2018 г. в 11:27, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com
> >:
>
> > Andrey,
> >
> > My point is, it will be very cool if there is some component who will
> know
> > about persistence folder and files,
> > and we can move all generic code, for work with files,  to this
> component.
> >
> > On Wed, May 16, 2018 at 12:59 AM, Pavel Kovalenko <jokserfn@gmail.com>
> > wrote:
> >
> > > Andrey,
> > >
> > > I think the main idea of that not about how to unify writing content to
> > > file. It's about creating peristence files and folders. For other
> > > persistence components (WAL, Checkpoint, etc.) it should be like
> > > FileFactory object with methods like "public File
> > > getOrCreatePartitionFile(...), public void commitFIle(..) and etc". So,
> > all
> > > logic about how files should look will be moved to centralized point
> and
> > it
> > > will be more easier to understand our persistence structure.
> > >
> > > 2018-05-16 0:42 GMT+03:00 Andrey Gura <agura@apache.org>:
> > >
> > > > Hi,
> > > >
> > > > I understand you idea but it just increases dependencies of different
> > > > component from one that is in general bad practice.
> > > >
> > > > We have different components where each one can use different
> approach
> > > for
> > > > file management. For example page store and WAL have different file
> IO
> > > > implementations due to performance reasons and we have to provide
> > > different
> > > > mechanics for work with files.
> > > >
> > > > Of course we can refactor mentioned components in more structured
> > manner
> > > > but we should not strongly link it with one implementation.
> > > >
> > > > вт, 15 мая 2018 г., 20:10 Dmitry Pavlov <dpavlov.spb@gmail.com>:
> > > >
> > > > > Hi Maxim,
> > > > >
> > > > > FileWriteAheadLogManager  &  FsyncModeFileWriteAheadLogManager
> were
> > > > > intentionally copy-pasted in hope we will soon delete FsyncManager.
> > > > >
> > > > > But it is still shows this tool works good. Probably we could
> > integrate
> > > > > this tool to our processes.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > >
> > > > > вт, 15 мая 2018 г. в 20:06, Maxim Muzafarov <maxmuzaf@gmail.com>:
> > > > >
> > > > > > +1 also for something like "resource manager".
> > > > > >
> > > > > > Recently, I've found for myself sonarcloud.io tool for code
> > > analisys.
> > > > > It's
> > > > > > free for open source project and I've made Ignite project initial
> > run
> > > > > [1].
> > > > > >
> > > > > > I've prepeared analisys for mysefl and found a lot of duplicated
> > code
> > > > > > blocks [1]. Of course it's not the ideal tool but gave us
> direction
> > > of
> > > > > > thoughts. E.g. these classes [3]:
> > > > > > 1) FileWriteAheadLogManager.java
> > > > > > 2) FsyncModeFileWriteAheadLogManager.java
> > > > > >
> > > > > >
> > > > > > [1] https://sonarcloud.io/dashboard?id=org.apache.
> > > > ignite%3Aapache-ignite
> > > > > > [2]
> > > > > >
> > > > > >
> > > > > https://sonarcloud.io/component_measures?id=org.
> > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks
> > > > > > [3]
> > > > > >
> > > > > >
> > > > > https://sonarcloud.io/component_measures?id=org.
> > > > apache.ignite%3Aapache-ignite&metric=duplicated_blocks&
> > > > selected=org.apache.ignite%3Aignite-core%3Asrc%2Fmain%
> > > > 2Fjava%2Forg%2Fapache%2Fignite%2Finternal%2Fprocessors%2Fcache%
> > > > 2Fpersistence%2Fwal%2FFsyncModeFileWriteAheadLogManager.java
> > > > > >
> > > > > > вт, 15 мая 2018 г. в 19:26, Pavel Kovalenko <jokserfn@gmail.com
> >:
> > > > > >
> > > > > > > +1 to this approach,
> > > > > > >
> > > > > > > It can be also very helpful in failover scenarios when
> something
> > > > wrong
> > > > > > > happened with disk. In this case we're reducing the number
of
> > > points
> > > > of
> > > > > > > failure.
> > > > > > >
> > > > > > > 2018-05-15 18:37 GMT+03:00 Dmitriy Govorukhin <
> > > > > > > dmitriy.govorukhin@gmail.com>
> > > > > > > :
> > > > > > >
> > > > > > > > Hi Ignite,
> > > > > > > >
> > > > > > > > Do we have a general approach to work with a file
and
> > > directories?
> > > > > > > > I see many duplication logic for write through .tmp
file.
> > > > > > > >
> > > > > > > > For example,
> > > > > > > >
> > > > > > > > GridCacheDatabaseSharedManager.writeCheckpointEntry();
> > > > > > > > GridCacheDatabaseSharedManage.nodeStart();
> > > > > > > > FileWriteAheadLogManager.FileArchiver.archiveSegment();
> > > > > > > >
> > > > > > > > All of these methods implement the same logic, write
to tmp
> > file
> > > > and
> > > > > > > rename
> > > > > > > > to normal name.
> > > > > > > >
> > > > > > > > I guess, will be better if we stopping write duplication
> logic
> > > code
> > > > > and
> > > > > > > > start to consolidate all in one place.
> > > > > > > >
> > > > > > > > Also, I think that current approach to creating files
is not
> > > quite
> > > > > > right
> > > > > > > > faithful. Each internal component
> > > > > > > > create/delete files inside himself, and nobody knows
where
> > which
> > > > > files
> > > > > > > > located.
> > > > > > > >
> > > > > > > > I suggest refactoring code and create something (maybe
new
> > > manager)
> > > > > > that
> > > > > > > > will know about all files inside the node. All internal
> > > components
> > > > > must
> > > > > > > > create files only through this one.
> > > > > > > >
> > > > > > > > It makes help to write tests for persistence easier
and
> reduce
> > > > > > > duplication
> > > > > > > > code in working with files.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message