ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Persistent Distributed Store Metrics
Date Tue, 06 Jun 2017 19:56:38 GMT
> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <cos@apache.org> wrote:
>> 
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>> 
>> 
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
> 
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.

Just in case I’m working on the high-level storage documentation. Here is a draft for your
reference:
https://apacheignite.readme.io/docs/distributed-persistent-store <https://apacheignite.readme.io/docs/distributed-persistent-store>

The page is invisible for wide audience at the moment. Only for Ignite committers.

—
Denis

> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik <cos@apache.org> wrote:
>> 
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>> 
>> 
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
> 
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.
> 
>> There is also an active stabilization thread for the persistence branch
>> [3]. I encouraged the community to get involved and post any questions or
>> concerns there as well.
>> 
>> There is an upcoming in-memory computing conference that is coming up in
>> June in Amsterdam [4], where there are many Ignite talks scheduled. It
>> would be great to be able to talk about the persistence features of Ignite
>> there as well. I would like to ask the community to mobilize with reviewing
>> the donated code, so we could have something concrete to tell the audience
>> on the conference.
> 
> I am sure that having the code on the branch is good enough to be able to talk
> about this. Having this in the release isn't really a contingency to be able
> to make a presentation, right?
> 
> Cos
> 
>> [2]
>> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
>> [3]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
>> [4] https://imcsummit.org/
>> 
>> 
>> 
>>> And from what I am seeing in the discussions like this [1], we need to be
>>> extra careful.
>>> 
>> 
>> I would keep the discussion in [1] separate from the persistence store.
>> These are 2 unrelated issues. I will respond on [1] either today or
>> tomorrow, but I agree in general that it should be fixed ASAP.
>> 
>> 
>>> BTW, you have sent this email 9 days before the vote had happened! A bit
>>> too
>>> soon, if you ask me.
>>> 
>> 
>> Cos, this email was sent 1 week after the initial donated code branch was
>> presented to the community (see [2] above). The developers involved were
>> eager to make it available to the users as soon as possible, but no code
>> has been merged to the master branch yet.
>> 
>> I would like to encourage everyone in the community to participate in the
>> persistence branch coding discussions, like the one in this thread, and
>> familiarize themselves with the code.
>> 
>> 
>>> 
>>> [1] https://is.gd/UQCr51
>>> 
>>> Cos
>>> 
>>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
>>>> Folk,
>>>> 
>>>> As you know, ignite 2.1 will contain new module (pds), it will be
>>>> provide ability to store data on disk. Let's discuss what type of
>>>> metrics we need for this?
>>>> I think it must be metrics per memory policy, per cache, checkpoint,
>>>> and global metrics which will be aggregate all metrics.
>>>> 
>>>> I did sketch.
>>>> 
>>>> PersistentStoreMetrics.java
>>>> 
>>>> public interface PersistentStoreMetrics {
>>>> 
>>>>    // Global metrics.
>>>> 
>>>>    public long getMemorySize();
>>>> 
>>>>    public long getDiskSize();
>>>> 
>>>>    public long getPagesInMemory();
>>>> 
>>>>    public long getPagesSizeInMemory();
>>>> 
>>>>    public long getPagesOnDisk();
>>>> 
>>>>    public long getPagesSizeOnDisk();
>>>> 
>>>>    public long getFreePages();
>>>> 
>>>>    public long getFreePagesSize();
>>>> 
>>>>    public long getDirtyPages();
>>>> 
>>>>    public long getDirtyPagesSize();
>>>> 
>>>>    public long walLog();
>>>> 
>>>>    public long walLogSize();
>>>> 
>>>>    // Frequency.
>>>> 
>>>>    public long getPagesRead();
>>>> 
>>>>    public long getPagesWrite();
>>>> 
>>>>    public long getFsync();
>>>> 
>>>>    public long getWal();
>>>> 
>>>>    public long getAverageWalFsyncTime();
>>>> 
>>>>    // Per cache.
>>>> 
>>>>    public PersistentStoreCacheMetrics cache(String name);
>>>> 
>>>>    public PersistentStoreCacheMetrics cache(int cacheId);
>>>> 
>>>>    // For last checkpoint.
>>>> 
>>>>    public PersistentStoreCheckpointMetrics getLastCheckPoint();
>>>> }
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>> 
>>>> PersistentStoreCacheMetrics.java
>>>> 
>>>> public interface PersistentStoreCacheMetrics {
>>>> 
>>>>    public String name();
>>>> 
>>>>    public double getFillFactor();
>>>> 
>>>>    public double getFillFactor(int part);
>>>> 
>>>>    public long getMemorySize();
>>>> 
>>>>    public long getDiskSize();
>>>> 
>>>>    public long getPagesInMemory();
>>>> 
>>>>    public long getPagesSizeInMemory();
>>>> 
>>>>    public long getPagesOnDisk();
>>>> 
>>>>    public long getPagesSizeOnDisk();
>>>> 
>>>>    public long getFreePages();
>>>> 
>>>>    public long getFreePagesSize();
>>>> 
>>>>    public long getDirtyPages();
>>>> 
>>>>    public long getDirtyPagesSize();
>>>> 
>>>>    public long getPagesRead();
>>>> 
>>>>    public long getPagesWritten();
>>>> }
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>> 
>>>> PersistentStoreCheckpointMetrics.java
>>>> 
>>>> public interface PersistentStoreCheckpointMetrics {
>>>> 
>>>>    public long getTotalPages();
>>>> 
>>>>    //TODO Page type is internal?
>>>>    public long[] pagesType();
>>>> 
>>>>    public long getExecutingTime();
>>>> 
>>>>    public long getFsyncTime();
>>>> 
>>>>    public long getPagesCopyOnWrite();
>>>> }
>>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message