ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Rakov <ivan.glu...@gmail.com>
Subject Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC
Date Tue, 27 Mar 2018 11:56:35 GMT
Igniters,

I've completed development of 
https://issues.apache.org/jira/browse/IGNITE-7754. TeamCity state is ok. 
Please, review my changes.
Please note that it will be possible to track time of WAL fsync on 
checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint 
started" message.

Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057 
with description of possible further improvement of WAL fsync on 
checkpoint begin.

Best Regards,
Ivan Rakov

On 26.03.2018 23:45, Valentin Kulichenko wrote:
> Ivan,
>
> It's all good then :) Thanks!
>
> -Val
>
> On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <ivan.glukos@gmail.com> wrote:
>
>> Val,
>>
>> There's no any sense to use WalMode.NONE in production environment, it's
>> kept for testing and debugging purposes (including possible user activities
>> like capacity planning).
>> We already print a warning at node start in case WalMode.NONE is set:
>>
>> U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
>>> persisted data may be lost in " +
>>>       "a case of unexpected node failure. Make sure to deactivate the
>>> cluster before shutdown.");
>>>
>> Best Regards,
>> Ivan Rakov
>>
>>
>> On 24.03.2018 1:40, Valentin Kulichenko wrote:
>>
>>> Dmitry,
>>>
>>> Thanks for clarification. So it sounds like if we fix all other modes as
>>> we
>>> discuss here, NONE would be the only one allowing corruption. I also don't
>>> see much sense in this and I think we should clearly state this in the
>>> doc,
>>> as well print out a warning if NONE mode is used. Eventually, if it's
>>> confirmed that there are no reasonable use cases for it, we can deprecate
>>> it.
>>>
>>> -Val
>>>
>>> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>>> wrote:
>>>
>>> Hi Val,
>>>> NONE means that the WAL log is disabled and not written at all. Use of
>>>> the
>>>> mode is at your own risk. It is possible that restore state after the
>>>> crash
>>>> at the middle of checkpoint will not succeed. I do not see much sence in
>>>> it, especially in production.
>>>>
>>>> BACKGROUND is full functional WAL mode, but allows some delay before
>>>> flush
>>>> to disk.
>>>>
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>>
>>>> сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
>>>> valentin.kulichenko@gmail.com>:
>>>>
>>>> I agree. In my view, any possibility to get a corrupted storage is a bug
>>>>> which needs to be fixed.
>>>>>
>>>>> BTW, can someone explain semantics of NONE mode? What is the difference
>>>>> from BACKGROUND from user's perspective? Is there any particular use
>>>>> case
>>>>> where it can be used?
>>>>>
>>>>> -Val
>>>>>
>>>>> On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Ivan,
>>>>>> IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?
>>>>>>
>>>>>> Sincerely,
>>>>>> Dmitriy Pavlov
>>>>>>
>>>>>> пт, 23 мар. 2018 г. в 12:23, Ivan Rakov <ivan.glukos@gmail.com>:
>>>>>>
>>>>>> Igniters, there's another important question about this matter.
>>>>>>> Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think
that
>>>>>>>
>>>>>> we
>>>>> have to do it: it will cause similar performance drop, but if we
>>>>>>> consider LOG_ONLY broken without these fixes, BACKGROUND is broken
as
>>>>>>>
>>>>>> well.
>>>>>>
>>>>>>> Best Regards,
>>>>>>> Ivan Rakov
>>>>>>>
>>>>>>> On 23.03.2018 10:27, Ivan Rakov wrote:
>>>>>>>
>>>>>>>> Fixes are quite simple.
>>>>>>>> I expect them to be merged in master in a week in worst case.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Ivan Rakov
>>>>>>>>
>>>>>>>> On 22.03.2018 17:49, Denis Magda wrote:
>>>>>>>>
>>>>>>>>> Ivan,
>>>>>>>>>
>>>>>>>>> How quick are you going to merge the fix into the master?
Many
>>>>>>>>> persistence
>>>>>>>>> related optimizations have already stacked up. Probably,
we can
>>>>>>>>>
>>>>>>>> release
>>>>>>> them sooner if the community agrees.
>>>>>>>>> --
>>>>>>>>> Denis
>>>>>>>>>
>>>>>>>>> On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <
>>>>>>>>>
>>>>>>>> ivan.glukos@gmail.com>
>>>>> wrote:
>>>>>>>>> Thanks all!
>>>>>>>>>> We seem to have reached a consensus on this issue.
I'll just add
>>>>>>>>>> necessary
>>>>>>>>>> fsyncs under IGNITE-7754.
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Ivan Rakov
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 22.03.2018 15:13, Ilya Lantukh wrote:
>>>>>>>>>>
>>>>>>>>>> +1 for fixing LOG_ONLY. If current implementation
doesn't
>>>>>>>>>> protect
>>>>> from
>>>>>>> data
>>>>>>>>>>> corruption, it doesn't make sence.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 21, 2018 at 10:38 PM, Denis Magda
<
>>>>>>>>>>>
>>>>>>>>>> dmagda@apache.org>
>>>>> wrote:
>>>>>>>>>>> +1 for the fix of LOG_ONLY
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 21, 2018 at 11:23 AM, Alexey
Goncharuk <
>>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> +1 for fixing LOG_ONLY to enforce corruption
safety given the
>>>>>>>>>>>> provided
>>>>>>>>>>>>
>>>>>>>>>>>>> performance results.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-03-21 18:20 GMT+03:00 Vladimir Ozerov
<
>>>>>>>>>>>>>
>>>>>>>>>>>> vozerov@gridgain.com
>>>>>> :
>>>>>>>> +1 for accepting drop in LOG_ONLY. 7% is not that much and
>>>>>>>>>>>> not a
>>>>> drop
>>>>>>>>>>>>> at
>>>>>>>>>>>>> all, provided that we fixing a bug. I.e.
should we implement
>>>>>>>>>>>>>
>>>>>>>>>>>> it
>>>>> correctly
>>>>>>>>>>>>> in the first place we would never notice
any "drop".
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not understand why someone would
like to use current
>>>>>>>>>>>>>>
>>>>>>>>>>>>> broken
>>>>>> mode.
>>>>>>>>>>>>>> On Wed, Mar 21, 2018 at 6:11 PM,
Dmitry Pavlov
>>>>>>>>>>>>>> <dpavlov.spb@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, I think option 1 is better. As
Val said any mode that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> allows
>>>>>> corruption
>>>>>>>>>>>>>> does not make much sense.
>>>>>>>>>>>>>>> What Ivan mentioned here as drop,
in relation to old mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DEFAULT
>>>>>>> (FSYNC
>>>>>>>>>>>>>> now), is still significant perfromance
boost.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sincerely,
>>>>>>>>>>>>>>> Dmitriy Pavlov
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ср, 21 мар. 2018 г. в
17:56, Ivan Rakov <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ivan.glukos@gmail.com
>>>>>> :
>>>>>>>> I've attached benchmark results to the JIRA ticket.
>>>>>>>>>>>>>>>> We observe ~7% drop in "fair"
LOG_ONLY_SAFE mode,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> independent
>>>>> of
>>>>>>> WAL
>>>>>>>>>>>>>> compaction enabled flag. It's pretty
significant drop: WAL
>>>>>>>>>>>>>> compaction
>>>>>>>>>>>>>> itself gives only ~3% drop.
>>>>>>>>>>>>>> I see two options here:
>>>>>>>>>>>>>>>> 1) Change LOG_ONLY behavior.
That implies that we'll be
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ready
>>>>> to
>>>>>>> release
>>>>>>>>>>>>>>> AI 2.5 with 7% drop.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) Introduce LOG_ONLY_SAFE,
make it default, add release
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> note
>>>>> to AI
>>>>>>>>>>>>>>>> 2.5
>>>>>>>>>>>>>> that we added power loss durability
in default mode, but user
>>>>>>>>>>>>>>
>>>>>>>>>>>>> may
>>>>>> fallback to previous LOG_ONLY in order to retain
>>>>>>>>>>>>>>> performance.
>>>>> Thoughts?
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Ivan Rakov
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 20.03.2018 16:00, Ivan
Rakov wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Val,
>>>>>>>>>>>>>>>>> If a storage is in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> corrupted state,
does it mean that it needs to be
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> completely
>>>>>> removed
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> cluster needs to be restarted
without data?
>>>>>>>>>>>>>>>>> Yes, there's a chance
that in LOG_ONLY all local data will
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> be
>>>>>> lost,
>>>>>>>>>>>>>>> but only in *power loss**/ OS
crash* case.
>>>>>>>>>>>>>> kill -9, JVM crash, death of critical
system thread and
>>>>>>>>>>>>>>>> all
>>>>> other
>>>>>>>>>>>>>>>>> cases that usually take
place are variations of *process
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> crash*.
>>>>>>> All
>>>>>>>>>>>>>>> WAL modes (except NONE, of course)
ensure corruption-safety
>>>>>>>>>>>>> in
>>>>> case
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> process crash.
>>>>>>>>>>>>>>>> If so, I'm not sure any mode
>>>>>>>>>>>>>>>>>> that allows corruption
makes much sense to me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It depends on performance
impact of enforcing power-loss
>>>>>>>>>>>>>>>>> corruption
>>>>>>>>>>>>>>> safety. Price of full protection
from power loss is high -
>>>>>>>>>>>> FSYNC
>>>>> is
>>>>>>>>>>>>>>> way slower (2-10 times) than
other WAL modes. The question is
>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>> ensuring weaker guarantees (corruption
can't happen, but loss
>>>>>>>>>>>>> of
>>>>>> last
>>>>>>>>>>>>>>> updates can) will affect performance
as badly as strong
>>>>>>>>>>>>>>> guarantees.
>>>>>>>>>>>>>>> I'll share benchmark results
soon.
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Ivan Rakov
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 20.03.2018 5:09, Valentin
Kulichenko wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>> What do we understand
under "data corruption" here? If a
>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> corrupted state, does it mean that
it needs to be completely
>>>>>>>>>>>>>>>> removed
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> cluster needs to be restarted
without data? If so, I'm not
>>>>>>>>>>>>>>> sure
>>>>>> any
>>>>>>>>>>>>>>>> mode
>>>>>>>>>>>>>>> that allows corruption makes
much sense to me. How am I
>>>>>>>>>>>>>>> supposed
>>>>>>> to
>>>>>>>>>>>>>>>> use a
>>>>>>>>>>>>>>> database, if virtually any failure
can end with complete
>>>>>>>>>>>>>>>>>> loss of
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> data?
>>>>>>>>>>>>>>>> In any case, this definitely
should not be a default
>>>>>>>>>>>>>> behavior.
>>>>> If
>>>>>>>>>>>>>>>> user ever
>>>>>>>>>>>>>> switches to corruption-unsafe mode,
there should be a
>>>>>>>>>>>>>>>>> clear
>>>>> warning
>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Mar 16, 2018
at 1:06 AM, Ivan Rakov <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ivan.glukos@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Ticket to track changes:
>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7754
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Ivan Rakov
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 16.03.2018
10:58, Dmitriy Setrakyan wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Mar 16,
2018 at 12:55 AM, Ivan Rakov <
>>>>>>>>>>>>>>>>>>> ivan.glukos@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Vladimir,
>>>>>>>>>>>>>>>>>>>> Unlike BACKGROUND,
LOG_ONLY provides strict write
>>>>>>>>>>>>>>>>>>>> guarantees
>>>>>>> unless power
>>>>>>>>>>>>>>>>>>>>> loss
has happened.
>>>>>>>>>>>>>>>>>>>>> Seems
like we need to measure performance difference
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> to
>>>>> decide
>>>>>>>>>>>>>>>>>>> whether do
>>>>>>>>>>>>>> we need separate WAL mode. If it
will be invisible,
>>>>>>>>>>>>>>>>>>>> we'll
>>>>>> just
>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>> bugs without introducing
new mode; if it will be
>>>>>>>>>>>>>>>>>>>>> perceptible,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> we'll
>>>>>>>>>>>>>>>>>>> continue the
discussion about introducing LOG_ONLY_SAFE.
>>>>>>>>>>>>>>>> Makes sense?
>>>>>>>>>>>>>>>>>>>>> Yes,
this sounds like the right approach.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message