hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Sharma <ssha...@pivotal.io>
Subject Re: Hawq standby sync fails if there are existing connections to master
Date Tue, 14 Nov 2017 02:05:47 GMT
To close this thread, the source of this issue appears to be
tools/bin/hawq_ctl method _resync_standby. When this is called the command
formation does not include stop_mode options as passed to the arguments.

 def _resync_standby(self):
        logger.info("Re-sync standby")
        cmd = "%s; hawq stop master -a;" % source_hawq_env
        check_return_code(local_ssh(cmd, logger), logger, "Stop hawq
cluster failed, exit")
        ......
        ......

JIRA ID - HAWQ-1549

On Mon, Nov 13, 2017 at 5:56 PM, Shubham Sharma <ssharma@pivotal.io> wrote:

> Thanks, I think I found out where the issue is. Will create a JIRA and
> submit PR.
>
> On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <chang.lei.cn@gmail.com> wrote:
>
>> oh, yes, make sense, I overlooked the -M option here.
>>
>> Here we should respect the input "-M" option.
>>
>> Cheers
>> Lei
>>
>>
>>
>> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ssharma@pivotal.io>
>> wrote:
>>
>> > Thanks for the response Lei.
>> >
>> > I understand this completely and agree with the behavior that we should
>> not
>> > brutally terminate connections.
>> >
>> > However, the case here is, am deliberately trying to use stop mode using
>> > command `hawq init standby -n -v -M fast`. At this point as a user I
>> > understand that connections will be terminated without warning. When
>> > passing option `-M fast` hawq should not interrupt me.
>> >
>> > Basically, hawq command line is not respecting `-M fast`. Whereas hawq
>> init
>> > standby --help documents this option. Let me know if this makes sense.
>> >
>> >
>> > hawq init standby --help
>> >
>> > Usage: HAWQ management scripts options
>> >
>> >
>> > Options:
>> >
>> >   -h, --help            show this help message and exit
>> >
>> >   -a, --prompt          Execute automatically
>> >
>> >   -M STOP_MODE, --mode=STOP_MODE
>> >
>> >                         HAWQ stop mode: smart/fast/immediate
>> >
>> >
>> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <chang.lei.cn@gmail.com>
>> wrote:
>> >
>> > > Hi Shubham,
>> > >
>> > > The behavior is intentional. If there are connections when HAWQ init
>> > > standby, it is better to warn the client instead of cutting the
>> > connections
>> > > brutally.
>> > >
>> > > Cheers
>> > > Lei
>> > >
>> > >
>> > >
>> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <ssharma@pivotal.io>
>> > > wrote:
>> > >
>> > > > Hello folks,
>> > > >
>> > > > Recently observed a behaviour while re-syncing standby from hawq
>> > command
>> > > > line.
>> > > >
>> > > > Here are the reproduction steps -
>> > > >
>> > > > 1 - Open a client connection to hawq using psql
>> > > > 2 - From a different terminal run command - hawq init standby -n -v
>> -M
>> > > fast
>> > > > 3 - Standby resync fails with error
>> > > >
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There
>> are
>> > > > other connections to this instance, shutdown mode smart aborted
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either
>> > > > remove connections, or use 'hawq stop master -M fast' or 'hawq stop
>> > > > master -M immediate'
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq
>> > > > stop --help for all options
>> > > >
>> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
>> > > > connections. Aborting shutdown...
>> > > >
>> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq
>> > > > cluster failed, exit
>> > > >
>> > > > 4 - My understanding is when -M (stop mode) is passed it should
>> > terminate
>> > > > existing client connections. Also, it seems like a good practice to
>> > > > terminate client connections before standby master resync.
>> > > >
>> > > > Is this an expected behavior in hawq ? If not, I can open a JIRA and
>> > work
>> > > > on a pull request to fix this.
>> > > >
>> > > > Looking forward to your thoughts on this.
>> > > > ‚Äč
>> > > > --
>> > > > Regards,
>> > > > Shubham Sharma
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Shubham Sharma
>> > Staff Customer Engineer
>> > Pivotal Global Support Services
>> > ssharma@pivotal.io
>> > Direct Tel: +1(510)-304-8201
>> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
>> > Out of Office Hours Contact +1 877-477-2269
>> >
>>
>
>
>
> --
> Regards,
> Shubham Sharma
> Staff Customer Engineer
> Pivotal Global Support Services
> ssharma@pivotal.io
> Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
>



-- 
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message