hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Sharma <ssha...@pivotal.io>
Subject Re: Default LOG level for "data locality ratio:" events contribute to Hawq Master log bloat
Date Sat, 02 Dec 2017 04:08:04 GMT
Congratulations on your first PR Kyle. Welcome to the community.

On Fri, Dec 1, 2017 at 5:53 PM, Kyle Roberts <kroberts@pivotal.io> wrote:

> Created JIRA: https://issues.apache.org/jira/browse/HAWQ-1560
>
> Submitted pull request: https://github.com/apache/incubator-hawq/pull/1318
>
> - Kyle
>
>
>
> On Fri, Dec 1, 2017 at 4:47 PM, Kyle Roberts <kroberts@pivotal.io> wrote:
>
> > Hi everyone,
> >
> > I will be creating a JIRA and submitting a pull request for this.
> >
> > Thanks,
> >
> > - Kyle
> >
> > On Thu, Nov 30, 2017 at 10:52 PM, 刘奎恩(局外) <kuien.lke@alibaba-inc.com>
> > wrote:
> >
> >>
> >> It makes sense to adjust its LOG level to reduce log size on master.
> >> 1. It affects master performance if log file size is
> >> huge.Query on hawq_toolkit.hawq_log_master_concise is slower
> >>  as time going.
> >> https://issues.apache.org/jira/projects/HAWQ/issues/HAWQ-1550
> >> 2. It has little value when segments are seperated from storage nodes.If
> >> storage nodes and computing nodes are geographically distributed,it is
> >> common on Cloud service, and no distributed cache systems are
> >> available,then we have little interest to keep them in log in such
> cases.
> >>
> >> data locality ratio
> >> -------------——
> >> Kuien Liu/奎恩
> >> ------------------------------------------------------------
> ------发件人:Kyle
> >> Roberts <kroberts@pivotal.io>发送时间:2017年12月1日(星期五)
05:41收件人:dev <
> >> dev@hawq.incubator.apache.org>主 题:Default LOG level for "data locality
> >> ratio:" events contribute to Hawq Master log bloat
> >> Issue:
> >>
> >>
> >> Default LOG level for "data locality ratio:" events contribute to Hawq
> >> Master log bloat.  Should only be needed when investigating/tuning
> >> performance.
> >>
> >> Case Scenario:
> >>
> >> *"data locality ratio:"* events can report multiple times and, depending
> >> upon the query, many times.
> >>
> >>
> >> Sample event:
> >>
> >>
> >> 2017-11-30 20:33:39.857251
> >> GMT,"gpadmin","testdb",p280349,th-909919968,"[local]",,2017-11-30 20
> >> :06:35
> >> GMT,1630750,con25419,cmd28,seg-1,,,,sx2,"LOG","00000","data
> >> locality ratio:
> >> 1.000; virtual segment number: 1; different host number: 1;
> >> virtual segment
> >> number per host(avg/min/max): (1/1/1); segment size(avg/min/
> >> max): (1016.000
> >> B/1016 B/1016 B); segment size with penalty(avg/min/max): (1
> >> 016.000 B/1016
> >> B/1016 B); continuity(avg/min/max): (1.000/1.000/1.000).",,,,,"SQL
> >> statement ""select count(*)::float4 from public.test_tbl as
> >> Ta where Ta.id
> >> is null""","analyze test_tbl;",0,,"cdbdatalocality.c",3372,
> >>
> >>
> >>
> >> For example, Hawq master logs are seen with close to *20% of
> >>  the file size*
> >> is from only the *"data locality ratio:" *events.
> >>
> >>
> >> Product Enhancement suggestion:
> >>
> >>
> >> Because these are only needed for performance tuning (or for
> >>  investigation):
> >>
> >>
> >> In:
> >>
> >>
> >> https://github.com/apache/incubator-hawq/blob/master/src/bac
> >> kend/cdb/cdbdatalocality.c#L3372
> >>
> >>
> >> Could we change default LOG level to something like DEBUG1, etc?
> >>
> >>
> >>
> >>
> >> - Kyle
> >>
> >
> >
> >
> > --
> > Kyle Roberts  |  Staff Customer Engineer  |  650-846-1667
> > <(650)%20846-1667>
> > Support.Pivotal.io
> > <http://www.google.com/url?q=http%3A%2F%2Fsupport.pivotal.
> io%2F&sa=D&sntz=1&usg=AFQjCNGDBr_XSKC18wot5h3OkKoZ84Vn7Q>
> >   |  Mon-Fri  9:00am to 5:00pm PST  |  1-877-477-2269 <(877)%20477-2269>
> > [image: support]
> > <https://www.google.com/url?q=https%3A%2F%2Fsupport.pivotal.
> io%2F&sa=D&sntz=1&usg=AFQjCNEvwKLjzu29inKwy4jJjKsboqGMCg>
> >  [image: twitter]
> > <https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%
> 2Fpivotal&sa=D&sntz=1&usg=AFQjCNG1FcqkH5ghKsSG6UkdeUzjSuDSHg>
> >  [image: linkedin]
> > <https://www.google.com/url?q=https%3A%2F%2Fwww.linkedin.
> com%2Fcompany%2F3048967&sa=D&sntz=1&usg=AFQjCNHOQGYmDYIQz06S3-vAuqzf8bN8Yw
> >
> >  [image: facebook]
> > <https://www.google.com/url?q=https%3A%2F%2Fwww.facebook.
> com%2Fpivotalsoftware&sa=D&sntz=1&usg=AFQjCNFQnPFtec1Rp3lKf6MuY1jcbA8j2A>
> >  [image: google plus] <https://plus.google.com/+Pivotal> [image:
> youtube]
> > <https://www.youtube.com/playlist?list=PLAdzTan_
> eSPScpj2J50ErtzR9ANSzv3kl>
> >
>
>
>
> --
> Kyle Roberts  |  Staff Customer Engineer  |  650-846-1667
> Support.Pivotal.io
> <http://www.google.com/url?q=http%3A%2F%2Fsupport.pivotal.
> io%2F&sa=D&sntz=1&usg=AFQjCNGDBr_XSKC18wot5h3OkKoZ84Vn7Q>
>   |  Mon-Fri  9:00am to 5:00pm PST  |  1-877-477-2269
> [image: support]
> <https://www.google.com/url?q=https%3A%2F%2Fsupport.pivotal.
> io%2F&sa=D&sntz=1&usg=AFQjCNEvwKLjzu29inKwy4jJjKsboqGMCg>
>  [image: twitter]
> <https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%
> 2Fpivotal&sa=D&sntz=1&usg=AFQjCNG1FcqkH5ghKsSG6UkdeUzjSuDSHg>
>  [image: linkedin]
> <https://www.google.com/url?q=https%3A%2F%2Fwww.linkedin.
> com%2Fcompany%2F3048967&sa=D&sntz=1&usg=AFQjCNHOQGYmDYIQz06S3-vAuqzf8bN8Yw
> >
>  [image: facebook]
> <https://www.google.com/url?q=https%3A%2F%2Fwww.facebook.
> com%2Fpivotalsoftware&sa=D&sntz=1&usg=AFQjCNFQnPFtec1Rp3lKf6MuY1jcbA8j2A>
>  [image: google plus] <https://plus.google.com/+Pivotal> [image: youtube]
> <https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl>
>



-- 
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message