drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: Profiles Gone in Web UI: The great profile heist
Date Wed, 01 Jun 2016 16:23:36 GMT
Thank you, I +1ed the JIRA, and added my user list comments. This should be
easily reproducible and fixable.



On Wed, Jun 1, 2016 at 11:17 AM, Krystal Nguyen <knguyen@maprtech.com>
wrote:

> Looks like a jira was opened a while back (status is unresolved) regarding
> similar issue you are seeing:
> https://issues.apache.org/jira/browse/DRILL-2593
>
> On Wed, Jun 1, 2016 at 5:52 AM, John Omernik <john@omernik.com> wrote:
>
> > Abdel and Jacques -
> >
> > It is absolutely a bad profile, however, Jacques, it was not a empty file
> > profile, instead, it was a profile where the write didn't complete
> (likely
> > due to my bad cluster state post).
> >
> > I was able to reproduce it synthetically like this, say you had a profile
> > that ended similar to this:
> >
> > ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> > 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
> >
> > (The ... represents the part I removed for brevity in this email, the
> rest
> > of the profile is below)
> >
> > We can reproduce the UI profiles issue by removing ,"user":"adm"}
> >
> > so it ends like this:
> >
> > ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> > 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}]
> >
> >
> > Let me know if you know of a JIRA, otherwise I'll open one, this is a
> pain
> > for users, because you have to go through an validate the json on every
> > profile to find the one that is breaking your UI.
> >
> > John
> >
> >
> >
> >
> >
> > Full "Working" Profile:
> >
> > {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> > "start":1464730828036,"end":1464730828131,"query":"alter session set
> > `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> > zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> > dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> > fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> > state":3,"minorFragmentId":0,"operatorProfile":[{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> > peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> > peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> > 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> > 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> > maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> > ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
> >
> >
> > Full "Broken" Profile:
> >
> > {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> > "start":1464730828036,"end":1464730828131,"query":"alter session set
> > `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> > zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> > dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> > fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> > state":3,"minorFragmentId":0,"operatorProfile":[{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> > peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> > peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> > 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> > 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> > maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> > ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}]
> >
> >
> > On Wed, Jun 1, 2016 at 6:22 AM, John Omernik <john@omernik.com> wrote:
> >
> > > I will walk through and look for the corrupt profile.  This could be
> the
> > > result of my "cluster in a bad state problem" (actually, it IS the
> result
> > > of that, I have Ben doing testing to recreate the issue). Is there a
> JIRA
> > > for this ? I agree the UI should handle this.
> > >
> > >
> > > On Tuesday, May 31, 2016, Jacques Nadeau <jacques@dremio.com> wrote:
> > >
> > >> Odds are one of your profiles is corrupt. Last I checked, there was a
> > bug
> > >> that a corrupt profile would cause no profiles to show up in list.
> Check
> > >> if
> > >> any of your profiles are zero bytes and delete them.
> > >>
> > >> The UI should really handle this.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Tue, May 31, 2016 at 12:49 PM, John Omernik <john@omernik.com>
> > wrote:
> > >>
> > >> > I am scratching my head at this one... I made some minor changes to
> my
> > >> > drill-env.sh to enable gclogging, and was using the profiles in the
> > >> webui
> > >> > just fine.  Due to some previously mentioned issues, I've had to
> > restart
> > >> > drill bits due to GC issues etc.
> > >> >
> > >> > Now, while my profiles directory still exists, and my
> > >> drill-override.conf
> > >> > has not been changed, no profiles now show up in the webui, even
> after
> > >> > drillbit restarts, and running more queries... The profiles are
> still
> > >> being
> > >> > created (I can see them being added to the same profiles directory)
> > just
> > >> > nothing shows up in the Web UI...
> > >> >
> > >> > What could be happening here?
> > >> >
> > >> > *scratching my head
> > >> >
> > >>
> > >
> > >
> > > --
> > > Sent from my iThing
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message