drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Minnow Noir <minnown...@gmail.com>
Subject Re: Drill 0.9 slowness/hanging
Date Tue, 12 May 2015 00:25:08 GMT
Jason:  That fixed the web UI slowness issue.  Thank you.

On Mon, May 11, 2015 at 8:20 PM, Jason Altekruse <altekrusejason@gmail.com>
wrote:

> You can fix the Web UI slowness issue for now by deleting this jar, it was
> pulled in as a transitive dependency, but we don't actually need it and it
> is causing intermittent class-loading conflicts with classes we are
> actually using for the Web UI. As stated before, the permanent fix is
> already in master and it will not be an issue in 1.0.
>
> jars/classb/servlet-api-2.5-20081211.jar
>
> On Mon, May 11, 2015 at 5:14 PM, Andries Engelbrecht <
> aengelbrecht@maprtech.com> wrote:
>
> > You set up the Storage plugin to use the loopback NFS to read the file,
> > which is not the best way to do it.
> >
> > Why not use the MapR-FS or HDFS plug ins , this is a setup that uses the
> > MapR-FS plugin which will be considerably better than the loopback NFS
> for
> > Drill
> >
> >
> > > {
> > >  "type": "file",
> > >  "enabled": true,
> > >  "connection": “maprfs:///",
> > >  "workspaces": {
> > >    "schema1": {
> > >      "location": "/user/theUser/test",
> > >      "writable": true,
> > >      "defaultInputFormat": null
> > >    }
> > >  },
> >
> >
> > —Andries
> >
> >
> > On May 11, 2015, at 5:07 PM, Minnow Noir <minnownoir@gmail.com> wrote:
> >
> > > Hanifi:  The web UI is extremely slow, and the style sheets aren't
> > loading,
> > > but here is what I see for "show databases"
> > >
> > > Major FragmentMinor Fragments ReportingFirst StartLast StartFirst
> EndLast
> > > Endtmintavgtmaxlast updatelast progressmemmax 00-xx-xx1 /
> > > 10.213s0.213s0.670s0.670s0.457s0.457s0.457s12:15:3012:15:303MB
> > >  Major Fragment: 00-xx-xx
> > > <
> >
> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#fragment-0
> > >
> > >  Minor FragmentHostStartEndTotal TimeMax RecordsMax BatchesLast
> > UpdateLast
> > > ProgressPeak MemoryState
> > 00-00-xxip-10-101-197-197.ec2.internal0.213s0.670s
> > > 0.457s6112:15:3012:15:303MBFINISHED
> > > Operator Profiles
> > >  Overview
> > > <
> >
> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-overview
> > >
> > >  OperatorTypeSetup (min)Setup (avg)Setup (max)Process (min)Process
> > > (avg)Process
> > > (max)Wait (min)Wait (avg)Wait (max)Mem (avg)Mem (max)
> > 00-xx-00SCREEN0.000s
> > > 0.000s0.000s0.000s0.000s0.000s0.000s0.000s0.000s--
> 00-xx-01PROJECT0.422s
> > > 0.422s0.422s0.001s0.001s0.001s0.000s0.000s0.000s-- 00-xx-02
> > >
> >
> INFO_SCHEMA_SUB_SCAN0.000s0.000s0.000s0.020s0.020s0.020s0.000s0.000s0.000s
> > > 17MB17MB
> > >   00-xx-00 - SCREEN
> > > <
> >
> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-0
> > >
> > >  Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-00
> > > 0.000s0.000s0.000s16-
> > >   00-xx-01 - PROJECT
> > > <
> >
> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-1
> > >
> > >  Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-01
> > > 0.422s0.001s0.000s16-
> > >   00-xx-02 - INFO_SCHEMA_SUB_SCAN
> > > <
> >
> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-2
> > >
> > > Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem
> > > 00-00-020.000s0.020s0.000s0017MB
> > >
> > > Andries:  That storage plugin is very simple.  I've obfuscated some
> > > proprietary info.  (Note that the web UI was slow/hanging before I
> added
> > > the storage plugin.)
> > >
> > > {
> > >  "type": "file",
> > >  "enabled": true,
> > >  "connection": "file:///",
> > >  "workspaces": {
> > >    "schema1": {
> > >      "location": "/mapr/theClusterName/user/theUser/test",
> > >      "writable": true,
> > >      "defaultInputFormat": null
> > >    }
> > >  },
> > >  "formats": {
> > >    "csv": {
> > >      "type": "text",
> > >      "extensions": [
> > >        "csv"
> > >      ],
> > >      "delimiter": ","
> > >    },
> > >    "tsv": {
> > >      "type": "text",
> > >      "extensions": [
> > >        "tsv"
> > >      ],
> > >      "delimiter": "\t"
> > >    },
> > >    "parquet": {
> > >      "type": "parquet"
> > >    }
> > >  }
> > > }
> > >
> > > Unfortunately, because the Web UI is having issues, it is presenting an
> > > unstyled page with no disable links for the storage plugin so I can't
> > > disable it. With the web UI not working, I don't know how to delete the
> > > plugin.
> > >
> > > Abdel: There are no errors in the logs.
> > >
> > >
> > > On Mon, May 11, 2015 at 7:22 PM, Hanifi Gunes <hgunes@maprtech.com>
> > wrote:
> > >
> > >> I would be interested in knowing where the time has been spent. Can
> you
> > >> inspect query profile from web ui for `show databases` and let us know
> > how
> > >> the profile looks like?
> > >>
> > >> On Mon, May 11, 2015 at 4:10 PM, Abdel Hakim Deneche <
> > >> adeneche@maprtech.com>
> > >> wrote:
> > >>
> > >>> any errors in the logs ?
> > >>>
> > >>> On Mon, May 11, 2015 at 4:03 PM, Minnow Noir <minnownoir@gmail.com>
> > >> wrote:
> > >>>
> > >>>> Yes.  The bit and server are idle, with no jobs of any sort running
> on
> > >>>> them.
> > >>>>
> > >>>> I just did started sqlline and it happened again.  I had to cancel
> the
> > >>> show
> > >>>> databases command after 30s.
> > >>>>
> > >>>> sqlline version 1.1.6
> > >>>> 0: jdbc:drill:zk=theServer:5181> show databases;
> > >>>> +-------------+
> > >>>> | SCHEMA_NAME |
> > >>>> +-------------+
> > >>>> | INFORMATION_SCHEMA |
> > >>>> | cp.default  |
> > >>>> | dfs.default |
> > >>>> | dfs.root    |
> > >>>> | dfs.tmp     |
> > >>>> | test.default |
> > >>>> | test.schema1 |
> > >>>> | sys         |
> > >>>> +-------------+
> > >>>> 8 rows selected (29.264 seconds)
> > >>>>
> > >>>> On Mon, May 11, 2015 at 6:57 PM, Hanifi Gunes <hgunes@maprtech.com>
> > >>> wrote:
> > >>>>
> > >>>>> Can you confirm if this happens even when the particular 0.9
bit is
> > >>> idle?
> > >>>>>
> > >>>>> -Hanifi
> > >>>>>
> > >>>>> On Mon, May 11, 2015 at 3:53 PM, Minnow Noir <minnownoir@gmail.com
> >
> > >>>> wrote:
> > >>>>>
> > >>>>>> Experiencing some slowness after upgrading one of our servers
from
> > >>> 0.8
> > >>>> to
> > >>>>>> 0.9.  In particular, despite restarting the Drillbit, and
even the
> > >>>>> server,
> > >>>>>> several times, the Drill web UI and sqlline are both so
slow as to
> > >> be
> > >>>>>> unusable.
> > >>>>>>
> > >>>>>> For example, doing a show databases hangs indefinitely,
until I
> > >>> CTRL-C
> > >>>>> the
> > >>>>>> command (e.g., after a minute).
> > >>>>>>
> > >>>>>> show databases;
> > >>>>>> +-------------+
> > >>>>>> | SCHEMA_NAME |
> > >>>>>> +-------------+
> > >>>>>> | INFORMATION_SCHEMA |
> > >>>>>> | cp.default  |
> > >>>>>> | dfs.default |
> > >>>>>> | dfs.root    |
> > >>>>>> | dfs.tmp     |
> > >>>>>> | test.default |
> > >>>>>> | test.schema1 |
> > >>>>>> | sys         |
> > >>>>>> +-------------+
> > >>>>>> 8 rows selected (59.044 seconds)
> > >>>>>>
> > >>>>>> I tried changing schemas, but had to cancel after waiting
several
> > >>>>> minutes.
> > >>>>>>
> > >>>>>> use test.schema1;
> > >>>>>> +------------+------------+
> > >>>>>> |     ok     |  summary   |
> > >>>>>> +------------+------------+
> > >>>>>> | true       | Default schema changed to 'test.schema1'
|
> > >>>>>> +------------+------------+
> > >>>>>> 1 row selected (208.089 seconds)
> > >>>>>>
> > >>>>>> There are no errors or obvious issues in the logs. The
server has
> > >>>> plenty
> > >>>>> of
> > >>>>>> disk, CPU, and RAM free.  It's running a recent version
of MAPR
> > >>> Hadoop
> > >>>> on
> > >>>>>> CentOS 6.5 on EC2.  It feels like something is hanging
> > >>>>>>
> > >>>>>> Note: This is just after starting sqlline; no actual work,
which
> > >>> might
> > >>>>> have
> > >>>>>> slowed things down, has happened yet.
> > >>>>>>
> > >>>>>> This is what's running from a Drill perspective:
> > >>>>>>
> > >>>>>> ps aux | grep drill
> > >>>>>> ec2-user 24662  0.0  0.0 106100  1328 pts/0    S    12:29
  0:00
> > >> bash
> > >>>>>> bin/drillbit.sh internal_start drillbit
> > >>>>>> ec2-user 24697  0.3  2.3 6486240 730568 pts/0  Sl   12:29
  0:36
> > >>>>>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/bin/java
> > >>>>>> -Dlog.path=/opt/mapr/drill/drill-0.9.0/logs/drillbit.log
-Xms1G
> > >>> -Xmx4G
> > >>>>>> -XX:MaxDirectMemorySize=8G -XX:MaxPermSize=512M
> > >>>>>> -XX:ReservedCodeCacheSize=1G -ea
> > >>>>>> -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
> > >>>>>> -Dzookeeper.sasl.client=false -XX:+CMSClassUnloadingEnabled
> > >>>>>> -XX:+UseConcMarkSweepGC -cp
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> /opt/mapr/drill/drill-0.9.0/conf:/opt/mapr/drill/drill-0.9.0/jars/*:/opt/mapr/drill/drill-0.9.0/jars/ext/*:/opt/mapr/drill/drill-0.9.0/jars/3rdparty/*:/opt/mapr/drill/drill-0.9.0/jars/classb/*
> > >>>>>> org.apache.drill.exec.server.Drillbit
> > >>>>>>
> > >>>>>> Drill has been started thusly:
> > >>>>>>
> > >>>>>> bin/sqlline -u jdbc:drill:zk=theServer:5181
> > >>>>>>
> > >>>>>> Any idea how I could troubleshoot this?
> > >>>>>>
> > >>>>>> Anyone else experiencing slowness after upgrading?
> > >>>>>>
> > >>>>>> Thank you
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Abdelhakim Deneche
> > >>>
> > >>> Software Engineer
> > >>>
> > >>>  <http://www.mapr.com/>
> > >>>
> > >>>
> > >>> Now Available - Free Hadoop On-Demand Training
> > >>> <
> > >>>
> > >>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >>>>
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message