drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khurram Faraaz <kfar...@mapr.com>
Subject Re: Heap memory - leak
Date Wed, 12 Jun 2019 19:56:40 GMT
Hi Dweep,

I see that you are using the REST API to execute the queries. The
recommended way for complex queries that are resource intensive, is to
always use either the JDBC/ODBC driver to submit queries for execution.
Here is a difference for your reference. Please use JDBC/ODBC to execute
your queries and let us know if that resolves the issue for you.

*REST API* is a stateless protocol so when a query request is submitted by
REST client then the request thread on server side will block until the
query is fully executed. When all the results are available on server side,
it is converted into strings to be sent back to the REST client. So if we
have query results very big that will end up consuming lots of memory on
the server side and at the same it will be difficult for certain REST client
(like browsers) to render all those requests at once. Due to this we see
unresponsive browser and high heap usage in Drillbit as well. Basically
there is lack of pagination kind of protocol because of statelessness.
Because of this limitation we always suggest to use RESTapi for smaller
footprint query or running limit query which can result in smaller number
of memory footprint.''

*JDBC/ODBC API* on the other hand is stateful protocol which means that
client side has some context when a query is actually completed and can get
results in incremental order. So using JDBC api Drill client doesn't have
to wait for receiving all the results row from server side before sending
to application. It can start sending results whenever the first batch is
available and JDBC provides api to receive incremental results (next call).
So the problems related to higher memory footprint to get all the results
row at once is not there. Now if JDBC application is pulling data slowly
then there are feedback mechanism built in which will affect the execution
of query too. It can tell server side to stop the query pipeline from
sending more results since application is not pulling it fast enough. Hence
there is feedback mechanism which comes into play and all these are because
of statefulness of the JDBC/ODBC side of protocol.

Thanks,
Khurram

On Fri, Jun 7, 2019 at 4:50 PM Paul Rogers <par0328@yahoo.com.invalid>
wrote:

> Hi Dweep,
>
> This mailing list does not support attachments. Consider filing a JIRA
> ticket and attaching your images there: [1]
>
> You mention you've assigned Drill 14 GB of heap. You also mention that
> your task ran out of heap. As it turns out, Drill also uses direct memory
> to store intermediate data. I wonder if the error condition is actually
> about direct memory. How much direct memory have you given to Drill?
>
> 14 GB of heap and the default direct memory (8GB) should be plenty for a
> query that produces a 48 MB Parquet file: assuming that the input size is
> similar: ~200 MB (uncompressed JSON).
>
>
> You mention that you run the JSON-to-Parquet conversion once per hour. Do
> you use this Drill instance for any other tasks? Are there other tasks
> running at the same time? How many nodes of Drill are in use?
>
>
> Finally, you mention you use the REST API. Perhaps something odd is
> happening there. A stack trace of your error would help. The stack trace
> may be in the error message, or in the Drill log file.
>
>
> Thanks,
> - Paul
>
> [1] https://issues.apache.org/jira
>
>
>     On Friday, June 7, 2019, 2:36:36 AM PDT, Dweep Sharma <
> dweep.sharma@redbus.com> wrote:
>
>  Hi Divya,
>
> The size is 48 MB (after converting to Parquet)
>
>
>
>
>
> On Fri, Jun 7, 2019 at 1:45 PM Divya Gehlot <divinedivyag@gmail.com>
> wrote:
>
> Can you share the more details .
> Query profile and other aspects like data size and all to have better view
> what’s happening
>
>
> Thanks ,
> Divya
>
> On Fri, 7 Jun 2019 at 4:13 PM, Dweep Sharma <dweep.sharma@redbus.com>
> wrote:
>
> > Data is in JSON format.
> >
> > On Fri, Jun 7, 2019 at 1:39 PM Dweep Sharma <dweep.sharma@redbus.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have a memory leak issue. 14GB memory is assigned to heap but it gets
> > > full within a day with just one cron running.
> > >
> > > Task is a CTAS query from Kafka to S3 once every hour. CTAS is issued
> via
> > > the Drill Rest API.
> > >
> > > Please assist on a resolution.
> > >
> > > -Dweep
> > >
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message