drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hanifi Gunes <hgu...@maprtech.com>
Subject Re: Max Buffer Size Reached
Date Mon, 08 Feb 2016 19:05:02 GMT
Thanks for the feedback. Yep my answer seems very much dev focused than
user.

The error is manifestation of extremely wide columns in your dataset. I
would recommend splitting the list if that's an option.

Assuming the problem column is a list of integers as below

{
"wide": [1,2,.....N]
}

after splitting it should look like

{
"wide0": [1,2,.....X],
"wide1": [Y,.......Z]
...
"wideN": [T,.......N]
}

Sounds like good idea to enhance the error reporting with file & column
name. Filed [1] to track this.

Thanks.

1: https://issues.apache.org/jira/browse/DRILL-4371


On Fri, Feb 5, 2016 at 6:28 PM, John Omernik <john@omernik.com> wrote:

> Excuse my basic questions, when you say we are you reference Drill coders?
> So what is Integer.MAX_VALUE bytes? Is that a query time setting? Drillbit
> setting? Is it editable?  How does that value get interpreted for complex
> data types (objects and arrays).
>
> Not only would the column be helpful, but the source file as well.  ( if
> this is an individual record issue....or is this a cumulative error where
> the max size of the sum of the length of multiple records of a column is at
> issue).
>
>
> Thoughts on how as a user I could address this in my dataset?
>
> Thanks!
>
> On Friday, February 5, 2016, Hanifi Gunes <hgunes@maprtech.com> wrote:
>
> > You see this exception because one of the columns in your dataset is
> larger
> > than an individual DrillBuf could store. The hard limit
> > is Integer.MAX_VALUE bytes. Around the time we are trying to expand one
> of
> > the buffers, we notice the allocation request is oversized and fail the
> > query. It would be nice if error message contained the column that raised
> > this issue though.
> >
> > On Fri, Feb 5, 2016 at 1:39 PM, John Omernik <john@omernik.com
> > <javascript:;>> wrote:
> >
> > > Any thoughts on how to troubleshoot this (I have some fat json data
> going
> > > into the buffers apparently) It's not huge data, just wide/complex
> (total
> > > size is 1.4 GB)  Any thoughts on how to troubleshoot or settings I can
> > use
> > > to work through these errors?
> > >
> > >
> > > Thanks!
> > >
> > >
> > > John
> > >
> > >
> > >
> > >
> > > Error: SYSTEM ERROR: OversizedAllocationException: Unable to expand the
> > > buffer. Max allowed buffer size is reached.
> > >
> > >
> > >
> > > Fragment 1:11
> > >
> > >
> > >
> > > [Error Id: db21dea0-ddd7-4fcf-9fea-b5031e358dad on node1
> > >
> > >
> > >
> > >   (org.apache.drill.exec.exception.OversizedAllocationException) Unable
> > to
> > > expand the buffer. Max allowed buffer size is reached.
> > >
> > >     org.apache.drill.exec.vector.UInt1Vector.reAlloc():214
> > >
> > >
>  org.apache.drill.exec.vector.UInt1Vector$Mutator.setValueCount():469
> > >
> > >
> > >
> >
> org.apache.drill.exec.vector.complex.ListVector$Mutator.setValueCount():324
> > >
> > >     org.apache.drill.exec.physical.impl.ScanBatch.next():247
> > >
> > >     org.apache.drill.exec.record.AbstractRecordBatch.next():119
> > >
> > >     org.apache.drill.exec.record.AbstractRecordBatch.next():109
> > >
> > >
>  org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> > >
> > >
> > >
> > >
> >
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132
> > >
> > >     org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > >
> > >     org.apache.drill.exec.record.AbstractRecordBatch.next():119
> > >
> > >
> > >
> > >
> >
> org.apache.drill.exec.test.generated.StreamingAggregatorGen1931.doWork():172
> > >
> > >
> > >
> > >
> >
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():167
> > >
> > >     org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > >
> > >     org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> > >
> > >
> > >
> > >
> >
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
> > >
> > >     org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> > >
> > >     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():256
> > >
> > >     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():250
> > >
> > >     java.security.AccessController.doPrivileged():-2
> > >
> > >     javax.security.auth.Subject.doAs():415
> > >
> > >     org.apache.hadoop.security.UserGroupInformation.doAs():1595
> > >
> > >     org.apache.drill.exec.work.fragment.FragmentExecutor.run():250
> > >
> > >     org.apache.drill.common.SelfCleaningRunnable.run():38
> > >
> > >     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> > >
> > >     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> > >
> > >     java.lang.Thread.run():745 (state=,code=0)
> > >
> >
>
>
> --
> Sent from my iThing
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message