drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hanifi Gunes <hgu...@maprtech.com>
Subject Re: Max Buffer Size Reached
Date Fri, 05 Feb 2016 22:24:48 GMT
You see this exception because one of the columns in your dataset is larger
than an individual DrillBuf could store. The hard limit
is Integer.MAX_VALUE bytes. Around the time we are trying to expand one of
the buffers, we notice the allocation request is oversized and fail the
query. It would be nice if error message contained the column that raised
this issue though.

On Fri, Feb 5, 2016 at 1:39 PM, John Omernik <john@omernik.com> wrote:

> Any thoughts on how to troubleshoot this (I have some fat json data going
> into the buffers apparently) It's not huge data, just wide/complex (total
> size is 1.4 GB)  Any thoughts on how to troubleshoot or settings I can use
> to work through these errors?
>
>
> Thanks!
>
>
> John
>
>
>
>
> Error: SYSTEM ERROR: OversizedAllocationException: Unable to expand the
> buffer. Max allowed buffer size is reached.
>
>
>
> Fragment 1:11
>
>
>
> [Error Id: db21dea0-ddd7-4fcf-9fea-b5031e358dad on node1
>
>
>
>   (org.apache.drill.exec.exception.OversizedAllocationException) Unable to
> expand the buffer. Max allowed buffer size is reached.
>
>     org.apache.drill.exec.vector.UInt1Vector.reAlloc():214
>
>     org.apache.drill.exec.vector.UInt1Vector$Mutator.setValueCount():469
>
>
> org.apache.drill.exec.vector.complex.ListVector$Mutator.setValueCount():324
>
>     org.apache.drill.exec.physical.impl.ScanBatch.next():247
>
>     org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
>     org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
>     org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
>
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132
>
>     org.apache.drill.exec.record.AbstractRecordBatch.next():162
>
>     org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
>
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen1931.doWork():172
>
>
>
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():167
>
>     org.apache.drill.exec.record.AbstractRecordBatch.next():162
>
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():104
>
>
>
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
>
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():94
>
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():256
>
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():250
>
>     java.security.AccessController.doPrivileged():-2
>
>     javax.security.auth.Subject.doAs():415
>
>     org.apache.hadoop.security.UserGroupInformation.doAs():1595
>
>     org.apache.drill.exec.work.fragment.FragmentExecutor.run():250
>
>     org.apache.drill.common.SelfCleaningRunnable.run():38
>
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
>
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
>
>     java.lang.Thread.run():745 (state=,code=0)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message