drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: Unit testing a simple function
Date Wed, 24 Jan 2018 11:18:55 GMT
Paul, Charles,

Thanks for the pointers.
These really helped in getting started.

I'm trying to fully understand the (simple) UDF Charles created a while ago
for a java library of mine
https://github.com/cgivre/drill-useragent-function

I noticed that it has not been updated for a long time so I decided to pull
it into my project.
This way I have a reason to dive into Drill and get to know it a bit better
and to make the UDF better maintained.
I already have UDFs for Pig, Hive, Flink and Beam so Drill would be a
sensible addition.

So as a first step I pulled the existing code in an tried to create a few
tests based on the examples and documentation you guys sent me.

Work in progress (which fails on the tests):
https://github.com/nielsbasjes/yauaa/tree/DrillUDF/udfs/drill

Some of the things I ran into that confuse me:
*1)* For some reason the netty version packaged with Drill 1.12.0
consistently fails with this error at the root cause when trying to run
something:
*Caused by: java.lang.NoSuchMethodError:
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolThreadCache;*

I found that the io.netty.buffer.PooledByteBufAllocatorL inherits
from io.netty.buffer.PooledByteBufAllocator which in the version I get from
the drill dependencies does not have that method.
My workaround was to exclude 'netty-all' from the 'drill-java-exec'
dependency and re-add 'netty-all' version 4.0.28.Final to my project.
This works but feels bad.

My first impression is that this is a bug in Drill 0.12.0.

*2)* I always see this warning when running:
11:49:38,556 [WARN ] GuavaPatcher                            :   40: Unable
to patch Guava classes.
javassist.CannotCompileException: by java.lang.LinkageError: loader
(instance of  sun/misc/Launcher$AppClassLoader): attempted  duplicate class
definition for name: "com/google/common/base/Stopwatch"
at javassist.ClassPool.toClass(ClassPool.java:1099)
at javassist.ClassPool.toClass(ClassPool.java:1042)
at javassist.ClassPool.toClass(ClassPool.java:1000)
at javassist.CtClass.toClass(CtClass.java:1140)
at
org.apache.drill.exec.util.GuavaPatcher.patchStopwatch(GuavaPatcher.java:66)
at org.apache.drill.exec.util.GuavaPatcher.patch(GuavaPatcher.java:36)


My project does not include any guava so this also seems like a drill issue.

*3)* In the unit test examples I see input/output using simple types like
float and int.
I could really use examples that show me how to do the same things with
NullableVarCharHolder, BaseWriter.ComplexWriter.
Especially how do I create an instance of DrillBuf to set the input of a
(Nullable)VarCharHolder?

So far I have downloaded the drill codebase to figure out how to do this
but I have to say that at this point it is still very hard.


Any help with these point is greatly appreciated.
Thanks.

Niels Basjes


On Mon, Jan 22, 2018 at 4:35 PM, Paul Rogers <par0328@yahoo.com.invalid>
wrote:

> Hi Niels,
>
> You can find detailed suggestions here: https://github.com/paul-
> rogers/drill/wiki/UDFs-Background-Information
>
> In particular, see the page on debugging UDFs.
>
> - Paul
>
> Sent from my iPhone
>
> > On Jan 22, 2018, at 7:09 AM, Charles Givre <cgivre@gmail.com> wrote:
> >
> > Hi Niels,
> > Take a look at this file:
> >
> > https://github.com/cgivre/drill/blob/67804e9d8a4634df8dbf60e848533a
> 62d64dee64/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/
> TestNetworkFunctions.java <https://github.com/cgivre/drill/blob/
> 67804e9d8a4634df8dbf60e848533a62d64dee64/exec/java-exec/src/
> test/java/org/apache/drill/exec/fn/impl/TestNetworkFunctions.java>
> >
> > This should be a pretty good template for unit tests for a function.
> > — C
> >
> >> On Jan 22, 2018, at 10:01, Niels Basjes <Niels@basjes.nl> wrote:
> >>
> >> Hi,
> >>
> >> I was reading through the tutorial page on how to write a custom
> function
> >> for Drill and I've alos looked at some functions I found on the
> internet.
> >>
> >> There is one thing I would really like to know that I have not yet
> found so
> >> far: How do I unut test if I did it right?
> >>
> >> Especially the effects of the various annotations and the fact that you
> >> need to specify all classes "fully" without an import.
> >>
> >> --
> >> Best regards / Met vriendelijke groeten,
> >>
> >> Niels Basjes
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message