spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik O'Shaughnessy" <>
Subject Integrating a Native ARchive (NAR) into Apache Spark
Date Mon, 20 Jun 2016 19:33:52 GMT

I’ve cobbled together a JNI interface and packaged it with the nar-maven-plugin, but I’m
out of my depth with Java/Scala/Maven and could use some pointers about how to make my new
object available for import in spark-shell.

My NAR project builds a simple Foo object that wraps some native libfoo calls. The unit tests
for the NAR indicate that my C APIs are being called successfully. I installed it into my
local .m2
repository cache and was able to include it as a dependency in a simple test project which
and calls my shared library.  So far so good.


I’ve tried adding Foo as a dependency in spark/pom.xml and adding nar-maven-plugin to the
section of the top level POM:



I had to do some hacking on nar-maven-plugin to teach it about my platform/os/linker, which
is the reason for the goofy nar-maven-plugin version number. 

A spark build succeeds, but:

scala> import
<console>:23: error: object Foo is not a member of package

I tried adding my Foo maven project as a module in the top-level POM and moving my project
into the Apache Spark tree.  It built everything including my Foo project, however the Foo
object wasn’t available in spark-shell like I had hoped.

I’m a totally neophyte when it comes to the Maven ecosystem, however I’ve come a long
way in
the last two weeks. I know it’s a weird request, but I would be most appreciative if someone
could provide some gentle hints about how to make my Foo object available within spark-shell.

Best regards,

Erik O'Shaughnessy - Principal Software Engineer
Strategic Application Engineering
Austin, TX 512-401-1070

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message