nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Ghigiarelli <briang...@gmail.com>
Subject Re: Class Loading Conflicts - Different JAR Versions
Date Tue, 27 Mar 2018 01:13:43 GMT
Thanks for the clarification, Mike.

My primary motivation for this upgrade is to support an aggregation
pipeline with the $out stage. Support for this in the Java driver was
introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior to
this, I believe you had to use the cursor to query all of the results,
which suffers from both query performance issues and concurrency issues for
the data flow if we're trying to overwrite an entire collection at once.
That said, if I'm missing something, I'm happy to learn!

Thanks again,
Brian

On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen <mikerthomsen@gmail.com> wrote:

> I was just following a convention at the time. So unless something breaks
> because of the move, I don't see any reason why it would be an issue to
> move it.
>
> In fact, with NIFI-4325 the only reason I put the new ElasticSearch
> service/api in the standard-services segment of the code base was I ran
> into a bizarre issue at runtime where I could define the controller
> service, but the processor had no idea it was there. So that too might
> warrant some extra eyes now that #4325 was added to master today.
>
> FWIW, I can't think of any reason why the MongoDB client service needs 3.4
> or 3.6 over 3.2.
>
> Brian, feel free to ping me on the review.
>
> Thanks,
>
> Mike
>
> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess <mattyb149@apache.org>
> wrote:
>
> > Brian (this one's all you ;),
> >
> > I think that PR is quite welcome in my opinion, but I'd like to get
> > Mike Thomsen's and others' opinions on the subject too, I think Mike
> > wrote the original service (or at the least is very knowledgable about
> > Mongo and NiFi), he might have run into issues that led him to put it
> > there for a reason. If not, let's do it, if you're willing to submit
> > the contribution I/we'd be more than happy to review/incorporate it :)
> >
> > Regards,
> > Matt
> >
> >
> > On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli <brianghig@gmail.com>
> > wrote:
> > > Matt,
> > >
> > > +1 [non-binding] for the idea to move the Mongo dependencies into that
> > > bundle. I think it will likely need the NAR dependency on
> > > nifi-standard-services-api in order to provide a link to the SSL
> Context
> > > Service. Is that ticket / PR worthy as a one-off in the short term? No
> > > doubt the Extension Registry will be a powerful capability.
> > >
> > > Also, +2 for the greeting, though Bryan may choose to put the "y"
> first.
> > :-)
> > >
> > > Thanks,
> > > Brian
> > >
> > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess <mattyb149@apache.org>
> > wrote:
> > >
> > >> Bri/yan,
> > >>
> > >> IMO I think we'd be better off with moving the
> > >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into
> > >> the nifi-mongodb-bundle. That's something I missed while reviewing the
> > >> PR that put them in standard services. I don't see a direct dependency
> > >> on nifi-standard-services, and if the nifi-mongodb-client-service-api
> > >> needs the nifi-standard-services-api as a dependency, it can make it a
> > >> NAR dependency.
> > >>
> > >> We might want to visit this as a much broader case, since I believe we
> > >> could run into this with other services APIs (Elasticsearch, HBase,
> > >> etc.)? Certainly when the Extension Registry becomes a thing, we might
> > >> not want "external" service APIs to be part of
> > >> nifi-standard-services-api.
> > >>
> > >> Regards,
> > >> Matt
> > >>
> > >>
> > >> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli <
> brianghig@gmail.com
> > >
> > >> wrote:
> > >> > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar
> as
> > a
> > >> > parent to use the standard SSLContextService. Sounds like upgrading
> > the
> > >> > mongo-java-driver for the base build is the way to go for us.
> > >> >
> > >> > Thanks again,
> > >> > Brian
> > >> > On Mon, Mar 26, 2018 at 15:29 Bryan Bende <bbende@gmail.com>
wrote:
> > >> >
> > >> >> Brian,
> > >> >>
> > >> >> Is your custom processor using the MongoDBClientService provided
by
> > >> >> NiFI's standard services API? or does your NAR have a parent of
> > >> >> nifi-standard-services-api-nar to use other services?
> > >> >>
> > >> >> Looking at where the Mongo JARs are from a build of master...
> > >> >>
> > >> >> find work/nar/ -name *mongo-java*.jar
> > >> >>
> > >> >>
> > >> work/nar//extensions/nifi-standard-services-api-nar-1.6.
> > 0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/
> > mongo-java-driver-3.2.2.jar
> > >> >>
> > >> >>
> > >> work/nar//extensions/nifi-mongodb-services-nar-1.6.0-
> > SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/
> > mongo-java-driver-3.2.2.jar
> > >> >>
> > >> >>
> > >> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.
> > nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar
> > >> >>
> > >> >> I think the issue is that if your NAR has
> > >> >> nifi-standard-services-api-nar as a parent NAR (which it probably
> > does
> > >> >> to use SSLContext service, or any other standard service) then
you
> > are
> > >> >> getting mongo-java-driver-3.2.2 from MongoDBClientService since
we
> > >> >> have parent first class loading.
> > >> >>
> > >> >> Given this situation I think there are only two options... not
> having
> > >> >> nifi-standard-services-api-nar as a parent (which stops you from
> > using
> > >> >> all standard services), or upgrading the version of
> mongo-java-driver
> > >> >> used by MongoDBClientService.
> > >> >>
> > >> >> -Bryan
> > >> >>
> > >> >>
> > >> >> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli <
> > brianghig@gmail.com
> > >> >
> > >> >> wrote:
> > >> >> > Hey all,
> > >> >> >
> > >> >> > Is there a means of troubleshooting a custom NAR that seems
to be
> > >> having
> > >> >> > runtime conflicts by picking up an older JAR that's provided
by
> the
> > >> NiFi
> > >> >> > standard bundle?
> > >> >> >
> > >> >> > In this particular case, I'm using some custom processors
that
> are
> > >> built
> > >> >> > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3.
At
> > runtime,
> > >> >> I'm
> > >> >> > seeing the processor use classes from MongoDB 3.2.2 that's
> > provided by
> > >> >> > NiFi. Both NiFi and the custom NAR compile successfully on
their
> > own,
> > >> but
> > >> >> > using the custom NAR in NiFi causes NoSuchMethodError's due
to a
> > new
> > >> >> method
> > >> >> > only available in MongoDB 3.4+.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Brian
> > >> >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message