nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Asynchronous JMS Consumer for IBM MQ
Date Thu, 05 Nov 2015 16:35:04 GMT
Alex,

Generally the threading in NiFi is handled by the concurrent tasks
configured for the processor, meaning more threads able to call the
onTrigger method. For some source/sink type processors, which sounds like
your processor, it can be acceptable to create a controlled number of
threads. There are a couple of processors that take the approach of
starting a background thread in an @OnScheduled method to put messages onto
a queue that are then pulled off for processing in the onTrigger method. A
good example to look at might be the GetTwitter processor which has a
BlockingQueue of messages. You will definitely want to set a capacity on
the queue to prevent out-of-memory scenarios if a high volume of messages
are coming through.

Let us know if this helps, or if there is any more information we can
provide.

-Bryan


On Thu, Nov 5, 2015 at 9:49 AM, McCullough, Alex <
Alex.McCullough@capitalone.com> wrote:

> Hey There,
>
> This is a bit of a generic question, and I am not sure if this is the
> right place to start but I am new to developing custom processors in Nifi
> but have been trying to write an asynchronous message consumer to consume
> from IBM MQ.
>
> I have read all of the developer documentation and have been testing a
> processor I built, but I think where I am struggling is conceptualizing
> exactly how I should set this up within the framework and how the threading
> within Nifi works exactly.
>
> Typically I am used to registering a listener and using the onMessage
> method to add messages that are received to local queue to then be
> processed. What would this look like within the Nifi framework? Are there
> any guiding principals you can point me towards or any source code that may
> be doing something somewhat similar?
>
> Sorry this is a pretty vague question and I’m sure there are many ways to
> skin this particular cat, but hoping you can help point me generally down
> the right path.
>
> Thanks a ton and please let me know if you want me to provide anymore
> detail!
>
> Thanks,
> Alex
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message