nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@gmail.com>
Subject Re: Common scheduler and add-hock thread creation
Date Tue, 17 Nov 2015 01:33:01 GMT
Is there a common use case for a single extra thread? I'm asking in ignorance but if the answer
is yes then I'd think there'd be best practices around that?

Regards,
Matt

Sent from my iPhone

> On Nov 16, 2015, at 8:22 PM, Tony Kurc <trkurc@gmail.com> wrote:
> 
> so, I believe threads in a processor in nifi are much, much easier than
> general threading in many other applications. There are defined boundaries
> on when a processor is built and torn down. Pretty much any state in the
> middle is up to the processor. you know when resources need to be stood up.
> you know when they need to be torn down.
> 
> Because threads have a localized scope, I'm not sure a global pool would be
> a help. If a processor needs higher throughput or shorter latency, now, the
> problem is generally isolated and there is a nice little cream center to
> optimize. If you're blocked on a global pool of threads because some other
> processor consumed all the threads in a pool, well, suddenly, your
> performance is no longer a localized problem.
> 
> because the common case is "don't use threads" (not everyone is going to
> build a complex service, contribute to the core framework or need threads
> in their processor) I actually think code review is a good way to shake out
> some poor decisions. because optimizing the threads in a processor for a
> use case a specialized task (the processor writer knows the critical
> sections and bottlenecks), I'm not sure whether there are massive strides
> that can be made, but I could be wrong. And we'll always have a weird edge
> case of some library that wants to do threads its own way that we're trying
> to integrate.
> 
> My guess is a lot of the behavior you mention above are because at the
> moment, performance isn't needed in that part of code and it was simpler
> for the author. Or its a bug!
> 
> 
> 
> On Mon, Nov 16, 2015 at 8:01 PM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
> 
>> Taking liberties - so let me throw few example. I am sure you’d agree that
>> Thread creation and management is an expensive and hard and error prone,
>> hence new java.util.concurrent and all the goodies in it.
>> - There is a patch currently in the queue where there is a creation of new
>> Thread() and then starting it. Is it necessary? Could we reuse the thread
>> from the common pool?
>> - We have many places where we have Thread.sleep(..) and in fact do sleep
>> considerable amount of time. That thread lays dormant where it could
>> actually be doing something. Is it necessary?
>> 
>> Cheers
>> Oleg
>> 
>> 
>>> On Nov 16, 2015, at 7:52 PM, Tony Kurc <trkurc@gmail.com> wrote:
>>> 
>>> the issue with a best practices guide on this subject is it will be
>>> dominated by edge cases. The common case should be "don't produce any
>>> threads".
>>> 
>>> That being said, I commented on a jira somewhere about
>> LinkedBlockingQueues
>>> used in so many producer/consumer style processors and possibly needing a
>>> library to have some consistency in using those queues in a consistent
>>> thread safe manner.
>>> 
>>> Also, I'm not quite sure of what you mean by taking liberties?
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Nov 16, 2015 at 7:39 PM, Oleg Zhurakousky <
>>> ozhurakousky@hortonworks.com> wrote:
>>> 
>>>> Guys
>>>> 
>>>> I am noticing many modules where we have things like "new
>>>> Thread(..).start()”, creation of new executors and schedulers,
>>>> Thread.sleep(..)  etc.,. I am sure many would agree that taking such
>>>> liberties with Threads will have consequences (not IF but WHEN)
>>>> On several threads several of us mentioned a “must read” for anyone who
>> is
>>>> getting into concurrent code -
>> http://ptgmedia.pearsoncmg.com/images/9780321349606/samplepages/9780321349606.pdf
>>>> and indeed we can/should definitely grab some best practices from this
>> book.
>>>> 
>>>> At least we can start from what’s our strategy around thread management
>>>> for NAR developers? Basically should/should not a user create Threads,
>>>> Executors, Schedulers etc.
>>>> 
>>>> Cheers
>>>> Oleg
>> 
>> 

Mime
View raw message