nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: Common scheduler and add-hock thread creation
Date Tue, 17 Nov 2015 02:37:37 GMT
The only one I've seen is the catch 22 where you need to shut down executor from the task executed
by this executor. So in this case you have no choice but to create a Thread and start it

Sent from my iPhone

> On Nov 16, 2015, at 20:33, Matt Burgess <mattyb149@gmail.com> wrote:
> 
> Is there a common use case for a single extra thread? I'm asking in ignorance but if
the answer is yes then I'd think there'd be best practices around that?
> 
> Regards,
> Matt
> 
> Sent from my iPhone
> 
>> On Nov 16, 2015, at 8:22 PM, Tony Kurc <trkurc@gmail.com> wrote:
>> 
>> so, I believe threads in a processor in nifi are much, much easier than
>> general threading in many other applications. There are defined boundaries
>> on when a processor is built and torn down. Pretty much any state in the
>> middle is up to the processor. you know when resources need to be stood up.
>> you know when they need to be torn down.
>> 
>> Because threads have a localized scope, I'm not sure a global pool would be
>> a help. If a processor needs higher throughput or shorter latency, now, the
>> problem is generally isolated and there is a nice little cream center to
>> optimize. If you're blocked on a global pool of threads because some other
>> processor consumed all the threads in a pool, well, suddenly, your
>> performance is no longer a localized problem.
>> 
>> because the common case is "don't use threads" (not everyone is going to
>> build a complex service, contribute to the core framework or need threads
>> in their processor) I actually think code review is a good way to shake out
>> some poor decisions. because optimizing the threads in a processor for a
>> use case a specialized task (the processor writer knows the critical
>> sections and bottlenecks), I'm not sure whether there are massive strides
>> that can be made, but I could be wrong. And we'll always have a weird edge
>> case of some library that wants to do threads its own way that we're trying
>> to integrate.
>> 
>> My guess is a lot of the behavior you mention above are because at the
>> moment, performance isn't needed in that part of code and it was simpler
>> for the author. Or its a bug!
>> 
>> 
>> 
>> On Mon, Nov 16, 2015 at 8:01 PM, Oleg Zhurakousky <
>> ozhurakousky@hortonworks.com> wrote:
>> 
>>> Taking liberties - so let me throw few example. I am sure you’d agree that
>>> Thread creation and management is an expensive and hard and error prone,
>>> hence new java.util.concurrent and all the goodies in it.
>>> - There is a patch currently in the queue where there is a creation of new
>>> Thread() and then starting it. Is it necessary? Could we reuse the thread
>>> from the common pool?
>>> - We have many places where we have Thread.sleep(..) and in fact do sleep
>>> considerable amount of time. That thread lays dormant where it could
>>> actually be doing something. Is it necessary?
>>> 
>>> Cheers
>>> Oleg
>>> 
>>> 
>>>> On Nov 16, 2015, at 7:52 PM, Tony Kurc <trkurc@gmail.com> wrote:
>>>> 
>>>> the issue with a best practices guide on this subject is it will be
>>>> dominated by edge cases. The common case should be "don't produce any
>>>> threads".
>>>> 
>>>> That being said, I commented on a jira somewhere about
>>> LinkedBlockingQueues
>>>> used in so many producer/consumer style processors and possibly needing a
>>>> library to have some consistency in using those queues in a consistent
>>>> thread safe manner.
>>>> 
>>>> Also, I'm not quite sure of what you mean by taking liberties?
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Mon, Nov 16, 2015 at 7:39 PM, Oleg Zhurakousky <
>>>> ozhurakousky@hortonworks.com> wrote:
>>>> 
>>>>> Guys
>>>>> 
>>>>> I am noticing many modules where we have things like "new
>>>>> Thread(..).start()”, creation of new executors and schedulers,
>>>>> Thread.sleep(..)  etc.,. I am sure many would agree that taking such
>>>>> liberties with Threads will have consequences (not IF but WHEN)
>>>>> On several threads several of us mentioned a “must read” for anyone
who
>>> is
>>>>> getting into concurrent code -
>>> http://ptgmedia.pearsoncmg.com/images/9780321349606/samplepages/9780321349606.pdf
>>>>> and indeed we can/should definitely grab some best practices from this
>>> book.
>>>>> 
>>>>> At least we can start from what’s our strategy around thread management
>>>>> for NAR developers? Basically should/should not a user create Threads,
>>>>> Executors, Schedulers etc.
>>>>> 
>>>>> Cheers
>>>>> Oleg
> 

Mime
View raw message