nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject Re: Solr Cloud support
Date Sat, 05 Sep 2015 03:51:10 GMT
Understood. Thanks!

Srikanth

On Fri, Sep 4, 2015 at 8:31 AM, Bryan Bende <bbende@gmail.com> wrote:

> Srikanth,
>
> Sure... if you have two nifi instances, or clusters, you can send data
> between them using site-to-site communication. You can do a push or pull
> model, but lets say a push model...
> The receiving nifi instance would have an Input Port on the graph, and the
> sending nifi instance would have a Remote Process Group connected to that
> Input Port. Any data routed to that Remote Process Group will get sent
> directly to the other instance/cluster.
>
> Now with in a cluster you can use the same technique, on the graph you
> would have:
> - Input Port -> Processor2
> - Processor1 (Primary Node Only scheduling) -> Remote Process Group
> (connected to its own Input Port)
>
> When data leaves Processor1 and goes into the Remote Process Group, NiFi
> will then distribute the data to all of the other nodes which each have the
> Input Port.
> Currently this is the way to re-distribute the data across the cluster
> from a processor that is only running on the primary node.
>
> -Bryan
>
> On Thu, Sep 3, 2015 at 10:25 PM, Srikanth <srikanth.ht@gmail.com> wrote:
>
>> Bryan,
>>
>> That was my first thought too but then I felt I was over complicating it
>> ;-)
>> I didn't realize such pattern was used in other processors.
>>
>> If you don't mind, can you elaborate more on this...
>>
>>> *"In a cluster you would likely set this up by having
>>> DistributeSolrCommand send to a Remote Process Group that points back to an
>>> input port of itself, and the input port feeds into ExecuteSolrCommand."*
>>
>>
>> Srikanth
>>
>> On Thu, Sep 3, 2015 at 9:32 AM, Bryan Bende <bbende@gmail.com> wrote:
>>
>>> Srikanth/Joe,
>>>
>>> I think I understand the scenario a little better now, and to Joe's
>>> points - it will probably be clearer how to do this in a more generic way
>>> as we work towards the High-Availability NCM.
>>>
>>> Thinking out loud here given the current state of things, I'm wondering
>>> if the desired functionality could be achieve by doing something similar to
>>> ListHDFS and FetchHDFS... what if there was a DistributeSolrCommand and
>>> ExecuteSolrCommand?
>>>
>>> DistributeSolrCommand would be set to run on the Primary Node and would
>>> be configured with similar properties to what GetSolr has now (zk hosts, a
>>> query, timestamp field, distrib=false, etc), it would query ZooKeeper and
>>> produce a FlowFile for each shard, and the FlowFile would use either the
>>> attributes, or payload, to capture all of the processor property values
>>> plus the shard information, basically producing a command for a downstream
>>> processor to run.
>>>
>>> ExecuteSolrCommand would be running on every node and would be
>>> responsible for interpreting the incoming FlowFile and executing whatever
>>> operation was being specified, and then passing on the results.
>>>
>>> In a cluster you would likely set this up by having
>>> DistributeSolrCommand send to a Remote Process Group that points back to an
>>> input port of itself, and the input port feeds into ExecuteSolrCommand.
>>> This would get the automatic querying of each shard, but you would still
>>> have DistributeSolrCommand running on one node and needing to be manually
>>> failed over, until we address the HA stuff.
>>>
>>> This would be a fair amount of work, but food for thought.
>>>
>>> -Bryan
>>>
>>>
>>> On Wed, Sep 2, 2015 at 11:08 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>>
>>>> <- general commentary not specific to the solr case ->
>>>>
>>>> This concept of being able to have nodes share information about
>>>> 'which partition' they should be responsible for is a generically
>>>> useful and very powerful thing.  We need to support it.  It isn't
>>>> immediately obvious to me how best to do this as a generic and useful
>>>> thing but a controller service on the NCM could potentially assign
>>>> 'partitions' to the nodes.  Zookeeper could be an important part.  I
>>>> think we need to tackle the HA NCM construct we talked about months
>>>> ago before we can do this one nicely.
>>>>
>>>> On Wed, Sep 2, 2015 at 7:47 PM, Srikanth <srikanth.ht@gmail.com> wrote:
>>>> > Bryan,
>>>> >
>>>> > <Bryan> --> "I'm still a little bit unclear about the use case
for
>>>> querying
>>>> > the shards individually... is the reason to do this because of a
>>>> > performance/failover concern?"
>>>> > <Srikanth> --> Reason to do this is to achieve better performance
>>>> with the
>>>> > convenience of automatic failover.
>>>> > In the current mode, we do get very good failover offered by Solr.
>>>> Failover
>>>> > is seamless.
>>>> > At the same time, we are not getting best performance. I guess its
>>>> clear to
>>>> > us why having each NiFi process query each shard with distrib=false
>>>> will
>>>> > give better performance.
>>>> >
>>>> > Now, question is how do we achieve this. Making user configure one
>>>> NiFi
>>>> > processor for each Solr node is one way to go.
>>>> > I'm afraid this will make failover a tricky process. May even need
>>>> human
>>>> > intervention.
>>>> >
>>>> > Another approach is to have cluster master in NiFi talk to ZK and
>>>> decide
>>>> > which shards to query. Divide these shards among slave nodes.
>>>> > My understanding is NiFi cluster master is not indented for such
>>>> purpose.
>>>> > I'm not sure if this even possible.
>>>> >
>>>> > Hope I'm a bit more clear now.
>>>> >
>>>> > Srikanth
>>>> >
>>>> > On Wed, Sep 2, 2015 at 5:58 PM, Bryan Bende <bbende@gmail.com>
wrote:
>>>> >>
>>>> >> Srikanth,
>>>> >>
>>>> >> Sorry you hadn't seen the reply, but hopefully you are subscribed
to
>>>> both
>>>> >> the dev and users list now :)
>>>> >>
>>>> >> I'm still a little bit unclear about the use case for querying the
>>>> shards
>>>> >> individually... is the reason to do this because of a
>>>> performance/failover
>>>> >> concern? or is it something specific about how the data is shared?
>>>> >>
>>>> >> Lets say you have your Solr cluster with 10 shards, each on their
>>>> own node
>>>> >> for simplicity, and then your ZooKeeper cluster.
>>>> >> Then you also have a NiFi cluster with 3 nodes each with their own
>>>> nifi
>>>> >> instance, the first node designated as the primary, and a fourth
>>>> node as the
>>>> >> cluster manager.
>>>> >>
>>>> >> Now if you want to extract data from your Solr cluster, you would
do
>>>> the
>>>> >> following...
>>>> >> - Drag GetSolr on to the graph
>>>> >> - Set type to "cloud"
>>>> >> - Set the Solr Location to the ZK hosts string
>>>> >> - Set the scheduling to "Primary Node"
>>>> >>
>>>> >> When you start the processor it is now only running on the first
NiFi
>>>> >> node, and it it is extracting data from all your shards at the same
>>>> time.
>>>> >> If a Solr shard/node fails this would be handled for us by the SolrJ
>>>> >> SolrCloudClient which is using ZooKeeper to know about the state
of
>>>> things,
>>>> >> and would choose a healthy replica of the shard if it existed.
>>>> >> If the primary NiFi node failed, you would manually elect a new
>>>> primary
>>>> >> node and the extraction would resume on that node (this will get
>>>> better in
>>>> >> the future).
>>>> >>
>>>> >> I think if we expose the distrib=false it would allow you to query
>>>> shards
>>>> >> individually, either by having a nifi instance with a GetSolr
>>>> processor per
>>>> >> shard, or several mini-NiFis each with a single GetSolr, but
>>>> >> I'm not sure if we could achieve the dynamic assignment you are
>>>> thinking
>>>> >> of.
>>>> >>
>>>> >> Let me know if I'm not making sense, happy to keep discussing and
>>>> trying
>>>> >> to figure out what else can be done.
>>>> >>
>>>> >> -Bryan
>>>> >>
>>>> >> On Wed, Sep 2, 2015 at 4:38 PM, Srikanth <srikanth.ht@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>>
>>>> >>> Bryan,
>>>> >>>
>>>> >>> That is correct, having the ability to query nodes with
>>>> "distrib=false"
>>>> >>> is what I was talking about.
>>>> >>>
>>>> >>> Instead of user having to configure each Solr node in a separate
>>>> NiFi
>>>> >>> processor, can we provide a single configuration??
>>>> >>> It would be great if we can take just Zookeeper(ZK) host as
input
>>>> from
>>>> >>> user and
>>>> >>>   i) Determine all nodes for a container from ZK
>>>> >>>   ii) Let each NiFi processor takes ownership of querying a
node
>>>> with
>>>> >>> "distrib=false"
>>>> >>>
>>>> >>> From what I understand, NiFi slaves in cluster can't talk to
each
>>>> other.
>>>> >>> Will it be possible to do the ZK query part in cluster master
and
>>>> have
>>>> >>> individual Solr nodes propagated to each slave?
>>>> >>> I don't know how we can achieve this in NiFi, if at all.
>>>> >>>
>>>> >>> This will make Solr interface to NiFi much simpler. User needs
to
>>>> provide
>>>> >>> just ZK.
>>>> >>> We'll be able to take care rest. Including failing over to an
>>>> alternate
>>>> >>> Solr node with current one fails.
>>>> >>>
>>>> >>> Let me know your thoughts.
>>>> >>>
>>>> >>> Rgds,
>>>> >>> Srikanth
>>>> >>>
>>>> >>> P.S : I had subscribed only to digest and didn't receive your
>>>> original
>>>> >>> reply. Had to pull this up from mail archive.
>>>> >>> Only Dev list is in Nabble!!
>>>> >>>
>>>> >>>
>>>> >>>
>>>> ***************************************************************************************************
>>>> >>>
>>>> >>> Hi Srikanth,
>>>> >>>
>>>> >>> You are correct that in a NiFi cluster the intent would be to
>>>> schedule
>>>> >>> GetSolr on the primary node only (on the scheduling tab) so
that
>>>> only one
>>>> >>> node in your cluster was extracting data.
>>>> >>>
>>>> >>> GetSolr determines which SolrJ client to use based on the "Solr
>>>> Type"
>>>> >>> property, so if you select "Cloud" it will use SolrCloudClient.
It
>>>> would
>>>> >>> send the query to one node based on the cluster state from
>>>> ZooKeeper, and
>>>> >>> then that Solr node performs the distributed query.
>>>> >>>
>>>> >>> Did you have a specific use case where you wanted to query each
>>>> shard
>>>> >>> individually?
>>>> >>>
>>>> >>> I think it would be straight forward to expose something on
GetSolr
>>>> that
>>>> >>> would set "distrib=false" on the query so that Solr would not
>>>> execute a
>>>> >>> distributed query. You would then most likely create separate
>>>> instances
>>>> >>> of
>>>> >>> GetSolr and configure them as Standard type pointing at the
>>>> respective
>>>> >>> shards. Let us know if that is something you are interested
in.
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>> Bryan
>>>> >>>
>>>> >>>
>>>> >>> On Sun, Aug 30, 2015 at 7:32 PM, Srikanth <srikanth.ht@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> > Hello,
>>>> >>> >
>>>> >>> > I started to explore NiFi project a few days back. I'm
still
>>>> trying it
>>>> >>> > out.
>>>> >>> >
>>>> >>> > I have a few basic question on GetSolr.
>>>> >>> >
>>>> >>> > Should GetSolr be run as an Isolated Processor?
>>>> >>> >
>>>> >>> > If I have SolrCloud with 4 shards/nodes and NiFi cluster
with 4
>>>> nodes,
>>>> >>> > will GetSolr be able to query each shard from one specific
NiFi
>>>> node?
>>>> >>> > I'm
>>>> >>> > guessing it doesn't work that way.
>>>> >>> >
>>>> >>> >
>>>> >>> > Thanks,
>>>> >>> > Srikanth
>>>> >>> >
>>>> >>> >
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>

Mime
View raw message