nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Solr Cloud support
Date Fri, 04 Sep 2015 02:19:41 GMT
Here is a JIRA for it: https://issues.apache.org/jira/browse/NIFI-338
And a related one: https://issues.apache.org/jira/browse/NIFI-540

There may also be some discussion in the mailing list about it but I
think the main points are captured in those JIRA tickets.

Thanks
joe

On Thu, Sep 3, 2015 at 7:15 PM, Srikanth <srikanth.ht@gmail.com> wrote:
> Joe,
>
> Yes, that will be a really nice feature to have.
> Is there a jira or wiki on the discussion you guys had on HA NCM?
>
> Srikanth
>
> On Wed, Sep 2, 2015 at 11:08 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>> <- general commentary not specific to the solr case ->
>>
>> This concept of being able to have nodes share information about
>> 'which partition' they should be responsible for is a generically
>> useful and very powerful thing.  We need to support it.  It isn't
>> immediately obvious to me how best to do this as a generic and useful
>> thing but a controller service on the NCM could potentially assign
>> 'partitions' to the nodes.  Zookeeper could be an important part.  I
>> think we need to tackle the HA NCM construct we talked about months
>> ago before we can do this one nicely.
>>
>> On Wed, Sep 2, 2015 at 7:47 PM, Srikanth <srikanth.ht@gmail.com> wrote:
>> > Bryan,
>> >
>> > <Bryan> --> "I'm still a little bit unclear about the use case for
>> > querying
>> > the shards individually... is the reason to do this because of a
>> > performance/failover concern?"
>> > <Srikanth> --> Reason to do this is to achieve better performance with
>> > the
>> > convenience of automatic failover.
>> > In the current mode, we do get very good failover offered by Solr.
>> > Failover
>> > is seamless.
>> > At the same time, we are not getting best performance. I guess its clear
>> > to
>> > us why having each NiFi process query each shard with distrib=false will
>> > give better performance.
>> >
>> > Now, question is how do we achieve this. Making user configure one NiFi
>> > processor for each Solr node is one way to go.
>> > I'm afraid this will make failover a tricky process. May even need human
>> > intervention.
>> >
>> > Another approach is to have cluster master in NiFi talk to ZK and decide
>> > which shards to query. Divide these shards among slave nodes.
>> > My understanding is NiFi cluster master is not indented for such
>> > purpose.
>> > I'm not sure if this even possible.
>> >
>> > Hope I'm a bit more clear now.
>> >
>> > Srikanth
>> >
>> > On Wed, Sep 2, 2015 at 5:58 PM, Bryan Bende <bbende@gmail.com> wrote:
>> >>
>> >> Srikanth,
>> >>
>> >> Sorry you hadn't seen the reply, but hopefully you are subscribed to
>> >> both
>> >> the dev and users list now :)
>> >>
>> >> I'm still a little bit unclear about the use case for querying the
>> >> shards
>> >> individually... is the reason to do this because of a
>> >> performance/failover
>> >> concern? or is it something specific about how the data is shared?
>> >>
>> >> Lets say you have your Solr cluster with 10 shards, each on their own
>> >> node
>> >> for simplicity, and then your ZooKeeper cluster.
>> >> Then you also have a NiFi cluster with 3 nodes each with their own nifi
>> >> instance, the first node designated as the primary, and a fourth node
>> >> as the
>> >> cluster manager.
>> >>
>> >> Now if you want to extract data from your Solr cluster, you would do
>> >> the
>> >> following...
>> >> - Drag GetSolr on to the graph
>> >> - Set type to "cloud"
>> >> - Set the Solr Location to the ZK hosts string
>> >> - Set the scheduling to "Primary Node"
>> >>
>> >> When you start the processor it is now only running on the first NiFi
>> >> node, and it it is extracting data from all your shards at the same
>> >> time.
>> >> If a Solr shard/node fails this would be handled for us by the SolrJ
>> >> SolrCloudClient which is using ZooKeeper to know about the state of
>> >> things,
>> >> and would choose a healthy replica of the shard if it existed.
>> >> If the primary NiFi node failed, you would manually elect a new primary
>> >> node and the extraction would resume on that node (this will get better
>> >> in
>> >> the future).
>> >>
>> >> I think if we expose the distrib=false it would allow you to query
>> >> shards
>> >> individually, either by having a nifi instance with a GetSolr processor
>> >> per
>> >> shard, or several mini-NiFis each with a single GetSolr, but
>> >> I'm not sure if we could achieve the dynamic assignment you are
>> >> thinking
>> >> of.
>> >>
>> >> Let me know if I'm not making sense, happy to keep discussing and
>> >> trying
>> >> to figure out what else can be done.
>> >>
>> >> -Bryan
>> >>
>> >> On Wed, Sep 2, 2015 at 4:38 PM, Srikanth <srikanth.ht@gmail.com> wrote:
>> >>>
>> >>>
>> >>> Bryan,
>> >>>
>> >>> That is correct, having the ability to query nodes with
>> >>> "distrib=false"
>> >>> is what I was talking about.
>> >>>
>> >>> Instead of user having to configure each Solr node in a separate NiFi
>> >>> processor, can we provide a single configuration??
>> >>> It would be great if we can take just Zookeeper(ZK) host as input from
>> >>> user and
>> >>>   i) Determine all nodes for a container from ZK
>> >>>   ii) Let each NiFi processor takes ownership of querying a node with
>> >>> "distrib=false"
>> >>>
>> >>> From what I understand, NiFi slaves in cluster can't talk to each
>> >>> other.
>> >>> Will it be possible to do the ZK query part in cluster master and have
>> >>> individual Solr nodes propagated to each slave?
>> >>> I don't know how we can achieve this in NiFi, if at all.
>> >>>
>> >>> This will make Solr interface to NiFi much simpler. User needs to
>> >>> provide
>> >>> just ZK.
>> >>> We'll be able to take care rest. Including failing over to an
>> >>> alternate
>> >>> Solr node with current one fails.
>> >>>
>> >>> Let me know your thoughts.
>> >>>
>> >>> Rgds,
>> >>> Srikanth
>> >>>
>> >>> P.S : I had subscribed only to digest and didn't receive your original
>> >>> reply. Had to pull this up from mail archive.
>> >>> Only Dev list is in Nabble!!
>> >>>
>> >>>
>> >>>
>> >>> ***************************************************************************************************
>> >>>
>> >>> Hi Srikanth,
>> >>>
>> >>> You are correct that in a NiFi cluster the intent would be to schedule
>> >>> GetSolr on the primary node only (on the scheduling tab) so that only
>> >>> one
>> >>> node in your cluster was extracting data.
>> >>>
>> >>> GetSolr determines which SolrJ client to use based on the "Solr Type"
>> >>> property, so if you select "Cloud" it will use SolrCloudClient. It
>> >>> would
>> >>> send the query to one node based on the cluster state from ZooKeeper,
>> >>> and
>> >>> then that Solr node performs the distributed query.
>> >>>
>> >>> Did you have a specific use case where you wanted to query each shard
>> >>> individually?
>> >>>
>> >>> I think it would be straight forward to expose something on GetSolr
>> >>> that
>> >>> would set "distrib=false" on the query so that Solr would not execute
>> >>> a
>> >>> distributed query. You would then most likely create separate
>> >>> instances
>> >>> of
>> >>> GetSolr and configure them as Standard type pointing at the respective
>> >>> shards. Let us know if that is something you are interested in.
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Bryan
>> >>>
>> >>>
>> >>> On Sun, Aug 30, 2015 at 7:32 PM, Srikanth <srikanth.ht@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Hello,
>> >>> >
>> >>> > I started to explore NiFi project a few days back. I'm still trying
>> >>> > it
>> >>> > out.
>> >>> >
>> >>> > I have a few basic question on GetSolr.
>> >>> >
>> >>> > Should GetSolr be run as an Isolated Processor?
>> >>> >
>> >>> > If I have SolrCloud with 4 shards/nodes and NiFi cluster with 4
>> >>> > nodes,
>> >>> > will GetSolr be able to query each shard from one specific NiFi
>> >>> > node?
>> >>> > I'm
>> >>> > guessing it doesn't work that way.
>> >>> >
>> >>> >
>> >>> > Thanks,
>> >>> > Srikanth
>> >>> >
>> >>> >
>> >>
>> >>
>> >
>
>

Mime
View raw message