lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: SolrCloud and exernal file fields
Date Wed, 28 Nov 2012 13:56:44 GMT
Keep in mind that the distrib update proc will be auto inserted into chains! You have to include a proc that disables it - see the FAQ: http://wiki.apache.org/solr/SolrCloud#FAQ

- Mark

On Nov 28, 2012, at 7:25 AM, Mikhail Khludnev <mkhludnev@griddynamics.com> wrote:

> Martin,
> Right as far node in Zookeeper DistributedUpdateProcessor will broadcast
> commits to all peers. To hack this you can introduce dedicated
> UpdateProcessorChain without DistributedUpdateProcessor and send commit to
> that chain.
> 28.11.2012 13:16 пользователь "Martin Koch" <mak@issuu.com> написал:
> 
>> Mikhail
>> 
>> I haven't experimented further yet. I think that the previous experiment
>> of issuing a commit to a specific core proved that all cores get the
>> commit, so I don't think that this approach will work.
>> 
>> Thanks,
>> /Martin
>> 
>> 
>> On Tue, Nov 27, 2012 at 6:24 PM, Mikhail Khludnev <
>> mkhludnev@griddynamics.com> wrote:
>> 
>>> Martin,
>>> 
>>> It's still not clear to me whether you solve the problem completely or
>>> partially:
>>> Does reducing number of cores free some resources for searching during
>>> commit?
>>> Does the commiting one-by-one core prevents the "freeze"?
>>> 
>>> Thanks
>>> 
>>> 
>>> On Thu, Nov 22, 2012 at 4:31 PM, Martin Koch <mak@issuu.com> wrote:
>>> 
>>>> Mikhail
>>>> 
>>>> To avoid freezes we deployed the patches that are now on the 4.1 trunk
>>>> (bug
>>>> 3985). But this wasn't good enough, because SOLR would still take very
>>>> long
>>>> to restart when that was necessary.
>>>> 
>>>> I don't see how we could throw more hardware at the problem without
>>>> making
>>>> it worse, really - the only solution here would be *fewer* shards, not
>>>> 
>>>> more.
>>>> 
>>>> IMO it would be ideal if the lucene/solr community could come up with a
>>>> good way of updating fields in a document without reindexing. This could
>>>> be
>>>> by linking to some external data store, or in the lucene/solr internals.
>>>> If
>>>> it would make things easier, a good first step would be to have
>>>> dynamically
>>>> updateable numerical fields only.
>>>> 
>>>> /Martin
>>>> 
>>>> On Wed, Nov 21, 2012 at 8:51 PM, Mikhail Khludnev <
>>>> mkhludnev@griddynamics.com> wrote:
>>>> 
>>>>> Martin,
>>>>> 
>>>>> I don't think solrconfig.xml shed any light on. I've just found what I
>>>>> didn't get in your setup - the way of how to explicitly assigning core
>>>> to
>>>>> collection. Now, I realized most of details after all!
>>>>> Ball is on your side, let us know whether you have managed your cores
>>>> to
>>>>> commit one by one to avoid freeze, or could you eliminate pauses by
>>>>> allocating more hardware?
>>>>> Thanks in advance!
>>>>> 
>>>>> 
>>>>> On Wed, Nov 21, 2012 at 3:56 PM, Martin Koch <mak@issuu.com> wrote:
>>>>> 
>>>>>> Mikhail,
>>>>>> 
>>>>>> PSB
>>>>>> 
>>>>>> On Wed, Nov 21, 2012 at 10:08 AM, Mikhail Khludnev <
>>>>>> mkhludnev@griddynamics.com> wrote:
>>>>>> 
>>>>>>> On Wed, Nov 21, 2012 at 11:53 AM, Martin Koch <mak@issuu.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> I wasn't aware until now that it is possible to send a commit to
>>>> one
>>>>>> core
>>>>>>>> only. What we observed was the effect of curl
>>>>>>>> localhost:8080/solr/update?commit=true but perhaps we should
>>>>> experiment
>>>>>>>> with solr/coreN/update?commit=true. A quick trial run seems to
>>>>> indicate
>>>>>>>> that a commit to a single core causes commits on all cores.
>>>>>>>> 
>>>>>>> You should see something like this in the log:
>>>>>>> ... SolrCmdDistributor .... Distrib commit to: ...
>>>>>>> 
>>>>>>> Yup, a commit towards a single core results in a commit on all
>>>> cores.
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Perhaps I should clarify that we are using SOLR as a black box;
>>>> we do
>>>>>> not
>>>>>>>> touch the code at all - we only install the distribution WAR
>>>> file and
>>>>>>>> proceed from there.
>>>>>>>> 
>>>>>>> I still don't understand how you deploy/launch Solr. How many
>>>> jettys
>>>>> you
>>>>>>> start whether you have -DzkRun -DzkHost -DnumShards=2  or you
>>>> specifies
>>>>>>> shards= param for every request and distributes updates yourself?
>>>> What
>>>>>>> collections do you create and with which settings?
>>>>>>> 
>>>>>>> We let SOLR do the sharding using one collection with 16 SOLR cores
>>>>>> holding one shard each. We launch only one instance of jetty with the
>>>>>> folllowing arguments:
>>>>>> 
>>>>>> -DnumShards=16
>>>>>> -DzkHost=<zookeeperhost:port>
>>>>>> -Xmx10G
>>>>>> -Xms10G
>>>>>> -Xmn2G
>>>>>> -server
>>>>>> 
>>>>>> Would you like to see the solrconfig.xml?
>>>>>> 
>>>>>> /Martin
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Also from my POV such deployments should start at least from
>>>> *16*
>>>>>> 4-way
>>>>>>>>> vboxes, it's more expensive, but should be much better
>>>> available
>>>>>> during
>>>>>>>>> cpu-consuming operations.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Do you mean that you recommend 16 hosts with 4 cores each? Or 4
>>>> hosts
>>>>>>> with
>>>>>>>> 16 cores? Or am I misunderstanding something :) ?
>>>>>>>> 
>>>>>>> I prefer to start from 16 hosts with 4 cores each.
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Other details, if you use single jetty for all of them, are you
>>>>> sure
>>>>>>> that
>>>>>>>>> jetty's threadpool doesn't limit requests? is it large enough?
>>>>>>>>> You have 60G and set -Xmx=10G. are you sure that total size of
>>>>> cores
>>>>>>>> index
>>>>>>>>> directories is less than 45G?
>>>>>>>>> 
>>>>>>>>> The total index size is 230 GB, so it won't fit in ram, but
>>>> we're
>>>>>> using
>>>>>>>> an
>>>>>>>> SSD disk to minimize disk access time. We have tried putting the
>>>> EFF
>>>>>>> onto a
>>>>>>>> ram disk, but this didn't have a measurable effect.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> /Martin
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, Nov 21, 2012 at 2:07 AM, Martin Koch <mak@issuu.com>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Mikhail
>>>>>>>>>> 
>>>>>>>>>> PSB
>>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 20, 2012 at 7:22 PM, Mikhail Khludnev <
>>>>>>>>>> mkhludnev@griddynamics.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Martin,
>>>>>>>>>>> 
>>>>>>>>>>> Please find additional question from me below.
>>>>>>>>>>> 
>>>>>>>>>>> Simone,
>>>>>>>>>>> 
>>>>>>>>>>> I'm sorry for hijacking your thread. The only what I've
>>>> heard
>>>>>> about
>>>>>>>> it
>>>>>>>>> at
>>>>>>>>>>> recent ApacheCon sessions is that Zookeeper is supposed to
>>>>>>> replicate
>>>>>>>>>> those
>>>>>>>>>>> files as configs under solr home. And I'm really looking
>>>>> forward
>>>>>> to
>>>>>>>>> know
>>>>>>>>>>> how it works with huge files in production.
>>>>>>>>>>> 
>>>>>>>>>>> Thank You, Guys!
>>>>>>>>>>> 
>>>>>>>>>>> 20.11.2012 18:06 пользователь "Martin Koch" <mak@issuu.com
>>>>> 
>>>>>>> написал:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Mikhail
>>>>>>>>>>>> 
>>>>>>>>>>>> Please see answers below.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev <
>>>>>>>>>>>> mkhludnev@griddynamics.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Martin,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you for telling your own "war-story". It's really
>>>>>> useful
>>>>>>>> for
>>>>>>>>>>>>> community.
>>>>>>>>>>>>> The first question might seems not really conscious,
>>>> but
>>>>>> would
>>>>>>>> you
>>>>>>>>>> tell
>>>>>>>>>>> me
>>>>>>>>>>>>> what blocks searching during EFF reload, when it's
>>>>> triggered
>>>>>> by
>>>>>>>>>> handler
>>>>>>>>>>> or
>>>>>>>>>>>>> by listener?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> We continuously index new documents using CommitWithin
>>>> to get
>>>>>>>> regular
>>>>>>>>>>>> commits. However, we observed that the EFFs were not
>>>> re-read,
>>>>>> so
>>>>>>> we
>>>>>>>>> had
>>>>>>>>>>> to
>>>>>>>>>>>> do external commits (curl '.../solr/update?commit=true')
>>>> to
>>>>>> force
>>>>>>>>>> reload.
>>>>>>>>>>>> When this is done, solr blocks. I can't tell you exactly
>>>> why
>>>>>> it's
>>>>>>>>> doing
>>>>>>>>>>>> that (it was related to SOLR-3985).
>>>>>>>>>>> 
>>>>>>>>>>> Is there a chance to get a thread dump when they are
>>>> blocked?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Well I could try to recreate the situation. But the setup is
>>>>> fairly
>>>>>>>>> simple:
>>>>>>>>>> Create a large EFF in a largeish index with many shards.
>>>> Issue a
>>>>>>>> commit,
>>>>>>>>>> and then try to do a search. Solr will not respond to the
>>>> search
>>>>>>> before
>>>>>>>>> the
>>>>>>>>>> commit has completed, and this will take a long time.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> I don't really get the sentence about sequential
>>>> commits
>>>>> and
>>>>>>>> number
>>>>>>>>>> of
>>>>>>>>>>>>> cores. Do I get right that file is replicated via
>>>>> Zookeeper?
>>>>>>>>> Doesn't
>>>>>>>>>> it
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Again, this is observed behavior. When we issue a commit
>>>> on a
>>>>>>>> system
>>>>>>>>>> with
>>>>>>>>>>> a
>>>>>>>>>>>> system with many solr cores using EFFs, the system blocks
>>>>> for a
>>>>>>>> long
>>>>>>>>>> time
>>>>>>>>>>>> (15 minutes).  We do NOT use zookeeper for anything. The
>>>> EFF
>>>>>> is a
>>>>>>>>>> symlink
>>>>>>>>>>>> from each cores index dir to the actual file, which is
>>>>> updated
>>>>>> by
>>>>>>>> an
>>>>>>>>>>>> external process.
>>>>>>>>>>> 
>>>>>>>>>>> Hold on, I asked about Zookeeper because the subj mentions
>>>>>>> SolrCloud.
>>>>>>>>>>> 
>>>>>>>>>>> Do you use SolrCloud, SolrShards, or these cores are just
>>>>>> replicas
>>>>>>> of
>>>>>>>>> the
>>>>>>>>>>> same index?
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Ah - we use solr 4 out of the box, so I guess this is
>>>> SolrCloud.
>>>>>> I'm
>>>>>>> a
>>>>>>>>> bit
>>>>>>>>>> unsure about the terminology here, but we've got a single
>>>> index
>>>>>>> divided
>>>>>>>>>> into 16 shard. Each shard is hosted in a solr core.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Also, about simlink - Don't you share that file via some
>>>> NFS?
>>>>>>>>>>> 
>>>>>>>>>>> No, we generate the EFF on the local solr host (there is
>>>> only
>>>>> one
>>>>>>>>>> physical
>>>>>>>>>> host that holds all shards), so there is no need for NFS or
>>>>> copying
>>>>>>>> files
>>>>>>>>>> around. No need for Zookeeper either.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> how many cores you run per box?
>>>>>>>>>>> 
>>>>>>>>>> This box is a 16-virtual core (8 hyperthreaded cores)  with
>>>> 60GB
>>>>> of
>>>>>>>> RAM.
>>>>>>>>> We
>>>>>>>>>> run 16 solr cores on this box in Jetty.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Do boxes has plenty of ram to cache filesystem beside of
>>>> jvm
>>>>>> heaps?
>>>>>>>>>>> 
>>>>>>>>>>> Yes. We've allocated 10GB for jetty, and left the rest for
>>>> the
>>>>>> OS.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> I assume you use 64 bit linux and mmap directory. Please
>>>>> confirm
>>>>>>>> that.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> We use 64-bit linux. I'm not sure about the mmap directory or
>>>>> where
>>>>>>>> that
>>>>>>>>>> would be configured in solr - can you explain that?
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> causes scalability problem or long time to reload?
>>>> Will it
>>>>>> help
>>>>>>>> if
>>>>>>>>>>> we'll
>>>>>>>>>>>>> have, let's say ExternalDatabaseField which will pull
>>>>> values
>>>>>>> from
>>>>>>>>>> jdbc.
>>>>>>>>>>> ie.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I think the possibility of having some fields being
>>>> retrieved
>>>>>>> from
>>>>>>>> an
>>>>>>>>>>>> external, dynamically updatable store would be really
>>>>>>> interesting.
>>>>>>>>> This
>>>>>>>>>>>> could be JDBC, something in-memory like redis, or a NoSql
>>>>>> product
>>>>>>>>> (e.g.
>>>>>>>>>>>> Cassandra).
>>>>>>>>>>> 
>>>>>>>>>>> Ok. Let's have it in mind as a possible direction.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Alternatively, an API that would allow updating a single
>>>> field
>>>>> for
>>>>>> a
>>>>>>>>>> document might be an option.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> why all cores can't read these values simultaneously?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Again, this is a solr implementation detail that I can't
>>>>> answer
>>>>>>> :)
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Can you confirm that IDs in the file is ordered by the
>>>>> index
>>>>>>> term
>>>>>>>>>>> order?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, we sorted the files (standard UNIX sort).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> AFAIK it can impact load time.
>>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, it does
>>>>>>>>>>> 
>>>>>>>>>>> Ok, I've got that you aware of it, and your IDs are just
>>>>> strings,
>>>>>>> not
>>>>>>>>>>> integers.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Yes, ids are strings.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Regarding your post-query solution can you tell me if
>>>> query
>>>>>>> found
>>>>>>>>>> 10000
>>>>>>>>>>>>> docs, but I need to display only first page with 100
>>>> rows,
>>>>>>>> whether
>>>>>>>>> I
>>>>>>>>>>> need
>>>>>>>>>>>>> to pull all 10K results to frontend to order them by
>>>> the
>>>>>> rank?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> In our architecture, the clients query an API that
>>>> generates
>>>>>> the
>>>>>>>> SOLR
>>>>>>>>>>>> query, retrieves the relevant additional fields that we
>>>>> needs,
>>>>>>> and
>>>>>>>>>>> returns
>>>>>>>>>>>> the relevant JSON to the front-end.
>>>>>>>>>>>> 
>>>>>>>>>>>> In our use case, results are returned from SOLR by the
>>>> 10's,
>>>>>> not
>>>>>>> by
>>>>>>>>> the
>>>>>>>>>>>> 1000's, so it is a manageable job. Even so, if solr
>>>> returned
>>>>>>>>> thousands
>>>>>>>>>> of
>>>>>>>>>>>> results, it would be up to the implementation of the api
>>>> to
>>>>>>> augment
>>>>>>>>>> only
>>>>>>>>>>>> the results that needed to be returned to the front-end.
>>>>>>>>>>>> 
>>>>>>>>>>>> Even so, patching up a JSON structure with 10000 results
>>>>> should
>>>>>>> be
>>>>>>>>>>>> possible.
>>>>>>>>>>> 
>>>>>>>>>>> You are right. I'm concerned anyway because retrieving
>>>> whole
>>>>>> result
>>>>>>>> is
>>>>>>>>>>> expensive, and not always possible.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> In our case, getting the whole result is almost impossible,
>>>>> because
>>>>>>>> that
>>>>>>>>>> would be millions of documents, and returning the Nth result
>>>>> seems
>>>>>> to
>>>>>>>> be
>>>>>>>>> a
>>>>>>>>>> quadratic (or worse) operation in SOLR.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm really appreciate if you comment on the questions
>>>>> above.
>>>>>>>>>>>>> PS: It's time to pitch, how much
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/SOLR-4085
>>>> "Commit-free
>>>>>>>>>>>>> ExternalFileField" can help you?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It looks very interesting :) Does it make it possible
>>>> to
>>>>>> avoid
>>>>>>>>>>> re-reading
>>>>>>>>>>>> the EFF on every commit, and only re-read the values that
>>>>> have
>>>>>>>>> actually
>>>>>>>>>>>> changed?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> You don't need commit (in SOLR-4085) to reload file
>>>> content,
>>>>> but
>>>>>>>> after
>>>>>>>>>>> commit you need to read whole file and scan all key terms
>>>> and
>>>>>>>> postings.
>>>>>>>>>>> That's because EFF sits on top of top level searcher. it's
>>>> a
>>>>>>>> Solr-like
>>>>>>>>>> way.
>>>>>>>>>>> In some future we might have per-segment EFF, in this case
>>>>>> adding a
>>>>>>>>>> segment
>>>>>>>>>>> will trigger full file scan, but in the index only that new
>>>>>> segment
>>>>>>>>> will
>>>>>>>>>> be
>>>>>>>>>>> scanned. It should be faster. You know, straightforward
>>>> sharing
>>>>>>>>> internal
>>>>>>>>>>> data structures between different index views/generations
>>>> is
>>>>> not
>>>>>>>>>> possible.
>>>>>>>>>>> If you are asking about applying delta changes on external
>>>> file
>>>>>>>> that's
>>>>>>>>>>> something what we did ourselves http://goo.gl/P8GFq . This
>>>>>> feature
>>>>>>>> is
>>>>>>>>>> much
>>>>>>>>>>> more doubtful and vague, although it might be the next
>>>>>> contribution
>>>>>>>>> after
>>>>>>>>>>> SOLR-4085.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> /Martin
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch <
>>>>> mak@issuu.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Solr 4.0 does support using EFFs, but it might not
>>>> give
>>>>> you
>>>>>>>> what
>>>>>>>>>>> you're
>>>>>>>>>>>>>> hoping fore.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We tried using Solr Cloud, and have given up again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The EFF is placed in the parent of the index
>>>> directory in
>>>>>>> each
>>>>>>>>>> core;
>>>>>>>>>>> each
>>>>>>>>>>>>>> core reads the entire EFF and picks out the IDs that
>>>> it
>>>>> is
>>>>>>>>>>> responsible
>>>>>>>>>>>>> for.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In the current 4.0.0 release of solr, solr blocks
>>>>> (doesn't
>>>>>>>> answer
>>>>>>>>>>>>> queries)
>>>>>>>>>>>>>> while re-reading the EFF. Even worse, it seems that
>>>> the
>>>>>> time
>>>>>>> to
>>>>>>>>>>> re-read
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> EFF is multiplied by the number of cores in use
>>>> (i.e. the
>>>>>> EFF
>>>>>>>> is
>>>>>>>>>>> re-read
>>>>>>>>>>>>> by
>>>>>>>>>>>>>> each core sequentially). The contents of the EFF
>>>> become
>>>>>>> active
>>>>>>>>>> after
>>>>>>>>>>> the
>>>>>>>>>>>>>> first EXTERNAL commit (commitWithin does NOT work
>>>> here)
>>>>>> after
>>>>>>>> the
>>>>>>>>>>> file
>>>>>>>>>>>>> has
>>>>>>>>>>>>>> been updated.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In our case, the EFF was quite large - around 450MB
>>>> - and
>>>>>> we
>>>>>>>> use
>>>>>>>>> 16
>>>>>>>>>>>>> shards,
>>>>>>>>>>>>>> so when we triggered an external commit to force
>>>>>> re-reading,
>>>>>>>> the
>>>>>>>>>>> whole
>>>>>>>>>>>>>> system would block for several (10-15) minutes. This
>>>>> won't
>>>>>>> work
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>>>> production environment. The reason for the size of
>>>> the
>>>>> EFF
>>>>>> is
>>>>>>>>> that
>>>>>>>>>> we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> around 7M documents in the index; each document has
>>>> a 45
>>>>>>>>> character
>>>>>>>>>>> ID.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We got some help to try to fix the problem so that
>>>> the
>>>>>>> re-read
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>> EFF
>>>>>>>>>>>>>> proceeds in the background (see
>>>>>>>>>>>>>> here<https://issues.apache.org/jira/browse/SOLR-3985
>>>>> 
>>>>> for
>>>>>>>>>>>>>> a fix on the 4.1 branch). However, even though the
>>>>> re-read
>>>>>>>>> proceeds
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> background, the time required to launch solr now
>>>> takes at
>>>>>>> least
>>>>>>>>> as
>>>>>>>>>>> long
>>>>>>>>>>>>> as
>>>>>>>>>>>>>> re-reading the EFFs. Again, this is not good enough
>>>> for
>>>>> our
>>>>>>>>> needs.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The next issue is that you cannot sort on EFF fields
>>>>>> (though
>>>>>>>> you
>>>>>>>>>> can
>>>>>>>>>>>>> return
>>>>>>>>>>>>>> them as values using &fl=field(my_eff_field). This is
>>>>> also
>>>>>>>> fixed
>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>> 4.1
>>>>>>>>>>>>>> branch here <
>>>>>> https://issues.apache.org/jira/browse/SOLR-4022
>>>>>>>> .
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So: Even after these fixes, EFF performance is not
>>>> that
>>>>>>> great.
>>>>>>>>> Our
>>>>>>>>>>>>> solution
>>>>>>>>>>>>>> is as follows: The actual value of the popularity
>>>> measure
>>>>>>> (say,
>>>>>>>>>>> reads)
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> we want to report to the user is inserted into the
>>>> search
>>>>>>>>> response
>>>>>>>>>>>>>> post-query by our query front-end. This value will
>>>> then
>>>>> be
>>>>>>> the
>>>>>>>>>>>>>> authoritative value at the time of the query. The
>>>> value
>>>>> of
>>>>>>> the
>>>>>>>>>>> popularity
>>>>>>>>>>>>>> measure that we use for boosting in the ranking of
>>>> the
>>>>>> search
>>>>>>>>>> results
>>>>>>>>>>> is
>>>>>>>>>>>>>> only updated when the value has changed enough so
>>>> that
>>>>> the
>>>>>>>> impact
>>>>>>>>>> on
>>>>>>>>>>> the
>>>>>>>>>>>>>> boost will be significant (say, more than 2%). This
>>>> does
>>>>>>>> require
>>>>>>>>>>> frequent
>>>>>>>>>>>>>> re-indexing of the documents that have significant
>>>>> changes
>>>>>> in
>>>>>>>> the
>>>>>>>>>>> number
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> reads, but at least we won't have to update a
>>>> document if
>>>>>> it
>>>>>>>>> moves
>>>>>>>>>>> from,
>>>>>>>>>>>>>> say, 1000000 to 1000001 reads.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> /Martin Koch - ISSUU - senior systems architect.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni <
>>>>>>>>> simoneg@apache.org
>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> I'm planning to move a quite big Solr index to
>>>>> SolrCloud.
>>>>>>>>>> However,
>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> index, an external file field is used for
>>>> popularity
>>>>>>> ranking.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Does SolrCloud supports external file fields? How
>>>> does
>>>>> it
>>>>>>>> cope
>>>>>>>>>> with
>>>>>>>>>>>>>>> sharding and replication? Where should the external
>>>>> file
>>>>>> be
>>>>>>>>>> placed
>>>>>>>>>>> now
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> the index folder is not local but in the cloud?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Are there otherwise other best practices to deal
>>>> with
>>>>> the
>>>>>>> use
>>>>>>>>>> cases
>>>>>>>>>>>>>>> external file fields were used for, like
>>>>>>> popularity/ranking,
>>>>>>>> in
>>>>>>>>>>>>>> SolrCloud?
>>>>>>>>>>>>>>> Custom ValueSources going to something external?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>> Simone
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Sincerely yours
>>>>>>>>>>>>> Mikhail Khludnev
>>>>>>>>>>>>> Principal Engineer,
>>>>>>>>>>>>> Grid Dynamics
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <http://www.griddynamics.com>
>>>>>>>>>>>>> <mkhludnev@griddynamics.com>
>>>>>>>>>>>>> 
>>>>>>>>>>> 20.11.2012 18:06 пользователь "Martin Koch" <
>>>> mak@issuu.com>
>>>>>>>> написал:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Mikhail
>>>>>>>>>>>> 
>>>>>>>>>>>> Please see answers below.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev <
>>>>>>>>>>>> mkhludnev@griddynamics.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Martin,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you for telling your own "war-story". It's really
>>>>>> useful
>>>>>>>> for
>>>>>>>>>>>>> community.
>>>>>>>>>>>>> The first question might seems not really conscious,
>>>> but
>>>>>> would
>>>>>>>> you
>>>>>>>>>> tell
>>>>>>>>>>>> me
>>>>>>>>>>>>> what blocks searching during EFF reload, when it's
>>>>> triggered
>>>>>> by
>>>>>>>>>> handler
>>>>>>>>>>>> or
>>>>>>>>>>>>> by listener?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> We continuously index new documents using CommitWithin
>>>> to get
>>>>>>>> regular
>>>>>>>>>>>> commits. However, we observed that the EFFs were not
>>>> re-read,
>>>>>> so
>>>>>>> we
>>>>>>>>> had
>>>>>>>>>>> to
>>>>>>>>>>>> do external commits (curl '.../solr/update?commit=true')
>>>> to
>>>>>> force
>>>>>>>>>> reload.
>>>>>>>>>>>> When this is done, solr blocks. I can't tell you exactly
>>>> why
>>>>>> it's
>>>>>>>>> doing
>>>>>>>>>>>> that (it was related to SOLR-3985).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> I don't really get the sentence about sequential
>>>> commits
>>>>> and
>>>>>>>> number
>>>>>>>>>> of
>>>>>>>>>>>>> cores. Do I get right that file is replicated via
>>>>> Zookeeper?
>>>>>>>>> Doesn't
>>>>>>>>>> it
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Again, this is observed behavior. When we issue a commit
>>>> on a
>>>>>>>> system
>>>>>>>>>>> with a
>>>>>>>>>>>> system with many solr cores using EFFs, the system blocks
>>>>> for a
>>>>>>>> long
>>>>>>>>>> time
>>>>>>>>>>>> (15 minutes).  We do NOT use zookeeper for anything. The
>>>> EFF
>>>>>> is a
>>>>>>>>>> symlink
>>>>>>>>>>>> from each cores index dir to the actual file, which is
>>>>> updated
>>>>>> by
>>>>>>>> an
>>>>>>>>>>>> external process.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> causes scalability problem or long time to reload?
>>>> Will it
>>>>>> help
>>>>>>>> if
>>>>>>>>>>> we'll
>>>>>>>>>>>>> have, let's say ExternalDatabaseField which will pull
>>>>> values
>>>>>>> from
>>>>>>>>>> jdbc.
>>>>>>>>>>>> ie.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I think the possibility of having some fields being
>>>> retrieved
>>>>>>> from
>>>>>>>> an
>>>>>>>>>>>> external, dynamically updatable store would be really
>>>>>>> interesting.
>>>>>>>>> This
>>>>>>>>>>>> could be JDBC, something in-memory like redis, or a NoSql
>>>>>> product
>>>>>>>>> (e.g.
>>>>>>>>>>>> Cassandra).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> why all cores can't read these values simultaneously?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Again, this is a solr implementation detail that I can't
>>>>> answer
>>>>>>> :)
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Can you confirm that IDs in the file is ordered by the
>>>>> index
>>>>>>> term
>>>>>>>>>>> order?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, we sorted the files (standard UNIX sort).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> AFAIK it can impact load time.
>>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, it does.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Regarding your post-query solution can you tell me if
>>>> query
>>>>>>> found
>>>>>>>>>> 10000
>>>>>>>>>>>>> docs, but I need to display only first page with 100
>>>> rows,
>>>>>>>> whether
>>>>>>>>> I
>>>>>>>>>>> need
>>>>>>>>>>>>> to pull all 10K results to frontend to order them by
>>>> the
>>>>>> rank?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> In our architecture, the clients query an API that
>>>> generates
>>>>>> the
>>>>>>>> SOLR
>>>>>>>>>>>> query, retrieves the relevant additional fields that we
>>>>> needs,
>>>>>>> and
>>>>>>>>>>> returns
>>>>>>>>>>>> the relevant JSON to the front-end.
>>>>>>>>>>>> 
>>>>>>>>>>>> In our use case, results are returned from SOLR by the
>>>> 10's,
>>>>>> not
>>>>>>> by
>>>>>>>>> the
>>>>>>>>>>>> 1000's, so it is a manageable job. Even so, if solr
>>>> returned
>>>>>>>>> thousands
>>>>>>>>>> of
>>>>>>>>>>>> results, it would be up to the implementation of the api
>>>> to
>>>>>>> augment
>>>>>>>>>> only
>>>>>>>>>>>> the results that needed to be returned to the front-end.
>>>>>>>>>>>> 
>>>>>>>>>>>> Even so, patching up a JSON structure with 10000 results
>>>>> should
>>>>>>> be
>>>>>>>>>>>> possible.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm really appreciate if you comment on the questions
>>>>> above.
>>>>>>>>>>>>> PS: It's time to pitch, how much
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/SOLR-4085
>>>> "Commit-free
>>>>>>>>>>>>> ExternalFileField" can help you?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It looks very interesting :) Does it make it possible
>>>> to
>>>>>> avoid
>>>>>>>>>>> re-reading
>>>>>>>>>>>> the EFF on every commit, and only re-read the values that
>>>>> have
>>>>>>>>> actually
>>>>>>>>>>>> changed?
>>>>>>>>>>>> 
>>>>>>>>>>>> /Martin
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch <
>>>>> mak@issuu.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Solr 4.0 does support using EFFs, but it might not
>>>> give
>>>>> you
>>>>>>>> what
>>>>>>>>>>> you're
>>>>>>>>>>>>>> hoping fore.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We tried using Solr Cloud, and have given up again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The EFF is placed in the parent of the index
>>>> directory in
>>>>>>> each
>>>>>>>>>> core;
>>>>>>>>>>>> each
>>>>>>>>>>>>>> core reads the entire EFF and picks out the IDs that
>>>> it
>>>>> is
>>>>>>>>>>> responsible
>>>>>>>>>>>>> for.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In the current 4.0.0 release of solr, solr blocks
>>>>> (doesn't
>>>>>>>> answer
>>>>>>>>>>>>> queries)
>>>>>>>>>>>>>> while re-reading the EFF. Even worse, it seems that
>>>> the
>>>>>> time
>>>>>>> to
>>>>>>>>>>> re-read
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> EFF is multiplied by the number of cores in use
>>>> (i.e. the
>>>>>> EFF
>>>>>>>> is
>>>>>>>>>>>> re-read
>>>>>>>>>>>>> by
>>>>>>>>>>>>>> each core sequentially). The contents of the EFF
>>>> become
>>>>>>> active
>>>>>>>>>> after
>>>>>>>>>>>> the
>>>>>>>>>>>>>> first EXTERNAL commit (commitWithin does NOT work
>>>> here)
>>>>>> after
>>>>>>>> the
>>>>>>>>>>> file
>>>>>>>>>>>>> has
>>>>>>>>>>>>>> been updated.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In our case, the EFF was quite large - around 450MB
>>>> - and
>>>>>> we
>>>>>>>> use
>>>>>>>>> 16
>>>>>>>>>>>>> shards,
>>>>>>>>>>>>>> so when we triggered an external commit to force
>>>>>> re-reading,
>>>>>>>> the
>>>>>>>>>>> whole
>>>>>>>>>>>>>> system would block for several (10-15) minutes. This
>>>>> won't
>>>>>>> work
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>>>> production environment. The reason for the size of
>>>> the
>>>>> EFF
>>>>>> is
>>>>>>>>> that
>>>>>>>>>> we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> around 7M documents in the index; each document has
>>>> a 45
>>>>>>>>> character
>>>>>>>>>>> ID.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We got some help to try to fix the problem so that
>>>> the
>>>>>>> re-read
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>> EFF
>>>>>>>>>>>>>> proceeds in the background (see
>>>>>>>>>>>>>> here<https://issues.apache.org/jira/browse/SOLR-3985
>>>>> 
>>>>> for
>>>>>>>>>>>>>> a fix on the 4.1 branch). However, even though the
>>>>> re-read
>>>>>>>>> proceeds
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> background, the time required to launch solr now
>>>> takes at
>>>>>>> least
>>>>>>>>> as
>>>>>>>>>>> long
>>>>>>>>>>>>> as
>>>>>>>>>>>>>> re-reading the EFFs. Again, this is not good enough
>>>> for
>>>>> our
>>>>>>>>> needs.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The next issue is that you cannot sort on EFF fields
>>>>>> (though
>>>>>>>> you
>>>>>>>>>> can
>>>>>>>>>>>>> return
>>>>>>>>>>>>>> them as values using &fl=field(my_eff_field). This is
>>>>> also
>>>>>>>> fixed
>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>> 4.1
>>>>>>>>>>>>>> branch here <
>>>>>> https://issues.apache.org/jira/browse/SOLR-4022
>>>>>>>> .
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So: Even after these fixes, EFF performance is not
>>>> that
>>>>>>> great.
>>>>>>>>> Our
>>>>>>>>>>>>> solution
>>>>>>>>>>>>>> is as follows: The actual value of the popularity
>>>> measure
>>>>>>> (say,
>>>>>>>>>>> reads)
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> we want to report to the user is inserted into the
>>>> search
>>>>>>>>> response
>>>>>>>>>>>>>> post-query by our query front-end. This value will
>>>> then
>>>>> be
>>>>>>> the
>>>>>>>>>>>>>> authoritative value at the time of the query. The
>>>> value
>>>>> of
>>>>>>> the
>>>>>>>>>>>> popularity
>>>>>>>>>>>>>> measure that we use for boosting in the ranking of
>>>> the
>>>>>> search
>>>>>>>>>> results
>>>>>>>>>>>> is
>>>>>>>>>>>>>> only updated when the value has changed enough so
>>>> that
>>>>> the
>>>>>>>> impact
>>>>>>>>>> on
>>>>>>>>>>>> the
>>>>>>>>>>>>>> boost will be significant (say, more than 2%). This
>>>> does
>>>>>>>> require
>>>>>>>>>>>> frequent
>>>>>>>>>>>>>> re-indexing of the documents that have significant
>>>>> changes
>>>>>> in
>>>>>>>> the
>>>>>>>>>>>> number
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> reads, but at least we won't have to update a
>>>> document if
>>>>>> it
>>>>>>>>> moves
>>>>>>>>>>>> from,
>>>>>>>>>>>>>> say, 1000000 to 1000001 reads.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> /Martin Koch - ISSUU - senior systems architect.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni <
>>>>>>>>> simoneg@apache.org
>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> I'm planning to move a quite big Solr index to
>>>>> SolrCloud.
>>>>>>>>>> However,
>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> index, an external file field is used for
>>>> popularity
>>>>>>> ranking.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Does SolrCloud supports external file fields? How
>>>> does
>>>>> it
>>>>>>>> cope
>>>>>>>>>> with
>>>>>>>>>>>>>>> sharding and replication? Where should the external
>>>>> file
>>>>>> be
>>>>>>>>>> placed
>>>>>>>>>>>> now
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> the index folder is not local but in the cloud?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Are there otherwise other best practices to deal
>>>> with
>>>>> the
>>>>>>> use
>>>>>>>>>> cases
>>>>>>>>>>>>>>> external file fields were used for, like
>>>>>>> popularity/ranking,
>>>>>>>> in
>>>>>>>>>>>>>> SolrCloud?
>>>>>>>>>>>>>>> Custom ValueSources going to something external?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>> Simone
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Sincerely yours
>>>>>>>>>>>>> Mikhail Khludnev
>>>>>>>>>>>>> Principal Engineer,
>>>>>>>>>>>>> Grid Dynamics
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <http://www.griddynamics.com>
>>>>>>>>>>>>> <mkhludnev@griddynamics.com>
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Sincerely yours
>>>>>>>>> Mikhail Khludnev
>>>>>>>>> Principal Engineer,
>>>>>>>>> Grid Dynamics
>>>>>>>>> 
>>>>>>>>> <http://www.griddynamics.com>
>>>>>>>>> <mkhludnev@griddynamics.com>
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Sincerely yours
>>>>>>> Mikhail Khludnev
>>>>>>> Principal Engineer,
>>>>>>> Grid Dynamics
>>>>>>> 
>>>>>>> <http://www.griddynamics.com>
>>>>>>> <mkhludnev@griddynamics.com>
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Sincerely yours
>>>>> Mikhail Khludnev
>>>>> Principal Engineer,
>>>>> Grid Dynamics
>>>>> 
>>>>> <http://www.griddynamics.com>
>>>>> <mkhludnev@griddynamics.com>
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Sincerely yours
>>> Mikhail Khludnev
>>> Principal Engineer,
>>> Grid Dynamics
>>> 
>>> <http://www.griddynamics.com>
>>> <mkhludnev@griddynamics.com>
>>> 
>>> 
>> 


Mime
View raw message