lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: SolrCloud and exernal file fields
Date Wed, 28 Nov 2012 17:53:59 GMT
Mark,

Your comment is quite valuable. Let me mention the keyword to be able to
find later NoOpDistributingUpdateProcessorFactory.*
*Thanks*!
*


On Wed, Nov 28, 2012 at 5:56 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Keep in mind that the distrib update proc will be auto inserted into
> chains! You have to include a proc that disables it - see the FAQ:
> http://wiki.apache.org/solr/SolrCloud#FAQ
>
> - Mark
>
> On Nov 28, 2012, at 7:25 AM, Mikhail Khludnev <mkhludnev@griddynamics.com>
> wrote:
>
> > Martin,
> > Right as far node in Zookeeper DistributedUpdateProcessor will broadcast
> > commits to all peers. To hack this you can introduce dedicated
> > UpdateProcessorChain without DistributedUpdateProcessor and send commit
> to
> > that chain.
> > 28.11.2012 13:16 пользователь "Martin Koch" <mak@issuu.com> написал:
> >
> >> Mikhail
> >>
> >> I haven't experimented further yet. I think that the previous experiment
> >> of issuing a commit to a specific core proved that all cores get the
> >> commit, so I don't think that this approach will work.
> >>
> >> Thanks,
> >> /Martin
> >>
> >>
> >> On Tue, Nov 27, 2012 at 6:24 PM, Mikhail Khludnev <
> >> mkhludnev@griddynamics.com> wrote:
> >>
> >>> Martin,
> >>>
> >>> It's still not clear to me whether you solve the problem completely or
> >>> partially:
> >>> Does reducing number of cores free some resources for searching during
> >>> commit?
> >>> Does the commiting one-by-one core prevents the "freeze"?
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Thu, Nov 22, 2012 at 4:31 PM, Martin Koch <mak@issuu.com> wrote:
> >>>
> >>>> Mikhail
> >>>>
> >>>> To avoid freezes we deployed the patches that are now on the 4.1 trunk
> >>>> (bug
> >>>> 3985). But this wasn't good enough, because SOLR would still take very
> >>>> long
> >>>> to restart when that was necessary.
> >>>>
> >>>> I don't see how we could throw more hardware at the problem without
> >>>> making
> >>>> it worse, really - the only solution here would be *fewer* shards, not
> >>>>
> >>>> more.
> >>>>
> >>>> IMO it would be ideal if the lucene/solr community could come up with
> a
> >>>> good way of updating fields in a document without reindexing. This
> could
> >>>> be
> >>>> by linking to some external data store, or in the lucene/solr
> internals.
> >>>> If
> >>>> it would make things easier, a good first step would be to have
> >>>> dynamically
> >>>> updateable numerical fields only.
> >>>>
> >>>> /Martin
> >>>>
> >>>> On Wed, Nov 21, 2012 at 8:51 PM, Mikhail Khludnev <
> >>>> mkhludnev@griddynamics.com> wrote:
> >>>>
> >>>>> Martin,
> >>>>>
> >>>>> I don't think solrconfig.xml shed any light on. I've just found what
> I
> >>>>> didn't get in your setup - the way of how to explicitly assigning
> core
> >>>> to
> >>>>> collection. Now, I realized most of details after all!
> >>>>> Ball is on your side, let us know whether you have managed your cores
> >>>> to
> >>>>> commit one by one to avoid freeze, or could you eliminate pauses by
> >>>>> allocating more hardware?
> >>>>> Thanks in advance!
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 21, 2012 at 3:56 PM, Martin Koch <mak@issuu.com> wrote:
> >>>>>
> >>>>>> Mikhail,
> >>>>>>
> >>>>>> PSB
> >>>>>>
> >>>>>> On Wed, Nov 21, 2012 at 10:08 AM, Mikhail Khludnev <
> >>>>>> mkhludnev@griddynamics.com> wrote:
> >>>>>>
> >>>>>>> On Wed, Nov 21, 2012 at 11:53 AM, Martin Koch <mak@issuu.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> I wasn't aware until now that it is possible to send a commit to
> >>>> one
> >>>>>> core
> >>>>>>>> only. What we observed was the effect of curl
> >>>>>>>> localhost:8080/solr/update?commit=true but perhaps we should
> >>>>> experiment
> >>>>>>>> with solr/coreN/update?commit=true. A quick trial run seems to
> >>>>> indicate
> >>>>>>>> that a commit to a single core causes commits on all cores.
> >>>>>>>>
> >>>>>>> You should see something like this in the log:
> >>>>>>> ... SolrCmdDistributor .... Distrib commit to: ...
> >>>>>>>
> >>>>>>> Yup, a commit towards a single core results in a commit on all
> >>>> cores.
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Perhaps I should clarify that we are using SOLR as a black box;
> >>>> we do
> >>>>>> not
> >>>>>>>> touch the code at all - we only install the distribution WAR
> >>>> file and
> >>>>>>>> proceed from there.
> >>>>>>>>
> >>>>>>> I still don't understand how you deploy/launch Solr. How many
> >>>> jettys
> >>>>> you
> >>>>>>> start whether you have -DzkRun -DzkHost -DnumShards=2  or you
> >>>> specifies
> >>>>>>> shards= param for every request and distributes updates yourself?
> >>>> What
> >>>>>>> collections do you create and with which settings?
> >>>>>>>
> >>>>>>> We let SOLR do the sharding using one collection with 16 SOLR cores
> >>>>>> holding one shard each. We launch only one instance of jetty with
> the
> >>>>>> folllowing arguments:
> >>>>>>
> >>>>>> -DnumShards=16
> >>>>>> -DzkHost=<zookeeperhost:port>
> >>>>>> -Xmx10G
> >>>>>> -Xms10G
> >>>>>> -Xmn2G
> >>>>>> -server
> >>>>>>
> >>>>>> Would you like to see the solrconfig.xml?
> >>>>>>
> >>>>>> /Martin
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Also from my POV such deployments should start at least from
> >>>> *16*
> >>>>>> 4-way
> >>>>>>>>> vboxes, it's more expensive, but should be much better
> >>>> available
> >>>>>> during
> >>>>>>>>> cpu-consuming operations.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Do you mean that you recommend 16 hosts with 4 cores each? Or 4
> >>>> hosts
> >>>>>>> with
> >>>>>>>> 16 cores? Or am I misunderstanding something :) ?
> >>>>>>>>
> >>>>>>> I prefer to start from 16 hosts with 4 cores each.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Other details, if you use single jetty for all of them, are you
> >>>>> sure
> >>>>>>> that
> >>>>>>>>> jetty's threadpool doesn't limit requests? is it large enough?
> >>>>>>>>> You have 60G and set -Xmx=10G. are you sure that total size of
> >>>>> cores
> >>>>>>>> index
> >>>>>>>>> directories is less than 45G?
> >>>>>>>>>
> >>>>>>>>> The total index size is 230 GB, so it won't fit in ram, but
> >>>> we're
> >>>>>> using
> >>>>>>>> an
> >>>>>>>> SSD disk to minimize disk access time. We have tried putting the
> >>>> EFF
> >>>>>>> onto a
> >>>>>>>> ram disk, but this didn't have a measurable effect.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> /Martin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 21, 2012 at 2:07 AM, Martin Koch <mak@issuu.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Mikhail
> >>>>>>>>>>
> >>>>>>>>>> PSB
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Nov 20, 2012 at 7:22 PM, Mikhail Khludnev <
> >>>>>>>>>> mkhludnev@griddynamics.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Martin,
> >>>>>>>>>>>
> >>>>>>>>>>> Please find additional question from me below.
> >>>>>>>>>>>
> >>>>>>>>>>> Simone,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm sorry for hijacking your thread. The only what I've
> >>>> heard
> >>>>>> about
> >>>>>>>> it
> >>>>>>>>> at
> >>>>>>>>>>> recent ApacheCon sessions is that Zookeeper is supposed to
> >>>>>>> replicate
> >>>>>>>>>> those
> >>>>>>>>>>> files as configs under solr home. And I'm really looking
> >>>>> forward
> >>>>>> to
> >>>>>>>>> know
> >>>>>>>>>>> how it works with huge files in production.
> >>>>>>>>>>>
> >>>>>>>>>>> Thank You, Guys!
> >>>>>>>>>>>
> >>>>>>>>>>> 20.11.2012 18:06 пользователь "Martin Koch" <mak@issuu.com
> >>>>>
> >>>>>>> написал:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Mikhail
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please see answers below.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev <
> >>>>>>>>>>>> mkhludnev@griddynamics.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Martin,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for telling your own "war-story". It's really
> >>>>>> useful
> >>>>>>>> for
> >>>>>>>>>>>>> community.
> >>>>>>>>>>>>> The first question might seems not really conscious,
> >>>> but
> >>>>>> would
> >>>>>>>> you
> >>>>>>>>>> tell
> >>>>>>>>>>> me
> >>>>>>>>>>>>> what blocks searching during EFF reload, when it's
> >>>>> triggered
> >>>>>> by
> >>>>>>>>>> handler
> >>>>>>>>>>> or
> >>>>>>>>>>>>> by listener?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> We continuously index new documents using CommitWithin
> >>>> to get
> >>>>>>>> regular
> >>>>>>>>>>>> commits. However, we observed that the EFFs were not
> >>>> re-read,
> >>>>>> so
> >>>>>>> we
> >>>>>>>>> had
> >>>>>>>>>>> to
> >>>>>>>>>>>> do external commits (curl '.../solr/update?commit=true')
> >>>> to
> >>>>>> force
> >>>>>>>>>> reload.
> >>>>>>>>>>>> When this is done, solr blocks. I can't tell you exactly
> >>>> why
> >>>>>> it's
> >>>>>>>>> doing
> >>>>>>>>>>>> that (it was related to SOLR-3985).
> >>>>>>>>>>>
> >>>>>>>>>>> Is there a chance to get a thread dump when they are
> >>>> blocked?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Well I could try to recreate the situation. But the setup is
> >>>>> fairly
> >>>>>>>>> simple:
> >>>>>>>>>> Create a large EFF in a largeish index with many shards.
> >>>> Issue a
> >>>>>>>> commit,
> >>>>>>>>>> and then try to do a search. Solr will not respond to the
> >>>> search
> >>>>>>> before
> >>>>>>>>> the
> >>>>>>>>>> commit has completed, and this will take a long time.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I don't really get the sentence about sequential
> >>>> commits
> >>>>> and
> >>>>>>>> number
> >>>>>>>>>> of
> >>>>>>>>>>>>> cores. Do I get right that file is replicated via
> >>>>> Zookeeper?
> >>>>>>>>> Doesn't
> >>>>>>>>>> it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Again, this is observed behavior. When we issue a commit
> >>>> on a
> >>>>>>>> system
> >>>>>>>>>> with
> >>>>>>>>>>> a
> >>>>>>>>>>>> system with many solr cores using EFFs, the system blocks
> >>>>> for a
> >>>>>>>> long
> >>>>>>>>>> time
> >>>>>>>>>>>> (15 minutes).  We do NOT use zookeeper for anything. The
> >>>> EFF
> >>>>>> is a
> >>>>>>>>>> symlink
> >>>>>>>>>>>> from each cores index dir to the actual file, which is
> >>>>> updated
> >>>>>> by
> >>>>>>>> an
> >>>>>>>>>>>> external process.
> >>>>>>>>>>>
> >>>>>>>>>>> Hold on, I asked about Zookeeper because the subj mentions
> >>>>>>> SolrCloud.
> >>>>>>>>>>>
> >>>>>>>>>>> Do you use SolrCloud, SolrShards, or these cores are just
> >>>>>> replicas
> >>>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>> same index?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Ah - we use solr 4 out of the box, so I guess this is
> >>>> SolrCloud.
> >>>>>> I'm
> >>>>>>> a
> >>>>>>>>> bit
> >>>>>>>>>> unsure about the terminology here, but we've got a single
> >>>> index
> >>>>>>> divided
> >>>>>>>>>> into 16 shard. Each shard is hosted in a solr core.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Also, about simlink - Don't you share that file via some
> >>>> NFS?
> >>>>>>>>>>>
> >>>>>>>>>>> No, we generate the EFF on the local solr host (there is
> >>>> only
> >>>>> one
> >>>>>>>>>> physical
> >>>>>>>>>> host that holds all shards), so there is no need for NFS or
> >>>>> copying
> >>>>>>>> files
> >>>>>>>>>> around. No need for Zookeeper either.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> how many cores you run per box?
> >>>>>>>>>>>
> >>>>>>>>>> This box is a 16-virtual core (8 hyperthreaded cores)  with
> >>>> 60GB
> >>>>> of
> >>>>>>>> RAM.
> >>>>>>>>> We
> >>>>>>>>>> run 16 solr cores on this box in Jetty.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Do boxes has plenty of ram to cache filesystem beside of
> >>>> jvm
> >>>>>> heaps?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes. We've allocated 10GB for jetty, and left the rest for
> >>>> the
> >>>>>> OS.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> I assume you use 64 bit linux and mmap directory. Please
> >>>>> confirm
> >>>>>>>> that.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> We use 64-bit linux. I'm not sure about the mmap directory or
> >>>>> where
> >>>>>>>> that
> >>>>>>>>>> would be configured in solr - can you explain that?
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> causes scalability problem or long time to reload?
> >>>> Will it
> >>>>>> help
> >>>>>>>> if
> >>>>>>>>>>> we'll
> >>>>>>>>>>>>> have, let's say ExternalDatabaseField which will pull
> >>>>> values
> >>>>>>> from
> >>>>>>>>>> jdbc.
> >>>>>>>>>>> ie.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think the possibility of having some fields being
> >>>> retrieved
> >>>>>>> from
> >>>>>>>> an
> >>>>>>>>>>>> external, dynamically updatable store would be really
> >>>>>>> interesting.
> >>>>>>>>> This
> >>>>>>>>>>>> could be JDBC, something in-memory like redis, or a NoSql
> >>>>>> product
> >>>>>>>>> (e.g.
> >>>>>>>>>>>> Cassandra).
> >>>>>>>>>>>
> >>>>>>>>>>> Ok. Let's have it in mind as a possible direction.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Alternatively, an API that would allow updating a single
> >>>> field
> >>>>> for
> >>>>>> a
> >>>>>>>>>> document might be an option.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> why all cores can't read these values simultaneously?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Again, this is a solr implementation detail that I can't
> >>>>> answer
> >>>>>>> :)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Can you confirm that IDs in the file is ordered by the
> >>>>> index
> >>>>>>> term
> >>>>>>>>>>> order?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, we sorted the files (standard UNIX sort).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> AFAIK it can impact load time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Yes, it does
> >>>>>>>>>>>
> >>>>>>>>>>> Ok, I've got that you aware of it, and your IDs are just
> >>>>> strings,
> >>>>>>> not
> >>>>>>>>>>> integers.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Yes, ids are strings.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Regarding your post-query solution can you tell me if
> >>>> query
> >>>>>>> found
> >>>>>>>>>> 10000
> >>>>>>>>>>>>> docs, but I need to display only first page with 100
> >>>> rows,
> >>>>>>>> whether
> >>>>>>>>> I
> >>>>>>>>>>> need
> >>>>>>>>>>>>> to pull all 10K results to frontend to order them by
> >>>> the
> >>>>>> rank?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> In our architecture, the clients query an API that
> >>>> generates
> >>>>>> the
> >>>>>>>> SOLR
> >>>>>>>>>>>> query, retrieves the relevant additional fields that we
> >>>>> needs,
> >>>>>>> and
> >>>>>>>>>>> returns
> >>>>>>>>>>>> the relevant JSON to the front-end.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In our use case, results are returned from SOLR by the
> >>>> 10's,
> >>>>>> not
> >>>>>>> by
> >>>>>>>>> the
> >>>>>>>>>>>> 1000's, so it is a manageable job. Even so, if solr
> >>>> returned
> >>>>>>>>> thousands
> >>>>>>>>>> of
> >>>>>>>>>>>> results, it would be up to the implementation of the api
> >>>> to
> >>>>>>> augment
> >>>>>>>>>> only
> >>>>>>>>>>>> the results that needed to be returned to the front-end.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Even so, patching up a JSON structure with 10000 results
> >>>>> should
> >>>>>>> be
> >>>>>>>>>>>> possible.
> >>>>>>>>>>>
> >>>>>>>>>>> You are right. I'm concerned anyway because retrieving
> >>>> whole
> >>>>>> result
> >>>>>>>> is
> >>>>>>>>>>> expensive, and not always possible.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> In our case, getting the whole result is almost impossible,
> >>>>> because
> >>>>>>>> that
> >>>>>>>>>> would be millions of documents, and returning the Nth result
> >>>>> seems
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>> a
> >>>>>>>>>> quadratic (or worse) operation in SOLR.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I'm really appreciate if you comment on the questions
> >>>>> above.
> >>>>>>>>>>>>> PS: It's time to pitch, how much
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/SOLR-4085
> >>>> "Commit-free
> >>>>>>>>>>>>> ExternalFileField" can help you?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It looks very interesting :) Does it make it possible
> >>>> to
> >>>>>> avoid
> >>>>>>>>>>> re-reading
> >>>>>>>>>>>> the EFF on every commit, and only re-read the values that
> >>>>> have
> >>>>>>>>> actually
> >>>>>>>>>>>> changed?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> You don't need commit (in SOLR-4085) to reload file
> >>>> content,
> >>>>> but
> >>>>>>>> after
> >>>>>>>>>>> commit you need to read whole file and scan all key terms
> >>>> and
> >>>>>>>> postings.
> >>>>>>>>>>> That's because EFF sits on top of top level searcher. it's
> >>>> a
> >>>>>>>> Solr-like
> >>>>>>>>>> way.
> >>>>>>>>>>> In some future we might have per-segment EFF, in this case
> >>>>>> adding a
> >>>>>>>>>> segment
> >>>>>>>>>>> will trigger full file scan, but in the index only that new
> >>>>>> segment
> >>>>>>>>> will
> >>>>>>>>>> be
> >>>>>>>>>>> scanned. It should be faster. You know, straightforward
> >>>> sharing
> >>>>>>>>> internal
> >>>>>>>>>>> data structures between different index views/generations
> >>>> is
> >>>>> not
> >>>>>>>>>> possible.
> >>>>>>>>>>> If you are asking about applying delta changes on external
> >>>> file
> >>>>>>>> that's
> >>>>>>>>>>> something what we did ourselves http://goo.gl/P8GFq . This
> >>>>>> feature
> >>>>>>>> is
> >>>>>>>>>> much
> >>>>>>>>>>> more doubtful and vague, although it might be the next
> >>>>>> contribution
> >>>>>>>>> after
> >>>>>>>>>>> SOLR-4085.
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> /Martin
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch <
> >>>>> mak@issuu.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Solr 4.0 does support using EFFs, but it might not
> >>>> give
> >>>>> you
> >>>>>>>> what
> >>>>>>>>>>> you're
> >>>>>>>>>>>>>> hoping fore.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We tried using Solr Cloud, and have given up again.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The EFF is placed in the parent of the index
> >>>> directory in
> >>>>>>> each
> >>>>>>>>>> core;
> >>>>>>>>>>> each
> >>>>>>>>>>>>>> core reads the entire EFF and picks out the IDs that
> >>>> it
> >>>>> is
> >>>>>>>>>>> responsible
> >>>>>>>>>>>>> for.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the current 4.0.0 release of solr, solr blocks
> >>>>> (doesn't
> >>>>>>>> answer
> >>>>>>>>>>>>> queries)
> >>>>>>>>>>>>>> while re-reading the EFF. Even worse, it seems that
> >>>> the
> >>>>>> time
> >>>>>>> to
> >>>>>>>>>>> re-read
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> EFF is multiplied by the number of cores in use
> >>>> (i.e. the
> >>>>>> EFF
> >>>>>>>> is
> >>>>>>>>>>> re-read
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>> each core sequentially). The contents of the EFF
> >>>> become
> >>>>>>> active
> >>>>>>>>>> after
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> first EXTERNAL commit (commitWithin does NOT work
> >>>> here)
> >>>>>> after
> >>>>>>>> the
> >>>>>>>>>>> file
> >>>>>>>>>>>>> has
> >>>>>>>>>>>>>> been updated.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In our case, the EFF was quite large - around 450MB
> >>>> - and
> >>>>>> we
> >>>>>>>> use
> >>>>>>>>> 16
> >>>>>>>>>>>>> shards,
> >>>>>>>>>>>>>> so when we triggered an external commit to force
> >>>>>> re-reading,
> >>>>>>>> the
> >>>>>>>>>>> whole
> >>>>>>>>>>>>>> system would block for several (10-15) minutes. This
> >>>>> won't
> >>>>>>> work
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>>>> production environment. The reason for the size of
> >>>> the
> >>>>> EFF
> >>>>>> is
> >>>>>>>>> that
> >>>>>>>>>> we
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>> around 7M documents in the index; each document has
> >>>> a 45
> >>>>>>>>> character
> >>>>>>>>>>> ID.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We got some help to try to fix the problem so that
> >>>> the
> >>>>>>> re-read
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>> EFF
> >>>>>>>>>>>>>> proceeds in the background (see
> >>>>>>>>>>>>>> here<https://issues.apache.org/jira/browse/SOLR-3985
> >>>>>
> >>>>> for
> >>>>>>>>>>>>>> a fix on the 4.1 branch). However, even though the
> >>>>> re-read
> >>>>>>>>> proceeds
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> background, the time required to launch solr now
> >>>> takes at
> >>>>>>> least
> >>>>>>>>> as
> >>>>>>>>>>> long
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>> re-reading the EFFs. Again, this is not good enough
> >>>> for
> >>>>> our
> >>>>>>>>> needs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The next issue is that you cannot sort on EFF fields
> >>>>>> (though
> >>>>>>>> you
> >>>>>>>>>> can
> >>>>>>>>>>>>> return
> >>>>>>>>>>>>>> them as values using &fl=field(my_eff_field). This is
> >>>>> also
> >>>>>>>> fixed
> >>>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>>>> 4.1
> >>>>>>>>>>>>>> branch here <
> >>>>>> https://issues.apache.org/jira/browse/SOLR-4022
> >>>>>>>> .
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So: Even after these fixes, EFF performance is not
> >>>> that
> >>>>>>> great.
> >>>>>>>>> Our
> >>>>>>>>>>>>> solution
> >>>>>>>>>>>>>> is as follows: The actual value of the popularity
> >>>> measure
> >>>>>>> (say,
> >>>>>>>>>>> reads)
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>> we want to report to the user is inserted into the
> >>>> search
> >>>>>>>>> response
> >>>>>>>>>>>>>> post-query by our query front-end. This value will
> >>>> then
> >>>>> be
> >>>>>>> the
> >>>>>>>>>>>>>> authoritative value at the time of the query. The
> >>>> value
> >>>>> of
> >>>>>>> the
> >>>>>>>>>>> popularity
> >>>>>>>>>>>>>> measure that we use for boosting in the ranking of
> >>>> the
> >>>>>> search
> >>>>>>>>>> results
> >>>>>>>>>>> is
> >>>>>>>>>>>>>> only updated when the value has changed enough so
> >>>> that
> >>>>> the
> >>>>>>>> impact
> >>>>>>>>>> on
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> boost will be significant (say, more than 2%). This
> >>>> does
> >>>>>>>> require
> >>>>>>>>>>> frequent
> >>>>>>>>>>>>>> re-indexing of the documents that have significant
> >>>>> changes
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> number
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>> reads, but at least we won't have to update a
> >>>> document if
> >>>>>> it
> >>>>>>>>> moves
> >>>>>>>>>>> from,
> >>>>>>>>>>>>>> say, 1000000 to 1000001 reads.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> /Martin Koch - ISSUU - senior systems architect.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni <
> >>>>>>>>> simoneg@apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>> I'm planning to move a quite big Solr index to
> >>>>> SolrCloud.
> >>>>>>>>>> However,
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> index, an external file field is used for
> >>>> popularity
> >>>>>>> ranking.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Does SolrCloud supports external file fields? How
> >>>> does
> >>>>> it
> >>>>>>>> cope
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> sharding and replication? Where should the external
> >>>>> file
> >>>>>> be
> >>>>>>>>>> placed
> >>>>>>>>>>> now
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> the index folder is not local but in the cloud?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are there otherwise other best practices to deal
> >>>> with
> >>>>> the
> >>>>>>> use
> >>>>>>>>>> cases
> >>>>>>>>>>>>>>> external file fields were used for, like
> >>>>>>> popularity/ranking,
> >>>>>>>> in
> >>>>>>>>>>>>>> SolrCloud?
> >>>>>>>>>>>>>>> Custom ValueSources going to something external?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks in advance,
> >>>>>>>>>>>>>>> Simone
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Sincerely yours
> >>>>>>>>>>>>> Mikhail Khludnev
> >>>>>>>>>>>>> Principal Engineer,
> >>>>>>>>>>>>> Grid Dynamics
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <http://www.griddynamics.com>
> >>>>>>>>>>>>> <mkhludnev@griddynamics.com>
> >>>>>>>>>>>>>
> >>>>>>>>>>> 20.11.2012 18:06 пользователь "Martin Koch" <
> >>>> mak@issuu.com>
> >>>>>>>> написал:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Mikhail
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please see answers below.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev <
> >>>>>>>>>>>> mkhludnev@griddynamics.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Martin,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for telling your own "war-story". It's really
> >>>>>> useful
> >>>>>>>> for
> >>>>>>>>>>>>> community.
> >>>>>>>>>>>>> The first question might seems not really conscious,
> >>>> but
> >>>>>> would
> >>>>>>>> you
> >>>>>>>>>> tell
> >>>>>>>>>>>> me
> >>>>>>>>>>>>> what blocks searching during EFF reload, when it's
> >>>>> triggered
> >>>>>> by
> >>>>>>>>>> handler
> >>>>>>>>>>>> or
> >>>>>>>>>>>>> by listener?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> We continuously index new documents using CommitWithin
> >>>> to get
> >>>>>>>> regular
> >>>>>>>>>>>> commits. However, we observed that the EFFs were not
> >>>> re-read,
> >>>>>> so
> >>>>>>> we
> >>>>>>>>> had
> >>>>>>>>>>> to
> >>>>>>>>>>>> do external commits (curl '.../solr/update?commit=true')
> >>>> to
> >>>>>> force
> >>>>>>>>>> reload.
> >>>>>>>>>>>> When this is done, solr blocks. I can't tell you exactly
> >>>> why
> >>>>>> it's
> >>>>>>>>> doing
> >>>>>>>>>>>> that (it was related to SOLR-3985).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I don't really get the sentence about sequential
> >>>> commits
> >>>>> and
> >>>>>>>> number
> >>>>>>>>>> of
> >>>>>>>>>>>>> cores. Do I get right that file is replicated via
> >>>>> Zookeeper?
> >>>>>>>>> Doesn't
> >>>>>>>>>> it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Again, this is observed behavior. When we issue a commit
> >>>> on a
> >>>>>>>> system
> >>>>>>>>>>> with a
> >>>>>>>>>>>> system with many solr cores using EFFs, the system blocks
> >>>>> for a
> >>>>>>>> long
> >>>>>>>>>> time
> >>>>>>>>>>>> (15 minutes).  We do NOT use zookeeper for anything. The
> >>>> EFF
> >>>>>> is a
> >>>>>>>>>> symlink
> >>>>>>>>>>>> from each cores index dir to the actual file, which is
> >>>>> updated
> >>>>>> by
> >>>>>>>> an
> >>>>>>>>>>>> external process.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> causes scalability problem or long time to reload?
> >>>> Will it
> >>>>>> help
> >>>>>>>> if
> >>>>>>>>>>> we'll
> >>>>>>>>>>>>> have, let's say ExternalDatabaseField which will pull
> >>>>> values
> >>>>>>> from
> >>>>>>>>>> jdbc.
> >>>>>>>>>>>> ie.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think the possibility of having some fields being
> >>>> retrieved
> >>>>>>> from
> >>>>>>>> an
> >>>>>>>>>>>> external, dynamically updatable store would be really
> >>>>>>> interesting.
> >>>>>>>>> This
> >>>>>>>>>>>> could be JDBC, something in-memory like redis, or a NoSql
> >>>>>> product
> >>>>>>>>> (e.g.
> >>>>>>>>>>>> Cassandra).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> why all cores can't read these values simultaneously?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Again, this is a solr implementation detail that I can't
> >>>>> answer
> >>>>>>> :)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Can you confirm that IDs in the file is ordered by the
> >>>>> index
> >>>>>>> term
> >>>>>>>>>>> order?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, we sorted the files (standard UNIX sort).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> AFAIK it can impact load time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Yes, it does.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Regarding your post-query solution can you tell me if
> >>>> query
> >>>>>>> found
> >>>>>>>>>> 10000
> >>>>>>>>>>>>> docs, but I need to display only first page with 100
> >>>> rows,
> >>>>>>>> whether
> >>>>>>>>> I
> >>>>>>>>>>> need
> >>>>>>>>>>>>> to pull all 10K results to frontend to order them by
> >>>> the
> >>>>>> rank?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> In our architecture, the clients query an API that
> >>>> generates
> >>>>>> the
> >>>>>>>> SOLR
> >>>>>>>>>>>> query, retrieves the relevant additional fields that we
> >>>>> needs,
> >>>>>>> and
> >>>>>>>>>>> returns
> >>>>>>>>>>>> the relevant JSON to the front-end.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In our use case, results are returned from SOLR by the
> >>>> 10's,
> >>>>>> not
> >>>>>>> by
> >>>>>>>>> the
> >>>>>>>>>>>> 1000's, so it is a manageable job. Even so, if solr
> >>>> returned
> >>>>>>>>> thousands
> >>>>>>>>>> of
> >>>>>>>>>>>> results, it would be up to the implementation of the api
> >>>> to
> >>>>>>> augment
> >>>>>>>>>> only
> >>>>>>>>>>>> the results that needed to be returned to the front-end.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Even so, patching up a JSON structure with 10000 results
> >>>>> should
> >>>>>>> be
> >>>>>>>>>>>> possible.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I'm really appreciate if you comment on the questions
> >>>>> above.
> >>>>>>>>>>>>> PS: It's time to pitch, how much
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/SOLR-4085
> >>>> "Commit-free
> >>>>>>>>>>>>> ExternalFileField" can help you?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It looks very interesting :) Does it make it possible
> >>>> to
> >>>>>> avoid
> >>>>>>>>>>> re-reading
> >>>>>>>>>>>> the EFF on every commit, and only re-read the values that
> >>>>> have
> >>>>>>>>> actually
> >>>>>>>>>>>> changed?
> >>>>>>>>>>>>
> >>>>>>>>>>>> /Martin
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch <
> >>>>> mak@issuu.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Solr 4.0 does support using EFFs, but it might not
> >>>> give
> >>>>> you
> >>>>>>>> what
> >>>>>>>>>>> you're
> >>>>>>>>>>>>>> hoping fore.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We tried using Solr Cloud, and have given up again.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The EFF is placed in the parent of the index
> >>>> directory in
> >>>>>>> each
> >>>>>>>>>> core;
> >>>>>>>>>>>> each
> >>>>>>>>>>>>>> core reads the entire EFF and picks out the IDs that
> >>>> it
> >>>>> is
> >>>>>>>>>>> responsible
> >>>>>>>>>>>>> for.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the current 4.0.0 release of solr, solr blocks
> >>>>> (doesn't
> >>>>>>>> answer
> >>>>>>>>>>>>> queries)
> >>>>>>>>>>>>>> while re-reading the EFF. Even worse, it seems that
> >>>> the
> >>>>>> time
> >>>>>>> to
> >>>>>>>>>>> re-read
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> EFF is multiplied by the number of cores in use
> >>>> (i.e. the
> >>>>>> EFF
> >>>>>>>> is
> >>>>>>>>>>>> re-read
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>> each core sequentially). The contents of the EFF
> >>>> become
> >>>>>>> active
> >>>>>>>>>> after
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> first EXTERNAL commit (commitWithin does NOT work
> >>>> here)
> >>>>>> after
> >>>>>>>> the
> >>>>>>>>>>> file
> >>>>>>>>>>>>> has
> >>>>>>>>>>>>>> been updated.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In our case, the EFF was quite large - around 450MB
> >>>> - and
> >>>>>> we
> >>>>>>>> use
> >>>>>>>>> 16
> >>>>>>>>>>>>> shards,
> >>>>>>>>>>>>>> so when we triggered an external commit to force
> >>>>>> re-reading,
> >>>>>>>> the
> >>>>>>>>>>> whole
> >>>>>>>>>>>>>> system would block for several (10-15) minutes. This
> >>>>> won't
> >>>>>>> work
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>>>> production environment. The reason for the size of
> >>>> the
> >>>>> EFF
> >>>>>> is
> >>>>>>>>> that
> >>>>>>>>>> we
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>> around 7M documents in the index; each document has
> >>>> a 45
> >>>>>>>>> character
> >>>>>>>>>>> ID.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We got some help to try to fix the problem so that
> >>>> the
> >>>>>>> re-read
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>> EFF
> >>>>>>>>>>>>>> proceeds in the background (see
> >>>>>>>>>>>>>> here<https://issues.apache.org/jira/browse/SOLR-3985
> >>>>>
> >>>>> for
> >>>>>>>>>>>>>> a fix on the 4.1 branch). However, even though the
> >>>>> re-read
> >>>>>>>>> proceeds
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> background, the time required to launch solr now
> >>>> takes at
> >>>>>>> least
> >>>>>>>>> as
> >>>>>>>>>>> long
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>> re-reading the EFFs. Again, this is not good enough
> >>>> for
> >>>>> our
> >>>>>>>>> needs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The next issue is that you cannot sort on EFF fields
> >>>>>> (though
> >>>>>>>> you
> >>>>>>>>>> can
> >>>>>>>>>>>>> return
> >>>>>>>>>>>>>> them as values using &fl=field(my_eff_field). This is
> >>>>> also
> >>>>>>>> fixed
> >>>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>>>> 4.1
> >>>>>>>>>>>>>> branch here <
> >>>>>> https://issues.apache.org/jira/browse/SOLR-4022
> >>>>>>>> .
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So: Even after these fixes, EFF performance is not
> >>>> that
> >>>>>>> great.
> >>>>>>>>> Our
> >>>>>>>>>>>>> solution
> >>>>>>>>>>>>>> is as follows: The actual value of the popularity
> >>>> measure
> >>>>>>> (say,
> >>>>>>>>>>> reads)
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>> we want to report to the user is inserted into the
> >>>> search
> >>>>>>>>> response
> >>>>>>>>>>>>>> post-query by our query front-end. This value will
> >>>> then
> >>>>> be
> >>>>>>> the
> >>>>>>>>>>>>>> authoritative value at the time of the query. The
> >>>> value
> >>>>> of
> >>>>>>> the
> >>>>>>>>>>>> popularity
> >>>>>>>>>>>>>> measure that we use for boosting in the ranking of
> >>>> the
> >>>>>> search
> >>>>>>>>>> results
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> only updated when the value has changed enough so
> >>>> that
> >>>>> the
> >>>>>>>> impact
> >>>>>>>>>> on
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> boost will be significant (say, more than 2%). This
> >>>> does
> >>>>>>>> require
> >>>>>>>>>>>> frequent
> >>>>>>>>>>>>>> re-indexing of the documents that have significant
> >>>>> changes
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>> number
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>> reads, but at least we won't have to update a
> >>>> document if
> >>>>>> it
> >>>>>>>>> moves
> >>>>>>>>>>>> from,
> >>>>>>>>>>>>>> say, 1000000 to 1000001 reads.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> /Martin Koch - ISSUU - senior systems architect.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni <
> >>>>>>>>> simoneg@apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>> I'm planning to move a quite big Solr index to
> >>>>> SolrCloud.
> >>>>>>>>>> However,
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> index, an external file field is used for
> >>>> popularity
> >>>>>>> ranking.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Does SolrCloud supports external file fields? How
> >>>> does
> >>>>> it
> >>>>>>>> cope
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> sharding and replication? Where should the external
> >>>>> file
> >>>>>> be
> >>>>>>>>>> placed
> >>>>>>>>>>>> now
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> the index folder is not local but in the cloud?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are there otherwise other best practices to deal
> >>>> with
> >>>>> the
> >>>>>>> use
> >>>>>>>>>> cases
> >>>>>>>>>>>>>>> external file fields were used for, like
> >>>>>>> popularity/ranking,
> >>>>>>>> in
> >>>>>>>>>>>>>> SolrCloud?
> >>>>>>>>>>>>>>> Custom ValueSources going to something external?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks in advance,
> >>>>>>>>>>>>>>> Simone
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Sincerely yours
> >>>>>>>>>>>>> Mikhail Khludnev
> >>>>>>>>>>>>> Principal Engineer,
> >>>>>>>>>>>>> Grid Dynamics
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <http://www.griddynamics.com>
> >>>>>>>>>>>>> <mkhludnev@griddynamics.com>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Sincerely yours
> >>>>>>>>> Mikhail Khludnev
> >>>>>>>>> Principal Engineer,
> >>>>>>>>> Grid Dynamics
> >>>>>>>>>
> >>>>>>>>> <http://www.griddynamics.com>
> >>>>>>>>> <mkhludnev@griddynamics.com>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sincerely yours
> >>>>>>> Mikhail Khludnev
> >>>>>>> Principal Engineer,
> >>>>>>> Grid Dynamics
> >>>>>>>
> >>>>>>> <http://www.griddynamics.com>
> >>>>>>> <mkhludnev@griddynamics.com>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sincerely yours
> >>>>> Mikhail Khludnev
> >>>>> Principal Engineer,
> >>>>> Grid Dynamics
> >>>>>
> >>>>> <http://www.griddynamics.com>
> >>>>> <mkhludnev@griddynamics.com>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Sincerely yours
> >>> Mikhail Khludnev
> >>> Principal Engineer,
> >>> Grid Dynamics
> >>>
> >>> <http://www.griddynamics.com>
> >>> <mkhludnev@griddynamics.com>
> >>>
> >>>
> >>
>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mkhludnev@griddynamics.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message