lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Hoerner <br...@bretthoerner.com>
Subject Re: What's the purpose of the bits option in compositeId (Solr 4.5)?
Date Tue, 08 Oct 2013 23:31:05 GMT
This is my clusterstate.json:
https://gist.github.com/bretthoerner/0098f741f48f9bb51433

And these are my core sizes (note large ones are sorted to the end):
https://gist.github.com/bretthoerner/f5b5e099212194b5dff6

I've only "heavily sent" 2 shards by now (I'm sharding by hour and it's
been running for 2). There *is* a little old data in my stream, but not
that much (like <5%). What's confusing to me is that 5 of them are rather
large, when I'd expect 2 of them to be.


On Tue, Oct 8, 2013 at 5:45 PM, Yonik Seeley <yseeley@gmail.com> wrote:

> On Tue, Oct 8, 2013 at 6:29 PM, Brett Hoerner <brett@bretthoerner.com>
> wrote:
> > I'm curious what the later "shard-local" bits do, if anything?
> >
> > I have a very large cluster (256 shards) and I'm sending most of my data
> > with a single "composite", e.g. 1234!<unique_id>, but I'm noticing the
> data
> > is being split among many of the shards.
>
> That shouldn't be the case.  All of your shards should have a lower
> hash value with all 0 bits and an upper hash value of all 1s (i.e.
> 0x????0000 to 0x????ffff)
> So you see any shards where that's not true?
>
> Also, is the router set to compositeId?
>
> -Yonik
>
> > My guess right now is that since I'm only using the default 16 bits my
> data
> > is being split across multiple shards (because of my high # of shards).
> >
> > Thanks,
> > Brett
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message