lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xavi jmlucjav <jmluc...@gmail.com>
Subject Re: any changes about limitations on huge number of fields lately?
Date Sat, 30 May 2015 19:33:40 GMT
They reason for such a large number of fields:
- users create dynamically 'classes' of documents, say one user creates 10
classes on average
- for each 'class', the fields are created like this: "unique_id_"+fieldname
- there are potentially hundreds of thousands of users.

There is faceting in each users' fields.

So this will result in >1M fields, very sparsely populated. I warned them
this did not sound like a good design to me, but apparently someone very
knowledgeable in solr said this will work out fine. That is why I wanted to
double check...

On Sat, May 30, 2015 at 9:22 PM, Jack Krupansky <jack.krupansky@gmail.com>
wrote:

> Anything more than a few hundred seems very suspicious.
>
> Anything more than a few dozen or 50 or 75 seems suspicious as well.
>
> The point should not be how crazy can you get with Solr, but that craziness
> should be avoided altogether!
>
> Solr's design is optimal for a large number of relatively small documents,
> not large documents.
>
>
> -- Jack Krupansky
>
> On Sat, May 30, 2015 at 3:05 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
>
> > Nothing's really changed in that area lately. Your co-worker is
> > perhaps confusing the statement that "Solr has no a-priori limit on
> > the number of distinct fields that can be in a corpus" with supporting
> > an infinite number of fields. Not having a built-in limit is much
> > different than supporting....
> >
> > Whether Solr breaks with thousands and thousands of fields is pretty
> > dependent on what you _do_ with those fields. Simply doing keyword
> > searches isn't going to put the same memory pressure on as, say,
> > faceting on them all (even if in different queries).
> >
> > I'd really ask why so many fields are necessary though.
> >
> > Best,
> > Erick
> >
> > On Sat, May 30, 2015 at 6:18 AM, xavi jmlucjav <jmlucjav@gmail.com>
> wrote:
> > > Hi guys,
> > >
> > > someone I work with has been advised that currently Solr can support
> > > 'infinite' number of fields.
> > >
> > > I thought there was a practical limitation of say thousands of fields
> > (for
> > > sure less than a million), or    things can start to break (I think I
> > > remember seeings memory issues reported on the mailing list by several
> > > people).
> > >
> > >
> > > Was there any change I missed lately that makes having say 1M fields in
> > > Solr practical??
> > >
> > > thanks
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message