lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@gmail.com>
Subject Re: RamUsageCrawler
Date Thu, 06 Dec 2018 15:44:27 GMT
I don't think it makes much sense, to be honest. Without actual
reflection you're binding things to a particular implementation
(you're assuming this and that JDK implementation). That's why we
decided to remove it instead of making it overly complex (and possibly
untrue). If a test is using it, perhaps the test itself should be
fixed.

D.
On Thu, Dec 6, 2018 at 4:35 PM Michael Sokolov <msokolov@gmail.com> wrote:
>
> That's what it looked like to me, too. I wonder if it would be worth
> improving the estimate for some very common Collections classes? I see
> this comment eg in BaseIndexFileFormatTestCase:
>
>       // we have no way to estimate the size of these things in codecs although
>       // something like a Collections.newSetFromMap(new HashMap<>()) uses quite
>       // some memory... So for now the test ignores the overhead of such
>       // collections but can we do better?
>
> This is in testRamBytesUsed and there is a kind of fudge factor in
> there for handling mismeasurement errors of the sort we are talking
> about. Actually the test seems to be more about validating the
> RamUsageTester than about validating the accounting in SegmentReader!
> There are lots of other usages in tests, but I suppose they don't
> require very precise handling of Collections classes (since they
> pass)? Anyway it is certainly possible to improve the estimate quite a
> bit and pretty easily for HashMap by simply counting the size of the
> Node that is used for each entry, although given the dynamic nature of
> these data structures (HashMap eg can use TreeNodes sometimes
> depending on data distribution) it would be almost impossible to be
> 100% accurate.
> On Thu, Dec 6, 2018 at 7:14 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
> >
> > > It's entirely possible it fails to dig into Maps correctly with newer Java
> > > releases; maybe Dawid or Uwe would know?
> >
> > We have removed all reflection from that class a while ago exactly
> > because of encapsulation issues introduced in newer Java versions.
> >
> > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java
> >
> > I think you may be thinking of RamUsageTester which is in the test
> > framework and indeed accumulates only keys and values from iterables.
> > These methods are for tests only and are rough. You shouldn't rely on
> > them for accurate memory consumption accounting (instead, use the
> > Accountable interface).
> >
> > Dawid
> >
> >
> > D.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message